Scrubbing

To protect against bit-rot, Ceph periodically runs a process called scrubbing to verify the data stored across the OSDs. The scrubbing process works at the PG level and compares the contents of each of the PGs across all of the participating OSDs to check that each OSD has identical contents. If an OSD is found to have an object copy that differs to the others or is even missing the object, the PG is marked as inconsistent. Inconsistent PGs can be repaired by instructing Ceph to repair the PG; this is covered in further detail in Chapter 11, Troubleshooting.

There are two types of scrubbing: normal and deep. Normal scrubbing simply checks for the existence of the object and that its metadata is correct; deep scrubbing is when the actual data is compared. Deep scrubbing tends to be much more I/O-intensive than normal scrubbing.

Although BlueStore now supports checksums, the need for scrubbing is not completely redundant. BlueStore only compares the checksums against the data being actively read, and so for cold data that is very rarely written, data loss or corruption could occur and only the scrubbing process would detect this.

There are a number of scrubbing tuning options that are covered later in Chapter 9, Tuning Ceph; they influence the scheduling of when scrubbing takes place and the impact on client I/O.