Object Healing
What is healing?
Healing is Buckit’s ability to restore an object that has been damaged, corrupted, or partially lost. The loss can come from multiple types of corruptions or loss, such as but not limited to:
drive-level errors or failure
OS or filesystem errors or failure
Healing and Erasure Coding
The ability of Buckit to restore a damaged object relates directly to the following:
total number of drives in the erasure set where the object exists
number of drives available with intact parts of the object
parity setting for the erasure set
Parity refers to the number of dedicated recovery shards Buckit creates when writing the object. For example, an erasure set may have eight total drives and use three drives during a write for parity. In this scenario, Buckit splits an object into 5 data shards and create 3 parity shards. Buckit distributes these eight shards across the drives in the erasure set. No one drive contains only parity shards or only data shards. Instead, Buckit writes shards for each object in a randomized way to distribute reads evenly across drives.
When Buckit needs to provide the object, it looks for the data shards for the object. If any of the data shards are missing or damaged, Buckit uses one or more of the parity shards to restore the object. When looking for the parity shards, if any of the parity shards are missing or damaged, Buckit restores those as well, provided there are sufficient other shards to serve the object. For this scenario, up to three of data shard parts can be lost or damaged and Buckit can still successfully restore and serve the object.
The number of drives available with intact data or parity shards of the object must meet or exceed the number of drives used for data shards in the erasure set. In the scenario above, five drives with intact shards must be online and available for Buckit to successfully serve the object.
When does Buckit heal an object?
Buckit has a robust system for healing objects.
Healing during GET requests
Buckit automatically checks the consistency of an object’s data shards each time you request an object with a GET or HEAD operation.
For versioned buckets, Buckit also checks for consistency during PUT operation.
If all of the data shards are found intact, Buckit serves the object from the data shards without inspecting the corresponding parity shards.
If the object has missing or damaged data shards, Buckit uses the available parity shards to heal the object before serving it as part of the operation. There must be an intact parity shard available for each lost or damaged data shard, otherwise the object cannot be recovered. If any parity shards are lost or damaged, Buckit restores the parity shard, provided there are sufficient other parity shards to serve the object.
Healing with the object scanner
Buckit uses an object scanner to perform a number of tasks related to objects. One of these tasks checks the integrity of objects and, if found damaged or corrupted, heals them.
On each scanning pass, Buckit uses a hash of the object name to select one out of every 1,024 objects to check.
If any object is found to have lost shards, Buckit heals the object from available shards. By default, Buckit does not check for bit rot corruption using the scanner. This can be an expensive operation to perform and the risk of bit rot across multiple disks is low.
Healing by manual request
Administrators can use bm admin heal to initiate a full system healing.
The procedure is very resource intensive and not typically needed.
Consult with Buckit Engineers before manually starting a healing process on a deployment.
Healing metrics
Buckit provides several healing metrics to monitor the status of healing processes on a deployment.
Refer to the Metrics and alerts for more information on available endpoints and configuration.
