Recover after Hardware Failure
Distributed Buckit deployments rely on Erasure Coding to provide built-in tolerance for multiple drive or node failures. Depending on the deployment topology and the selected erasure code parity, Buckit can tolerate the loss of up to half the drives or nodes in the deployment while maintaining read access (“read quorum”) to objects.
The following table lists the typical types of failure in a Buckit deployment and links to procedures for recovering from each:
Failure Type |
Description |
|---|---|
Buckit supports hot-swapping failed drives with new healthy drives. |
|
Buckit detects when a node rejoins the deployment and begins proactively healing the node shortly after it is joined back to the cluster healing data previously stored on that node. |
|
Buckit Site Replication supports complete resynchronization of buckets, objects, and replication-eligible configuration settings after total site loss. |
Since Buckit can operate in a degraded state without significant performance loss, administrators can schedule hardware replacement in proportion to the rate of hardware failure. “Normal” failure rates (single drive or node failure) may allow for a more reasonable replacement timeframe, while “critical” failure rates (multiple drives or nodes) may require a faster response.
For nodes with one or more drives that are either partially failed or operating in a degraded state (increasing drive errors, SMART warnings, timeouts in Buckit logs, etc.), you can safely unmount the drive if the cluster has sufficient remaining healthy drives to maintain read and write quorum. Missing drives are less disruptive to the deployment than drives that are consistently producing read and write errors.
Exclusive access to drives
Buckit requires exclusive access to the drives or volumes provided for object storage. No other processes, software, scripts, or persons should perform any actions directly on the drives or volumes provided to Buckit or the objects or files Buckit places on them.
Do not use scripts or tools to directly modify, delete, or move any of the data shards, parity shards, or metadata files on the provided drives, including from one drive or node to another. Such operations are very likely to result in widespread corruption and data loss beyond Buckit’s ability to heal.
Buckit Professional Support
Engage Buckit support when planning or executing recovery for critical hardware failures, especially when multiple drives, nodes, or an entire site are affected.
