Node Failure Recovery

If a Buckit node suffers complete hardware failure (e.g. loss of all drives, data, etc.), the node begins healing operations once it rejoins the deployment. Buckit healing occurs only on the replaced hardware and does not typically impact deployment performance.

Buckit healing ensures consistency and correctness of all data restored onto the drive.

Exclusive access to drives

Buckit requires exclusive access to the drives or volumes provided for object storage. No other processes, software, scripts, or persons should perform any actions directly on the drives or volumes provided to Buckit or the objects or files Buckit places on them.

Do not use scripts or tools to directly modify, delete, or move any of the data shards, parity shards, or metadata files on the provided drives, including from one drive or node to another. Such operations are very likely to result in widespread corruption and data loss beyond Buckit’s ability to heal.

The replacement node hardware should be substantially similar to the failed node. There are no negative performance implications to using improved hardware.

The replacement drive hardware should be substantially similar to the failed drive. For example, replace a failed SSD with another SSD drive of the same capacity. While you can use drives with larger capacity, Buckit uses the smallest drive’s capacity as the ceiling for all drives in the Server Pool.

The following steps provide a more detailed walkthrough of node replacement. These steps assume a Buckit deployment where each node has a DNS hostname as per the documented prerequisites.

1) Start the Replacement Node

Ensure the new node has received all necessary security, firmware, and OS updates as per industry, regulatory, or organizational standards and requirements.

The new node software configuration must match that of the other nodes in the deployment, including but not limited to the OS and Kernel versions and configurations. Heterogeneous software configurations may result in unexpected or undesired behavior in the deployment.

2) Update Hostname for the New Node

Optional This step is only required if the replacement node has a different IP address from the failed host.

Ensure the hostname associated to the failed node now resolves to the new node.

For example, if https://buckit-1.example.net previously resolved to the failed host, it should now resolve to the new host.

3) Prepare and Rejoin the Replacement Node

Choose one of the following options to prepare the replacement node and rejoin it to the deployment.

Option 1: Use Buckit Manager Web UI (bm web)

Install Buckit Manager if needed. See Install the Buckit Manager.
If the cluster is not already registered in Buckit Manager, click + Import existing cluster and complete the import.
Open the target cluster.
If SSH credentials are not already configured for the cluster, open the cluster settings and save the SSH credentials Buckit Manager uses to reach the cluster hosts.
Click the failed node’s replacement host to open its node details page.
Click the Node Actions dropdown menu.
Select Provision replacement node… and follow the prompts.

Buckit Manager copies the required configuration from a healthy peer in the same pool, installs the matching Buckit package version, starts the service, and waits for the node to report healthy.

The replacement host must be clean before starting this workflow:

No existing Buckit or MinIO configuration files on the target host.
Data drives mounted at the intended mount points with no remaining data.

Option 2: Use CLI

Follow the deployment procedure to install and run Buckit on the replacement node using a matching configuration as the other nodes in the deployment.

The Buckit server version must match across all nodes.
The Buckit service and environment file configurations must match across all nodes.

Start the buckit server process on the node and monitor the process output using bm admin logs or by monitoring the Buckit service logs using journalctl -u buckit for systemd managed installations.

The server output should indicate that it has detected the other nodes in the deployment and begun healing operations.

4) Monitor Healing

Use bm admin heal to monitor overall healing status on the deployment. Buckit aggressively heals the node to ensure rapid recovery from the degraded state.

5) Next Steps

Continue monitoring the deployment until healing completes. Deployments with persistent and repeated node failures should schedule dedicated maintenance to identify the root cause.