Expand a Distributed Buckit Deployment

Networking and Firewalls

Each node should have full bidirectional network access to every other node in the deployment. For containerized or orchestrated infrastructures, this may require specific configuration of networking and routing components such as ingress or load balancers. Certain operating systems may also require setting firewall rules. For example, the following command explicitly opens the default Buckit server API port 9000 on servers using firewalld:

firewall-cmd --permanent --zone=public --add-port=9000/tcp
firewall-cmd --reload

All Buckit servers in the deployment must use the same listen port.

If you set a static Buckit Console port (e.g. :9001) you must also grant access to that port to ensure connectivity from external clients.

Buckit strongly recomends using a load balancer to manage connectivity to the cluster. The Load Balancer should use a “Least Connections” algorithm for routing requests to the Buckit deployment, since any Buckit node in the deployment can receive, route, or process client requests.

The following load balancers are known to work well with Buckit:

• NGINX

• HAProxy

Configuring firewalls or load balancers to support Buckit is out of scope for this procedure. The Configure NGINX Proxy for Buckit Server reference provides a baseline configuration for using NGINX as a reverse proxy with basic load balancing configured.

Sequential Hostnames

Buckit requires using expansion notation {x...y} to denote a sequential series of Buckit hosts when creating a server pool. Buckit therefore requires using sequentially-numbered hostnames to represent each buckit server process in the pool.

Create the necessary DNS hostname mappings prior to starting this procedure. For example, the following hostnames would support a 4-node distributed server pool:

• buckit5.example.com

• buckit6.example.com

• buckit7.example.com

• buckit8.example.com

You can specify the entire range of hostnames using the expansion notation buckit{5...8}.example.com.

Configuring DNS to support Buckit is out of scope for this procedure.

Storage Requirements

The following requirements summarize the Storage section of Buckit’s hardware recommendations:

Use Local Storage

Direct-Attached Storage (DAS) has significant performance and consistency advantages over networked storage (NAS, SAN, NFS). Buckit strongly recommends flash storage (NVMe, SSD) for primary or “hot” data.

Use XFS-Formatting for Drives

Buckit strongly recommends provisioning XFS formatted drives for storage. Buckit uses XFS as part of internal testing and validation suites, providing additional confidence in performance and behavior at all scales.

Buckit does not test nor recommend any other filesystem, such as EXT4, BTRFS, or ZFS.

Use Consistent Type of Drive

Buckit does not distinguish drive types and does not benefit from mixed storage types. Each pool must use the same type (NVMe, SSD)

For example, deploy a pool consisting of only NVMe drives. If you deploy some drives as SSD or HDD, Buckit treats those drives identically to the NVMe drives. This can result in performance issues, as some drives have differing or worse read/write characteristics and cannot respond at the same rate as the NVMe drives.

Use Consistent Size of Drive

Buckit limits the size used per drive to the smallest drive in the pool.

For example, deploy a pool consisting of the same number of NVMe drives with identical capacity of 7.68TiB. If you deploy one drive with 3.84TiB, Buckit treats all drives in the pool as having that smaller capacity.

Configure Sequential Drive Mounting

Buckit uses Go expansion notation {x...y} to denote a sequential series of drives when creating the new server pool, where all nodes in the server pool have an identical set of mounted drives. Configure drive mounting paths as a sequential series to best support this notation. For example, mount your drives using a pattern of /mnt/drive-n, where n starts at 1 and increments by 1 per drive.

Persist Drive Mounting and Mapping Across Reboots

Use /etc/fstab to ensure consistent drive-to-mount mapping across node reboots.

Non-Linux Operating Systems should use the equivalent drive mount management tool.

Minimum Drives for Erasure Code Parity

Buckit requires each pool satisfy the deployment erasure code settings. Specifically the new pool topology must support a minimum of 2 x EC:N drives per erasure set, where EC:N is the Standard parity storage class of the deployment. This requirement ensures the new server pool can satisfy the expected SLA of the deployment.

You can use the Erasure Code Calculator to check the Erasure Code Stripe Size (K+M) of your new pool. If the highest listed value is at least 2 x EC:N, the pool supports the deployment’s erasure parity settings.

Time Synchronization

Multi-node systems must maintain synchronized time and date to maintain stable internode operations and interactions. Make sure all nodes sync to the same time server regularly. Operating systems vary for methods used to synchronize time and date, such as with ntp, timedatectl, or timesyncd.

Check the documentation for your operating system for how to set up and maintain accurate and identical system clock times across nodes.

Back Up Cluster Settings First

Use the bm admin cluster bucket export and bm admin cluster iam export commands to take a snapshot of the bucket metadata and IAM configurations respectively prior to starting decommissioning. You can use these snapshots to restore bucket and IAM settings to recover from user or process errors as necessary.

Writing Files

Buckit does not automatically rebalance objects across the new server pools. Instead, Buckit performs new write operations to the pool with the most free storage weighted by the amount of free space on the pool divided by the free space across all available pools.

The formula to determine the probability of a write operation on a particular pool is

\(FreeSpaceOnPoolA / FreeSpaceOnAllPools\)

Consider a situation where a group of three pools has a total of 10 TiB of free space distributed as:

• Pool A has 3 TiB of free space

• Pool B has 2 TiB of free space

• Pool C has 5 TiB of free space

Buckit calculates the probability of a write operation to each of the pools as:

• Pool A: 30% chance (\(3TiB / 10TiB\))

• Pool B: 20% chance (\(2TiB / 10TiB\))

• Pool C: 50% chance (\(5TiB / 10TiB\))

In addition to the free space calculation, if a write option (with parity) would bring a drive usage above 99% or a known free inode count below 1000, Buckit does not write to the pool.

If desired, you can manually initiate a rebalance procedure with bm admin rebalance. For more about how rebalancing works, see managing objects across a deployment.

Likewise, Buckit does not write to pools in a decommissioning process.

Expansion is Non-Disruptive

Adding a new server pool requires restarting all buckit server processes in the deployment at around same time.

Buckit strongly recommends restarting all Buckit Server processes in a deployment simultaneously. Buckit operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.

Do not perform “rolling” (e.g. one node at a time) restarts.

Capacity-Based Planning

Buckit recommends planning storage capacity sufficient to store at least 2 years of data before reaching 70% usage. Performing server pool expansion more frequently or on a “just-in-time” basis generally indicates an architecture or planning issue.

For example, consider an application suite expected to produce at least 100 TiB of data per year and a 3 year target before expansion. The deployment has ~500TiB of usable storage in the initial server pool, such that the cluster safely met the 70% threshold with some buffer for data growth. The new server pool should ideally meet at minimum 500TiB of additional storage to allow for a similar lifespan before further expansion.

Since Buckit erasure coding requires some storage for parity, the total raw storage must exceed the planned usable capacity. Consider using the Erasure Code Calculator for guidance in planning capacity around specific erasure code settings.

Recommended Operating Systems

This tutorial assumes all hosts running Buckit use a recommended Linux operating system.

All hosts in the deployment should run with matching software configurations.

1) Install the Buckit Package on Each Node in the New Server Pool

Use the package format appropriate for each new Linux host in the server pool.

curl -fsSL https://buckit-io.github.io/buckit/install-linux.sh | sh
sudo dnf install ./buckit.rpm

curl -fsSL https://buckit-io.github.io/buckit/install-linux.sh | sh
sudo apt install ./buckit.deb

2) Add TLS/SSL Certificates (Optional)

If the current deployment already has TLS enabled, configure matching TLS certificates on each new node before continuing.

If the current deployment does not have TLS enabled, you can skip this step.

Buckit enables Transport Layer Security (TLS) 1.2+ automatically upon detecting a valid x.509 certificate (.crt) and private key (.key) in the configured certificate directory.

Specify the certificate directory using the buckit server --certs-dir commandline argument. For systemd-managed deployments, modify the MINIO_OPTS variable in /etc/default/minio to set this option.

The systemd user which runs the buckit server process must have read and listing permissions for the specified directory.

• Place TLS certificates into the configured certificate directory on each host.

• If any Buckit server or client uses certificates signed by an unknown Certificate Authority (self-signed or internal CA), you must place the CA certs in the CAs subdirectory under the configured certificate directory on all Buckit hosts in the deployment. Buckit rejects invalid certificates (untrusted, expired, or malformed).

For more specific guidance on configuring Buckit for TLS, including multi-domain support via Server Name Indication (SNI), see Network Encryption (TLS). You can optionally skip this step to deploy without TLS enabled. Buckit strongly recommends against non-TLS deployments outside of early development.

3) Create the systemd Service File

The .deb or .rpm packages install the following systemd service file to /usr/lib/systemd/system/minio.service. For binary installations, create this file manually on all Buckit hosts.

[Unit]
Description=Buckit
Documentation=/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local

User=minio-user
Group=minio-user
ProtectProc=invisible

EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# Buckit RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

The minio.service file runs as the minio-user User and Group by default. You can create the user and group using the groupadd and useradd commands. The following example creates the user, group, and sets permissions to access the folder paths intended for use by Buckit. These commands typically require root (sudo) permissions.

groupadd -r minio-user
useradd -M -r -g minio-user minio-user
chown minio-user:minio-user /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4

The specified drive paths are provided as an example. Change them to match the path to those drives intended for use by Buckit.

Alternatively, change the User and Group values to another user and group on the system host with the necessary access and permissions.

Buckit publishes additional startup script examples on github.com/minio/minio-service.

To update deployments managed using systemctl, see Update systemctl-Managed Buckit Deployments.

4) Create the Service Environment File

Create an environment file at /etc/default/minio. The Buckit service uses this file as the source of all environment variables used by Buckit and the minio.service file.

The following examples assumes that:

• The deployment has a single server pool consisting of four Buckit server hosts with sequential hostnames.

  buckit1.example.com   buckit3.example.com
  buckit2.example.com   buckit4.example.com
  

  Each host has 4 locally attached drives with sequential mount points:

  /mnt/disk1/buckit   /mnt/disk3/buckit
  /mnt/disk2/buckit   /mnt/disk4/buckit
  

• The new server pool consists of eight new Buckit hosts with sequential hostnames:

  buckit5.example.com   buckit9.example.com
  buckit6.example.com   buckit10.example.com
  buckit7.example.com   buckit11.example.com
  buckit8.example.com   buckit12.example.com
  

• All hosts have eight locally-attached drives with sequential mount-points:

  /mnt/disk1/buckit  /mnt/disk5/buckit
  /mnt/disk2/buckit  /mnt/disk6/buckit
  /mnt/disk3/buckit  /mnt/disk7/buckit
  /mnt/disk4/buckit  /mnt/disk8/buckit
  

• The deployment has a load balancer running at https://buckit.example.net that manages connections across all Buckit hosts. The load balancer should not be routing requests to the new hosts at this step, but should have the necessary configuration updates planned.

Modify the example to reflect your deployment topology:

# Set the hosts and volumes Buckit uses at startup
# The command uses Buckit expansion notation {x...y} to denote a
# sequential series.
#
# The following example starts the Buckit server with two server pools.
#
# The space delimiter indicates a seperate server pool
#
# The second set of hostnames and volumes is the newly added pool.
# The pool has sufficient stripe size to meet the existing erasure code
# parity of the deployment (2 x EC:4)
#
# The command includes the port on which the Buckit servers listen for each
# server pool.

MINIO_VOLUMES="https://buckit{1...4}.example.net:9000/mnt/disk{1...4}/buckit https://buckit{5...12}.example.net:9000/mnt/disk{1...8}/buckit"

# Set all Buckit server options
#
# The following explicitly sets the Buckit Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.

MINIO_OPTS="--console-address :9001"

# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.

MINIO_ROOT_USER=buckitadmin

# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.

MINIO_ROOT_PASSWORD=buckit-secret-key-CHANGE-ME

You may specify other environment variables or server commandline options as required by your deployment. All Buckit nodes in the deployment should include the same environment variables with the matching values.

5) Restart the Buckit Deployment with Expanded Configuration

Issue the following commands on each node simultaneously in the deployment to restart the Buckit service:

sudo systemctl restart buckit.service

Use the following commands to confirm the service is online and functional:

sudo systemctl status buckit.service
journalctl -f -u buckit.service

Buckit may log an increased number of non-critical warnings while the server processes connect and synchronize. These warnings are typically transient and should resolve as the deployment comes online.

Buckit strongly recommends restarting all Buckit Server processes in a deployment simultaneously. Buckit operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.

Do not perform “rolling” (e.g. one node at a time) restarts.

6) Next Steps

• Update any load balancers, reverse proxies, or other network control planes to route client requests to the new hosts in the Buckit distributed deployment. While Buckit automatically manages routing internally, having the control planes handle initial connection management may reduce network hops and improve efficiency.

• Review the Buckit Console to confirm the updated cluster topology and monitor performance.