Hardware Checklist

Networking

Buckit recommends high speed networking to support the maximum possible throughput of the attached storage (aggregated drives, storage controllers, and PCIe busses). The following table provides a general guideline for the maximum storage throughput supported by a given physical or virtual network interface. This table assumes all network infrastructure components, such as routers, switches, and physical cabling, also supports the NIC bandwidth.

Networking has the greatest impact on Buckit performance, where low per-host bandwidth artificially constrains the potential performance of the storage. The following examples of network throughput constraints assume spinning disks with ~100MB/S sustained I/O

• 1GbE network link can support up to 125MB/s, or one spinning disk

• 10GbE network can support approximately 1.25GB/s, potentially supporting 10-12 spinning disks

• 25GbE network can support approximately 3.125GB/s, potentially supporting ~30 spinning disks

Memory

Memory primarily constrains the number of concurrent simultaneous object API requests per node.

Buckit automatically admits requests based on an estimated in-flight request memory budget when api requests_max is unset. The request memory budget is calculated as:

\(freeSystemRam * 0.90 * 0.75\)

Buckit uses the following request memory estimates for object reads and writes:

erasureSetDriveCount is the number of drives in the largest erasure set.

The following table lists example maximum concurrent requests on a node based on free system RAM. GET capacity is independent of erasure set width. PUT capacity depends on the largest erasure set drive count:

The following table provides general guidelines for allocating memory for use by Buckit based on the total amount of local storage on the node:

Storage

mkfs.xfs /dev/sdb -L BUCKITDRIVE1
mkfs.xfs /dev/sdc -L BUCKITDRIVE2
mkfs.xfs /dev/sdd -L BUCKITDRIVE3
mkfs.xfs /dev/sde -L BUCKITDRIVE4

$ nano /etc/fstab

# <file system>        <mount point>    <type>  <options>         <dump>  <pass>
LABEL=BUCKITDRIVE1     /mnt/drive-1     xfs     defaults,noatime  0       2
LABEL=BUCKITDRIVE2     /mnt/drive-2     xfs     defaults,noatime  0       2
LABEL=BUCKITDRIVE3     /mnt/drive-3     xfs     defaults,noatime  0       2
LABEL=BUCKITDRIVE4     /mnt/drive-4     xfs     defaults,noatime  0       2

#!/bin/bash

for i in $(df -h | grep /mnt/drive | awk '{ print $1 }'); do
      mountPath="$(df -h | grep $i | awk '{ print $6 }')"
      deviceName="$(basename $i)"
      echo "Modifying xfs max_retries and retry_timeout_seconds for drive $i mounted at $mountPath"
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/EIO/max_retries
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/ENOSPC/max_retries
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/default/max_retries
done
exit 0

@reboot /opt/minio/xfs-retry-settings.sh

Operating System Diagnostic Tools

Test each drive independently on all servers to ensure they are identical in performance. Use the results of these OS-level tools to verify the capabilities of your storage hardware. Record the results for later reference.

1. Test the drive’s performance during write operations

   This tests checks a drive’s ability to write new data (uncached) to the drive by creating a specified number of blocks at up to a certain number of bytes at a time to mimic how a drive would function with writing uncached data. This allows you to see the actual drive performance with consistent file I/O.

   dd if=/dev/zero of=/mnt/driveN/testfile bs=128k count=80000 oflag=direct conv=fdatasync > dd-write-drive1.txt
   

   Replace driveN with the path for the drive you are testing.

   dd	The command to copy and paste data.
   if=/dev/zero	Read from /dev/zero, an system-generated endless stream of 0 bytes used to create a file of a specified size
   of=/mnt/driveN/testfile	Write to /mnt/driveN/testfile
   bs=128k	Write up to 128,000 bytes at a time
   count=80000	Write up to 80000 blocks of data
   oflag=direct	Use direct I/O to write to avoid data from caching
   conv=fdatasync	Physically write output file data before finishing
   > dd-write-drive1.txt	Write the contents of the operation’s output to dd-write-drive1.txt in the current working directory

   The operation returns the number of files written, total size written in bytes, the total length of time for the operation (in seconds), and the speed of the writing in some order of bytes per second.

2. Test the drive’s performance during read operations

   dd if=/mnt/driveN/testfile of=/dev/null bs=128k iflag=direct > dd-read-drive1.txt
   

   Replace driveN with the path for the drive you are testing.

   dd	The command to copy and paste data
   if=/mnt/driveN/testfile	Read from /mnt/driveN/testfile; replace with the path to the file to use for testing the drive’s read performance
   of=/dev/null	Write to /dev/null, a virtual file that does not persist after the operation completes
   bs=128k	Write up to 128,000 bytes at a time
   count=80000	Write up to 80000 blocks of data
   iflag=direct	Use direct I/O to read and avoid data from caching
   > dd-read-drive1.txt	Write the contents of the operation’s output to dd-read-drive1.txt in the current working directory

   Use a sufficiently sized file that mimics the primary use case for your deployment to get accurate read test results.

   The following guidelines may help during performance testing:

   • Small files: < 128KB

   • Normal files: 128KB – 1GB

   • Large files: > 1GB

   You can use the head command to create a file to use. The following command example creates a 10 Gigabyte file called testfile.

   head -c 10G </dev/urandom > testfile
   

   The operation returns the number of files read, total size read in bytes, the total length of time for the operation (in seconds), and the speed of the reading in bytes per second.

Third Party Diagnostic Tools

IO Controller test

Use IOzone to test the input/output controller and all drives in combination. Document the performance numbers for each server in your deployment.

iozone -s 1g -r 4m -i 0 -i 1 -i 2 -I -t 160 -F /mnt/sdb1/tmpfile.{1..16} /mnt/sdc1/tmpfile.{1..16} /mnt/sdd1/tmpfile.{1..16} /mnt/sde1/tmpfile.{1..16} /mnt/sdf1/tmpfile.{1..16} /mnt/sdg1/tmpfile.{1..16} /mnt/sdh1/tmpfile.{1..16} /mnt/sdi1/tmpfile.{1..16} /mnt/sdj1/tmpfile.{1..16} /mnt/sdk1/tmpfile.{1..16} > iozone.txt