Transition Objects from Buckit to Azure

Install and Configure bm

This procedure uses bm for performing operations on the Buckit cluster. Install bm on a machine with network access to both source and destination clusters. See the bm Installation Quickstart for instructions on downloading and installing bm.

Use the bm alias set command to create an alias for the source Buckit cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying transition operations.

Required Buckit Permissions

Buckit requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

• s3:PutLifecycleConfiguration

• s3:GetLifecycleConfiguration

Buckit also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

• admin:SetTier

• admin:ListTier

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:.

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Required Azure Permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, Buckit requires the Azure credentials provide read, write, list, and delete permissions for the remote storage account and container.

Refer to the Azure RBAC documentation for more complete guidance on configuring the required permissions.

Remote Storage Account and Container Must Exist

Create the remote Azure storage account and container prior to configuring lifecycle management tiers or rules using that resource as the target. When creating the Azure storage account, ensure the storage account corresponds to either Standard or Premium blob storage with the locally redundant storage (LRS) redundancy option. The Azure Go SDK API used by Buckit does not support any other redundancy options.

If you set a Storage Account default access tier, Buckit uses that default if you do not specify a storage class when defining the remote tier. Ensure you document the settings of both your Azure storage account and Buckit tiering configuration to avoid any potential confusion, misconfiguration, or other unexpected outcomes.

For more information on Azure storage accounts, see Storage accounts.

Exclusive Access to Remote Data

Buckit requires exclusive access to the transitioned data on the remote storage tier. Object metadata on the “hot” Buckit source is strongly linked to the object data on the “warm/cold” remote tier. Buckit cannot retrieve object data without access to the remote, nor can the remote be used to restore lost metadata on the source.

All access to the transitioned objects must occur through Buckit via S3 API operations only. Manually modifying a transitioned object - whether the metadata on the “hot” Buckit tier or the object data on the remote “warm/cold” tier - may result in loss of that object data.

Buckit ignores any objects in the remote bucket or bucket prefix not explicitly managed by the Buckit deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

• No external mutation, migration, or deletion of objects on the remote storage.

• No lifecycle management rules (e.g. transition or expiration) on the remote storage bucket.

Buckit stores all transitioned objects in the remote storage bucket or resource under a unique per-deployment prefix value. This value is not intended to support identifying the source deployment from the backend. Buckit supports an additional optional human-readable prefix when configuring the remote target, which may facilitate operations related to diagnostics, maintenance, or disaster recovery.

Buckit recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other Buckit deployments. This tutorial includes the necessary syntax for setting this prefix.

Availability of Remote Data

Buckit tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. Buckit therefore cannot support remote storage which requires rehydration, wait periods, or manual intervention.

Buckit creates metadata for each transitioned object that identifies its location on the remote storage. Applications cannot trivially identify and access a transitioned object independent of Buckit. Availability of the transitioned data therefore depends on the same core protections that erasure coding and distributed deployment topologies provide for all objects on the Buckit deployment. Using object transition does not provide any additional business continuity or disaster recovery benefits.

Workloads that require BC/DR protections should implement Buckit Server-Side replication. Replication ensures objects remains preserved on the remote replication site, such that you can resynchronize from the remote in the event of partial or total data loss. See Resynchronization (Disaster Recovery) for more complete documentation on using replication to recover after partial or total data loss.

1) Configure User Accounts and Policies for Lifecycle Management

This step creates users and policies on the Buckit deployment for supporting lifecycle management operations. You can skip this step if the deployment already has users with the necessary permissions.

The following example uses Alpha as a placeholder alias for the Buckit deployment. Replace this value with the appropriate alias for the Buckit deployment on which you are configuring lifecycle management rules. Replace the password LongRandomSecretKey with a long, random, and secure secret key as per your organizations best practices for password generation.

cat > /tmp/LifecycleManagementAdmin.json <<'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetTier",
                "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
        },
        {
            "Action": [
                "s3:PutLifecycleConfiguration",
                "s3:GetLifecycleConfiguration"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableLifecycleRuleManagement"
        }
    ]
}
EOF
bm admin policy create Alpha LifecycleAdminPolicy /tmp/LifecycleManagementAdmin.json
bm admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey
bm admin policy attach Alpha LifecycleAdminPolicy --user=alphaLifecycleAdmin

This example assumes that the specified aliases have the necessary permissions for creating policies and users on the deployment. See User Management and Buckit Policy Based Access Control for more complete documentation on Buckit users and policies respectively.

2) Configure the Remote Storage Tier

Use the bm ilm tier add command to add a new remote storage tier:

bm ilm tier add azure TARGET TIER_NAME \
   --account-name ACCOUNT \
   --account-key KEY \
   --bucket CONTAINER \
   --endpoint ENDPOINT \
   --prefix PREFIX \
   --storage-class STORAGE_CLASS

The example above uses the following arguments:

3) Create and Apply the Transition Rule

Use the bm ilm rule add command to create a new transition rule for the bucket. The following example configures transition after the specified number of calendar days:

bm ilm rule add ALIAS/BUCKET \
--transition-tier TIERNAME \
--transition-days DAYS \
--noncurrent-transition-days NONCURRENT_DAYS
--noncurrent-transition-tier TIERNAME

The example above specifies the following arguments:

4) Verify the Transition Rule

Use the bm ilm rule ls command to review the configured transition rules:

bm ilm rule ls ALIAS/PATH --transition

• Replace ALIAS with the alias of the Buckit deployment.

• Replace PATH with the name of the bucket for which to retrieve the configured lifecycle management rules.