Requirements to Set Up Bucket Replication

Bucket replication uses rules to synchronize the contents of a bucket on one Buckit deployment to a bucket on a remote Buckit deployment.

Replication can be done in any of the following ways:

Active-Passive Eligible objects replicate from the source bucket to the remote bucket. Any changes on the remote bucket do not replicate back.
Active-Active Changes to eligible objects of either bucket replicate to the other bucket in a two-way direction.
Multi-Site Active-Active Changes to eligible objects on any bucket set up for bucket replication replicate to all of the other buckets.

Ensure you meet the following prerequisites before you set up any of these replication configurations.

Permissions Required for Setting Up Bucket Replication

Bucket replication requires specific permissions on the source and destination deployments to configure and enable replication rules.

Replication Admin

The following policy provides permissions for configuring and enabling replication on a deployment.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetBucketTarget",
                "admin:GetBucketTarget",
                "admin:ListBatchJobs",
                "admin:DescribeBatchJob",
                "admin:StartBatchJob",
                "admin:CancelBatchJob"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteBucketConfiguration"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:PutReplicationConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationRuleConfiguration"
        }
    ]
}

The "EnableRemoteBucketConfiguration" statement grants permission for creating a remote target for supporting replication.
The "EnableReplicationRuleConfiguration" statement grants permission for creating replication rules on a bucket. The "arn:aws:s3:::* resource applies the replication permissions to any bucket on the source deployment. You can restrict the user policy to specific buckets as-needed.

The following code creates a Buckit-managed user with the necessary policy. Replace the TARGET with the alias of the Buckit deployment on which you are configuring replication:

cat > /tmp/ReplicationAdminPolicy.json <<'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetBucketTarget",
                "admin:GetBucketTarget",
                "admin:ListBatchJobs",
                "admin:DescribeBatchJob",
                "admin:StartBatchJob",
                "admin:CancelBatchJob"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteBucketConfiguration"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:PutReplicationConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationRuleConfiguration"
        }
    ]
}
EOF
bm admin policy create TARGET ReplicationAdminPolicy /tmp/ReplicationAdminPolicy.json
bm admin user add TARGET ReplicationAdmin LongRandomSecretKey
bm admin policy attach TARGET ReplicationAdminPolicy --user=ReplicationAdmin

Buckit deployments configured for Active Directory/LDAP or OpenID Connect user management should instead create a dedicated access keys for bucket replication.

Replication Remote User

The following policy provides permissions for enabling synchronization of replicated data into the deployment.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationOnBucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ReplicateTags",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:PutObject",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectLegalHold",
                "s3:DeleteObject",
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicatingDataIntoBucket"
        }
    ]
}

The "EnableReplicationOnBucket" statement grants permission for a remote target to retrieve bucket-level configuration for supporting replication operations on all buckets in the Buckit deployment. To restrict the policy to specific buckets, specify those buckets as an element in the Resource array similar to "arn:aws:s3:::bucketName".
The "EnableReplicatingDataIntoBucket" statement grants permission for a remote target to synchronize data into any bucket in the Buckit deployment. To restrict the policy to specific buckets, specify those buckets as an element in the Resource array similar to "arn:aws:s3:::bucketName/*".

The following code creates a Buckit-managed user with the necessary policy. Replace TARGET with the alias of the Buckit deployment on which you are configuring replication:

cat > /tmp/ReplicationRemoteUserPolicy.json <<'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationOnBucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ReplicateTags",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:PutObject",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectLegalHold",
                "s3:DeleteObject",
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicatingDataIntoBucket"
        }
    ]
}
EOF
bm admin policy create TARGET ReplicationRemoteUserPolicy /tmp/ReplicationRemoteUserPolicy.json
bm admin user add TARGET ReplicationRemoteUser LongRandomSecretKey
bm admin policy attach TARGET ReplicationRemoteUserPolicy --user=ReplicationRemoteUser

Buckit deployments configured for Active Directory/LDAP or OpenID Connect user management should instead create a dedicated access keys for bucket replication.

See bm admin user, bm admin user svcacct, and bm admin policy for more complete documentation on adding users, access keys, and policies to a Buckit deployment.

Matching Object Encryption Settings for Bucket Replication

Buckit supports replication of objects encrypted using SSE-KMS and SSE-S3:

For objects encrypted using SSE-KMS, Buckit requires that the target bucket support SSE-KMS encryption of objects using the same key names used to encrypt objects on the source bucket.
For objects encrypted using SSE-S3, Buckit requires that the target bucket also support SSE-S3 encryption of objects regardless of key name.

As part of the replication process, Buckit decrypts the object on the source bucket and transmits the unencrypted object over the network. The destination Buckit deployment then re-encrypts the object using the encryption settings from the target. Buckit therefore strongly recommends enabling TLS on both source and destination deployments to ensure the safety of objects during transmission.

Buckit does not support replicating client-side encrypted objects (SSE-C).

Bucket Replication Requires Buckit Deployments

Buckit server-side replication only works between Buckit deployments. Both the source and destination deployments must run Buckit Server with matching versions.

To configure replication between arbitrary S3-compatible services, use bm mirror.

Replication Requires Versioning

Buckit relies on the immutability protections provided by versioning to support replication and resynchronization.

Use bm version info to validate the versioning status of both the source and remote buckets. Use the bm version enable command to enable versioning as necessary.

If you exclude a prefix or folder from versioning within the source bucket, Buckit cannot replicate objects within that folder or prefix.

Matching Object Locking State With Bucket Replication

Buckit supports replicating objects held under WORM Locking. Both replication buckets must have object locking enabled for Buckit to replicate the locked object. For active-active configuration, Buckit recommends using the same retention rules on both buckets to ensure consistent behavior across sites.

You must enable object locking during bucket creation as per S3 behavior. You can then configure object retention rules at any time. Configure the necessary rules on the unhealthy target bucket prior to beginning this procedure.