Ceph - How to grant access for multiple S3 users to access a single bucket

Solution Verified - Updated 6 Aug 2025

Environment

Red Hat Ceph Storage (RHCS)

Issue

Allowing multiple users to access a single RGW bucket for both reads and writes using s3cmd

Resolution

Below are the steps to create a new bucket and two users that will have access to the new bucket
Created a new bucket

[jquinn@jquinn s3]$ s3cmd -c /home/jquinn/s3/s3cfg mb s3://joe-bucket
Bucket 's3://joe-bucket/' created
[jquinn@jquinn s3]$

After creating bucket, joe is listed as the owner and is the only one in the ACL list

[jquinn@jquinn s3]$ s3cmd -c /home/jquinn/s3/s3cfg info s3://joe-bucket 
s3://joe-bucket/ (bucket):
   Location:  us-east-1
   Payer:     BucketOwner
   Expiration Rule: none
   policy:    <?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>joe-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>s3cfg</Key><LastModified>2017-03-09T14:17:20.370Z</LastModified><ETag>&quot;1b5136c3a724ae9a0ef97c701e1f3695&quot;</ETag><Size>2042</Size><StorageClass>STANDARD</StorageClass><Owner><ID>joe</ID><DisplayName>joe</DisplayName></Owner></Contents></ListBucketResult>
   cors:      none
   ACL:       joe: FULL_CONTROL
[jquinn@jquinn s3]$

Created a user tom, and provided him with "full" access. I did not assign anything specific to the bucket yet

[root@dhcp2-224 ~]# radosgw-admin user create --uid=tom --display-name="tom" --access=full
{
    "user_id": "tom",
    "display_name": "tom",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "tom",
            "access_key": "3FDCUXMIMZNAD923RV63",
            "secret_key": "7JJ8u3bbTM14AyvKE3kWq29Fju1ZBRnfUX9lM3Tn"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

[root@dhcp2-224 ~]#

The user tom is unable to access joe-bucket

[jquinn@jquinn tom]$ s3cmd -c /home/jquinn/s3/tom/s3cfg ls s3://joe-bucket
ERROR: Access to bucket 'joe-bucket' was denied
ERROR: S3 error: 403 (AccessDenied)
[jquinn@jquinn tom]$

I grant tom access to joe-bucket with the permission of "all"

[jquinn@jquinn s3]$ s3cmd -c /home/jquinn/s3/s3cfg setacl s3://joe-bucket --acl-grant=all:tom 
s3://joe-bucket/: ACL updated

Tom is now listed in the ACL list for the bucket with full_control

[jquinn@jquinn s3]$ s3cmd -c /home/jquinn/s3/s3cfg info s3://joe-bucket 
s3://joe-bucket/ (bucket):
   Location:  us-east-1
   Payer:     BucketOwner
   Expiration Rule: none
   policy:    <?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>joe-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>s3cfg</Key><LastModified>2017-03-09T14:17:20.370Z</LastModified><ETag>&quot;1b5136c3a724ae9a0ef97c701e1f3695&quot;</ETag><Size>2042</Size><StorageClass>STANDARD</StorageClass><Owner><ID>joe</ID><DisplayName>joe</DisplayName></Owner></Contents></ListBucketResult>
   cors:      none
   ACL:       joe: FULL_CONTROL
   ACL:       tom: FULL_CONTROL
[jquinn@jquinn s3]$

Tom is now able to browse joe-bucket

[jquinn@jquinn tom]$ s3cmd -c /home/jquinn/s3/tom/s3cfg ls s3://joe-bucket
2017-03-09 14:17      2042   s3://joe-bucket/s3cfg
[jquinn@jquinn tom]$

Tom is now able to upload files to the bucket

[jquinn@jquinn tom]$ s3cmd -c /home/jquinn/s3/tom/s3cfg put sosreport-* s3://joe-bucket
upload: 'sosreport-MAlam.01774732-20170118153312.tar.xz' -> 's3://joe-bucket/sosreport-xxxxx.01774732-20170118153312.tar.xz'  [1 of 3]
 321926600 of 321926600   100% in  506s   620.26 kB/s  done
upload: 'sosreport-MYOPTPDN01.01799553-20170226010210.tar.xz' -> 's3://joe-bucket/sosreport-xxxxxxxx.01799553-20170226010210.tar.xz'  [2 of 3]
 175440276 of 175440276   100% in  331s   516.40 kB/s  done
upload: 'sosreport-badri-20170127212410.tar.xz' -> 's3://joe-bucket/sosreport-xxxxxxx-20170127212410.tar.xz'  [3 of 3]
 190628436 of 190628436   100% in  301s   616.54 kB/s  done
[jquinn@jquinn tom]$ 
[jquinn@jquinn tom]$ s3cmd -c /home/jquinn/s3/tom/s3cfg ls s3://joe-bucket
2017-03-09 14:17      2042   s3://joe-bucket/s3cfg
2017-03-09 15:48 321926600   s3://joe-bucket/sosreport-xxxxx.01774732-20170118153312.tar.xz
2017-03-09 15:54 175440276   s3://joe-bucket/sosreport-xxxxxx.01799553-20170226010210.tar.xz
2017-03-09 15:59 190628436   s3://joe-bucket/sosreport-xxxxx-20170127212410.tar.xz
[jquinn@jquinn tom]$

Tom is able to pull his own files, but not ones written from Joe, same for Joe pulling files

jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/tom/s3cfg get s3://joe-bucket/*
download: 's3://joe-bucket/s3cfg' -> './s3cfg'  [1 of 4]
ERROR: S3 error: 403 (Forbidden)
[jquinn@jquinn get]$ 

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/tom/s3cfg get s3://joe-bucket/sos*
download: 's3://joe-bucket/sosreport-xxxxx.01774732-20170118153312.tar.xz' -> './sosreport-xxxxx.01774732-20170118153312.tar.xz'  [1 of 3]
   1310720 of 321926600     0% in    7s   171.30 kB/s

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/s3cfg get s3://joe-bucket/*
download: 's3://joe-bucket/s3cfg' -> './s3cfg'  [1 of 4]
 2042 of 2042   100% in    0s     2.47 kB/s  done
download: 's3://joe-bucket/sosreport-xxxxx.01774732-20170118153312.tar.xz' -> './sosreport-xxxxx.01774732-20170118153312.tar.xz'  [2 of 4]
ERROR: S3 error: 403 (Forbidden)
[jquinn@jquinn get]$

Using user Joe, I set grants again recursively for tom, and I can now pull all the files, including the one written from Joe. access denied message because it couldn't update the files owned by tom

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/s3cfg setacl s3://joe-bucket --acl-grant=all:tom --recursive
s3://joe-bucket/s3cfg: ACL updated
ERROR: S3 error: 403 (AccessDenied)
[jquinn@jquinn get]$

Joe can still only get his own files

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/s3cfg get s3://joe-bucket/*
download: 's3://joe-bucket/s3cfg' -> './s3cfg'  [1 of 4]
 2042 of 2042   100% in    0s     2.89 kB/s  done
download: 's3://joe-bucket/sosreport-xxxxxx.01774732-20170118153312.tar.xz' -> './sosreport-xxxxx.01774732-20170118153312.tar.xz'  [2 of 4]
ERROR: S3 error: 403 (Forbidden)
[jquinn@jquinn get]$

Set acl using tom for user joe, it updated on all files

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/tom/s3cfg setacl s3://joe-bucket --acl-grant=all:joe --recursive
s3://joe-bucket/s3cfg: ACL updated
s3://joe-bucket/sosreport-xxxx.01774732-20170118153312.tar.xz: ACL updated
s3://joe-bucket/sosreport-xxxxxx.01799553-20170226010210.tar.xz: ACL updated
s3://joe-bucket/sosreport-xxxxx-20170127212410.tar.xz: ACL updated

Tom is able to read all files and write to the bucket

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/tom/s3cfg get s3://joe-bucket/*
download: 's3://joe-bucket/s3cfg' -> './s3cfg'  [1 of 4]
 2042 of 2042   100% in    0s     2.79 kB/s  done
download: 's3://joe-bucket/sosreport-xxxx.01774732-20170118153312.tar.xz' -> './sosreport-xxxxx.01774732-20170118153312.tar.xz'  [2 of 4]
 321926600 of 321926600   100% in 1519s   206.84 kB/s  done
download: 's3://joe-bucket/sosreport-xxxxxx.01799553-20170226010210.tar.xz' -> './sosreport-xxxxx.01799553-20170226010210.tar.xz'  [3 of 4]
 175440276 of 175440276   100% in  961s   178.26 kB/s  done
download: 's3://joe-bucket/sosreport-xxxx-20170127212410.tar.xz' -> './sosreport-xxxx-20170127212410.tar.xz'  [4 of 4]
 190628436 of 190628436   100% in  998s   186.48 kB/s  done
[jquinn@jquinn get]$

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/tom/s3cfg put ~/yad_status s3://joe-bucket/
upload: '/home/jquinn/yad_status' -> 's3://joe-bucket/yad_status'  [1 of 1]
 91 of 91   100% in    1s    75.01 B/s  done
[jquinn@jquinn get]$

Joe is able to read all files and write to the bucket

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/s3cfg get s3://joe-bucket/*
download: 's3://joe-bucket/s3cfg' -> './s3cfg'  [1 of 4]
 2042 of 2042   100% in    0s     2.44 kB/s  done
download: 's3://joe-bucket/sosreport-xxxxxx.01774732-20170118153312.tar.xz' -> './sosrepor-xxxxxx.01774732-20170118153312.tar.xz'  [2 of 4]
 321926600 of 321926600   100% in 2768s   113.56 kB/s  done
download: 's3://joe-bucket/sosreport-xxxxxx.01799553-20170226010210.tar.xz' -> './sosreport-xxxxxx.01799553-20170226010210.tar.xz'  [3 of 4]
 175440276 of 175440276   100% in  999s   171.43 kB/s  done
download: 's3://joe-bucket/sosreport-xxxxx-20170127212410.tar.xz' -> './sosreport-xxxxx-20170127212410.tar.xz'  [4 of 4]
 190628436 of 190628436   100% in  821s   226.56 kB/s  done
[jquinn@jquinn get]$

[jquinn@jquinn get]$ s3cmd -c /home/jquinn/s3/s3cfg put ~/setuptools-7.0.tar.gz s3://joe-bucket/
upload: '/home/jquinn/setuptools-7.0.tar.gz' -> 's3://joe-bucket/setuptools-7.0.tar.gz'  [1 of 1]
 793737 of 793737   100% in    5s   129.41 kB/s  done
[jquinn@jquinn get]$

radosgw-admin bucket stats info for bucket below after all changes. I did not make any changes to the bucket with radosgw-admin

  {
        "bucket": "joe-bucket",
        "pool": "default.rgw.buckets.data",
        "index_pool": "default.rgw.buckets.index",
        "id": "22eabf3c-e068-4901-847e-a7d13078f936.288926.1",
        "marker": "22eabf3c-e068-4901-847e-a7d13078f936.288926.1",
        "owner": "joe",
        "ver": "0#19",
        "master_ver": "0#0",
        "mtime": "2017-03-09 21:07:21.672513",
        "max_marker": "0#",
        "usage": {
            "rgw.main": {
                "size_kb": 671873,
                "size_kb_actual": 671884,
                "num_objects": 4
            }
        },
        "bucket_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        }
    }

Radosgw-admin bucket info for Joe and Tom after all changes. I did not make any changes to the user with radosgw-admin after creation

[root@dhcp2-224 ~]# radosgw-admin user info --uid=joe
{
    "user_id": "joe",
    "display_name": "joe",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "joe",
            "access_key": "J1FFWK7WFFF9PLS4U88J",
            "secret_key": "37IJiZfim0q29FIfCB87AlVFVnWUJZjBZq273HqY"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

[root@dhcp2-224 ~]# radosgw-admin user info --uid=tom
{
    "user_id": "tom",
    "display_name": "tom",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "tom",
            "access_key": "3FDCUXMIMZNAD923RV63",
            "secret_key": "7JJ8u3bbTM14AyvKE3kWq29Fju1ZBRnfUX9lM3Tn"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

[root@dhcp2-224 ~]#

SBR

Ceph

Product(s)

Red Hat Ceph Storage

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.