Added automation for the Ceph Object Gateway multi-site setup

With this enhancement, zone group host names can now be set using the Ceph Object Gateway realm bootstrap command. Set the zonegroups_hostnames by using the specification file that is provided to the ceph rgw realm bootstrap command.

New automatic application of updated SSL certificates during Ceph rgw service updates

Previously, when updating SSL certificates for Ceph Object Gateway in the service specification, the changes did not take effect until the daemons were manually restarted. This manual step hindered automation and could leave services temporarily running with outdated certificates.

New ceph orch device replace HOST DEVICE_PATH command to simplify OSD device replacement

Previously, replacing a shared DB device was tedious and error-prone. Cephadm also often redeployed OSDs too quickly after destruction, before the physical device was replaced.

Improved core dump handling in cephadm systemd units

Previously, core dumps were not generated or were truncated when services crashed, especially in hard-to-reproduce cases, resulting in the loss of valuable debugging information.

New custom log rotate configurations available for Cephadm to deploy to each host

With this enhancement, users can now set custom logrotate configurations for both the rotation of cephadm.log and daemons logs that cephadm will deploy to each host.

New support for topographical labeling on hosts

This enhancement expands cephadm’s capabilities by introducing topological key/value properties for hosts. Administrators can now group hosts by meaningful, configurable labels, enabling more efficient rolling upgrades. Instead of issuing multiple commands for each service group (for example, distinct RGW services by rack), upgrades can iterate through a list of topographical labels—streamlining multi-rack operations. Additionally, these new properties open the door for enhanced RADOS read affinity by leveraging improved CRUSH location settings.

New metric allows quick detection of Ceph daemon problems

This enhancement provides the new ceph_daemon_socket_up metric for each Ceph daemon running in the same host as the ceph exporter. The ceph_daemon_socket_up metric provides the health status of a Ceph daemon based on its ability to respond through the admin socket, where a value of 1 indicates a healthy state and 0 indicates an unhealthy state. The metric serves as a tool for quickly detecting problems in any of the main Ceph daemon.

New bucket shard count displayed

Previously, shard counts were not displayed, limiting visibility into bucket configurations.

Ceph Dashboard now supports managing Storage Classes through the UI

Previously, users could not configure or manage Storage Classes through the Dashboard. Although Life Cycle (LC) policies introduced in 8.0 allowed data tiering between Storage Classes, the UI lacked the ability to define or manage the classes themselves.

KMIP is now added to the list of KMS providers under the Objects > Configuration section of the Dashboard

Previously, the Ceph dashboard supported only KMS providers for managing encryption keys.

Ceph Dashboard now requires users to type the resource name to confirm deletion of critical resources

Previously, users could delete one or more critical resources (such as images, snapshots, subvolumes, subvolume groups, pools, hosts, OSDs, buckets, and file systems) by simply selecting a checkbox. This made accidental deletions more likely.

cephfs-mirror daemon only transfers changed blocks in a file

Previously, cephfs-mirror daemon would transfer whole files, which is inefficient for large files.

Metadata and data pool names can now be used for creating the volume

With this enhancement, the ceph fs volume create command allows users to pass metadata and data pool names to be used for creating the volume. If either is not passed or if either is a non-empty pool, the command stops.

CephFS now supports hierarchical case-insensitive or normalized directory entry naming

With this enhancement, CephFS now supports performant case-insensitive file access protocols. As a result, CephFS performance is competitive with other case-insensitive native file systems.

FSCrypt encryption is now supported within user space CephFS

With this enhancement, FSCrypt encryption is supported, allowing other software stacks to enable encryption. As a result, encryption can now be enabled and used within CephFS.

New support for retrieving the path of a subvolume snapshot

With this enhancement, users can now obtain the path of a subvolume snapshot. Get the path of a subvolume snapshot, by using the new ceph fs subvolume snapshot getpath command. NOTE: If the snapshot does not exist, the command returns an ENOENT error.

New support for disabling always-on manager modules and plugins

This enhancement allows administrators to force-disable always-on modules and plugins in the Ceph MGR. Force disabling can help prevent flooding by module commands when the corresponding Ceph service is down or degraded.

quota.max_bytes is now set in more understandable size values

Previously, the quota.max_bytes value was set in bytes, resulting in often very large size values, which were hard to set or change.

New support for TPM 2.0 for encrypted OSDs

With this enhancement, users can now enroll a Trusted Platform Module (TPM) 2.0 token during OSD preparation to store Linux Unified Key Setup (LUKS) securely. As a result, key management is now improved by leveraging hardware-backed security.

Improved stability for DB partitions

With this enhancement, users can create a dedicated DB partition, even on a colocated OSD deployment scenario. Isolating the RocksDB helps improve stability and prevents fragmentation-related issues.

Sites can now configure Ceph Object Gateway error handling for existing bucket creation

Previously, Ceph Object Gateway (RGW) returned a success response when creating a bucket that already existed in the same zone, even if no new bucket was created. This caused confusion in automated workflows.

New cloud restore support for Glacier/Tape endpoints to retrieve objects

This enhancement introduces the new cloud-glacier-s3 tier-type to extend S3 endpoint support for Glacier/Tape.

Dynamic bucket resharding now has the ability to reduce the number of shards

When a bucket undergoes a reduction in the number of objects contained within for an extended period of time, the number of shards should be reduced automatically.

New support for restoration of versioned objects transitioned to Cloud

With this enhancement, versioned objects can now be restored from the Cloud back into the Ceph Object Gateway cluster.

Creation dates are now added as part of user keys

With this enhancement, when keys are added to a user, a creation stamp is now attached to it. As a result, keys are removed in the proper order when credentials are rotated.

HeadBucket requests are now less resource intensive

Previously, all HeadBucket requests required querying all the shards to assemble statistics, which made the requests resource intensive operations.

A clientID can now be removed from an OpenID Connect provider registered with Ceph Object Gateway

Previously, a clientID could be added to an OpenID Connect provider, but removal was not supported.

Administrators can now delete bucket index entries with a missing head object

Previously, using a radosgw-admin object rm command would not remove a bucket index entry with a head object missing. Instead of removing the bucket, an error message would be emitted.

AssumeRoleWithIdentity now supports validating JWT signatures

Previously, AssumeRoleWithWebIdenity supported JSON Web Token (JWT) signature validation using only x5c.

Cloud-transitioned objects can now be restored to a selected storage class

Previously, objects transitioned to cloud were restored only to STANDARD storage class. This was a limitation and can affect data usage of the cluster.

New support for PUT bucket notifications from other tenant users

With this enhancement, there is added support for cross tenant topic management, allowing PUT bucket notifications from other tenant users. Cross tenant management includes creating, deleting, and modifying topic management.

Support for user accounts through Identity and Access Management (IAM)

User accounts through IAM was previously available as limited release. This enhancement provides full availability for new and existing customers in production environments.

Ceph now optimizes OMAP listing at the OSD level.

OMAP listing at the Ceph OSD is optimized.

PG scrub performance improved by removing unnecessary object ID repair check.

Previously, every PG scrub triggered the repair_oinfo_oid() function, which addressed rare object ID mismatches caused by a historical filesystem bug. This added overhead, even when the conditions didn’t apply.

pg-upmap-primary mappings can now be removed from the OSDmap

With this enhancement, the new ceph osd rm-pg-upmap-primary-all command is introduced. The command allows users to clear all pg-upmap-primary mappings in the OSDmap at any time.

Cluster log level verbosity for external entities can now be controlled

Previously, debug verbosity logs were sent to all external logging systems regardless of their level settings. As a result, the /var/ filesystem would rapidly fill up.

Ceph now reports BlueStore fragmentation through the health warning subsystem

Previously, Ceph only logged BlueStore fragmentation issues in low-visibility log entries, making them easy to overlook.

Advance notifications are now provided on free fragmentation disk space

Previously, when free space on the disk was significantly fragmented, the searching for free space took longer and potentially impacted performance. While this did not immediately cause problems, impact only emerged at a very late stage, free disk space was very low.

New support for 2-site stretch cluster (stretch-mode)

This enhancement enables a two-site stretch cluster deployment, allowing users to extend Ceph’s failure domain from the OSD level to the data-center or zone level. In this configuration, OSDs and Monitors can be deployed across two data sites, while a third site (monitor-only) acts as a tie-breaker for MON quorum during site failure. This architecture enhances fault tolerance by enabling automatic failover, preventing split-brain scenarios, and supporting recovery to ensure continued cluster availability and data integrity, even during a full-site outage.

Reduced fast storage requirements with RocksDB compression enabled

With this enhancement, when RocksDB compression is enabled, Ceph Object Gateway has a reduced block.db reserved size. The new reduced requirement is changed from 4% to 2.5% of reserved space. The RocksDB compression is enabled by default.

RBD now supports mirroring between default and non-default namespaces.

With this enhancement, Ceph Block Device introduces a new init-only mode for the rbd mirror pool enable command. This command provides the ability to configure a pool for mirroring and disable mirroring on the default namespace. However, mirroring can still be configured for other namespaces. This feature allows a non-default namespace in the pool to be mirrored to the default namespace in a pool of the same name in the remote cluster.

New consistency group snapshot mirroring (CGSM)

Previously, disaster recovery relied on single-image mirroring between clusters. This approach supported isolated images but did not meet the needs of applications that depend on multiple volumes. For example, in a libvirt VM with several disks, each disk serves a different role. Restoring all volumes to a consistent, same-point-in-time state was challenging.

The haproxy daemon no longer fails deployment when using the haproxy_qat_support setting in the ingress specification

Previously, the haproxy_qat_support was present but not functional in the ingress specification. This was added to allow haproxy to offload encryption operations on machines with QAT hardware, intending to improve performance. The added function did not work as intended, due to an incomplete code update. If the haproxy_qat_support setting was used, then the haproxy daemon failed to deploy.

PROMETHEUS_API_HOST gets set during Cephadm Prometheus deployment

Previously, PROMETHEUS_API_HOST would not always get set when Cephadm initially deployed Prometheus. This issue was seen most commonly when bootstrapping a cluster with --skip-monitoring-stack, then deploying Prometheus at a later time. As a result monitoring information could be unavailable.

Creating a sub-user from the dashboard now works as expected

Previously, a Python coding error prevented the dashboard and API from creating sub-users.

Editing RGW-related configurations from the dashboard is now supported

Previously, the dashboard relied on incorrect flag data to determine whether a configuration was editable, preventing users from modifying RGW-related settings that were editable via the CLI.

Invalid headers during a no longer cause a segmentation fault during journal import

Previously, the cephfs-journal-tool did not check for headers during a journal import operation. This would cause a segmentation fault.

cephfs-data-scan during disaster recovery now completes as expected

Previously, in some cases, the cephfs-data-scan ran during disaster recovery but did not create a missing directory fragment from the backtraces or create duplicate links. As a result, directories were inaccessible or crashed the MDS.

Inode invalidation operations now complete faster

Previously, an extra reference to an inode was taken in some cases that was never released. As a result, operations requiring inode invalidation were delayed until a timeout elapsed, making them very slow.

Space larger than the NFS export disk size can no longer be allocated

Previously, an empty file could be created without storage blocks allocated. These empty files could cause write operations to fail when writing to the desired file region, with a command such as fallocate.

Proxy daemon logs now update immediately

Previously, log messages from the proxy daemon were buffered by the glibc library, causing delays in log file updates. As a result, in the event of a crash, some log entries could be lost, making troubleshooting and debugging more difficult.

Async write deadlock fixed under OSD full conditions

Previously, when asynchronous writes were ongoing and the OSD became full, the client received a notification to cancel the writes. The cancellation method and the callback invoked after the write was canceled both attempted to acquire the same lock. As a result, this led to a deadlock, causing the client to hang indefinitely during an OSD full scenario.

Expanded removexattr support for CephFS virtual extended attributes

Previously, removexattr was not supported on all appropriate Ceph virtual extended attributes, resulting in attempts to remove an extended attribute failing with a "No such attribute" error.

MDS and FS IDs are now verified during health warning checks for fail commands

Previously, the MDS and FS IDs were not checked when executing the ceph mds fail and ceph fs fail commands. As a result, these commands would fail with a "permission denied" error for healthy MDS or FS instances when another instance in the cluster exhibited health warnings.

Error mapping now displays specific error message

Previously, an incorrect mapping of the error code to the user message resulted in a generic message being displayed. As a result, users did not see the specific details of the error encountered.

fscrypt now decrypts long file names

Previously, the alternate name, which holds the raw encrypted version of the file name, was not provided in all decryption cases. As a result, long file names were not being decrypted correctly, and incomplete directory entry data was produced.

Snapshot names are now stored in plain text

Previously, snapshots could be created regardless of whether the fscrypt key was present. When a snapshot was created using the mgr subvolume snapshot create command without the key, the snapshot name was not encrypted during creation. As a result, subsequent attempts to decrypt the plain text name produced unreadable output.

Multipart uploads using AWS CLI no longer cause Ceph Object Gateway to crash

Previously, during a multipart upload using AWS CLI, RGW crashed due to checksum algorithms and reporting behavior introduced in AWS S3 and AWS SDKs, specifically the new CRC64NVME checksum algorithm.

Deleted objects no longer appear in the bucket listing

Previously, a race between CompleteMultipart and AbortMultipart uploads could lead to inconsistent results. As a result, objects could appear in the bucket listing, even when they were no longer present.

Ceph Object Gateway no longer crashes during object deletion

Previously, in some cases, an uninitialized check_objv parameter variable could lead to accessing an invalid memory address in the object delete path. As a result, there was a segmentation fault.

Tail objects no longer wrongly deleted with copy-object

Previously, there was a reference count invariant on tail objects that was not maintained when an object was copied to itself. This caused the existing object was changed, rather than copied. As a result, references to tail objects were being decremented. When the refcount on tail objects dropped to 0, they were deleted during the next garbage collection (GC) cycle.

AssumeRoleWithWebIdentity operations now fails as expected when incorrect thumbprints are added

Previously, due to a boolean flag being incorrectly set in the code, the AssumeRoleWithWebIdentity operation succeeded even when an incorrect thumbprint was registered in the CreateOIDCProvider call. As a result, AssumeRoleWithWebIdentity was able to succeed when it should have failed.

Ceph Object Gateway can now delete objects when RADOS is at maximum pool capacity

Previously, when a RADOS pool was near its maximum quota, the Ceph Object Gateway was not able to delete objects.

User Put Object permissions are now recognized on copied buckets

Previously, bucket policies of a copy source bucket with access permissions for Put Object were not recognized on the copied bucket. As a result, when accessing the copied bucket, an Access Denied error was emitted.

Large queries on Parquet objects no longer emit an Out of memory error

Previously, in some cases, when a query was processed on a Parquet object, that object was read in large chunks. This caused the Ceph Object Gateway to load a larger buffer into the memory, which was too big for low-end machines. The memory would especially be affected when Ceph Object Gateway was co-located with OSD processes, which consume a large amount of memory. With the Out of memory error, the OS killed the Ceph Object Gateway process.

radosgw-admin no longer crashes by non-positive values

Previously, when running a radosgw-admin bucket reshard command, using a non-positive --num-shards value, such as a zero or a negative number, would cause radosgw-admin to crash.

Ceph Object Gateway no longer fails during signature validation

Previously, if the JSON Web Token (JWT) was not signed using the first x5c certification for signature validation, the signature validation fails.

Objects are now removed as per the lifecycle rules set when bucket versioning is suspended

Previously, due to an error in the lifecycle code, the lifecycle process did not remove the objects if the bucket versioning was in the suspended state. As a result, the objects were still seen in the bucket listing.

Multipart uploads can now add object tags

Previously, the Ceph Object Gateway S3 multipart upload object tags were not recognized when sent by the client. As a result, clients were not able to successfully apply object tags during initial object creation during a multipart upload.

STS implementation now supports encryption keys larger than 1024 bytes

Previously, Ceph Object Gateway STS implementation did not support encryption keys larger than 1024 bytes.

Bucket logging configurations no longer allow setting the same source and target buckets

Previously, there was no check in place when setting a bucket logging configuration, verifying that the source and target buckets were different.

Ceph Object Gateway no longer crashes due to mishandled kafka error messages

Previously, error conditions with the kafka message broker were not handled correctly. As a result, in some cases, Ceph Objet Gateway would crash.

ACL bucket operations now work as expected

Previously, a local variable 'uri' shadowed a member variable with the same name. As a result, a subset of bucket ACL operations would fail.

Target buckets now needs a bucket policy for users to write logs to them

Previously, no permission checks were run on the target bucket for bucket logging. As a result, any user could write logs to a target bucket, without needing specific permissions.

S3 requests no longer rejected if local is listed before external for the authentication order

Previously, S3 requests were rejected when the request is not authenticated successfully by the local authentication engine. As a result, S3 requests using OpenStack Keystone EC2 credentials failed to authenticate with Ceph Object Gateway when the authentication order had local before external

Ceph Object Gateway internal HTTP headers are no longer sent while transitioning the object to Cloud

Previously, some Ceph Object Gateway internal HTTP header values were sent to the Cloud endpoint, when transitioning the object to Cloud. As a result, some S3 cloud services did not recognize the headers and failed the transition or failed to restore the operation of the objects.

The radosgw-admin bucket logging flush command now provides works as expected

Previously, using the radosgw-admin bucket logging flush command would return the next lob object name. As a result, the user did not know the name of the log object that was flushed without listing the log bucket.

Upgrading clusters now fetches notification_v2 topics correctly

Previously, upgrading clusters upgraded bucket notifications to notification_v2. As a result, topics in notification_v2 were not retrieved as expected.

olh get now completes as expected

Previously, a 2023 fix for a versioning-related bug caused an internal code path to reference an incorrect attribute name for the object logical head (OLH). As a result, an error would emit when running the radosgw-admin olh get command.

Swift container listings now report object last modified time

Previously, the Ceph Object Gateway Swift container listing implementation was missing the logic to send the last_modified JSON field. As a result, Swift container listings did not report the last modified time of objects.

Ceph Object Gateway now recognizes additional checksums from their checksum-type specific headers and trailers

Previously, the aws-sdk-go-v2 checksum behavior differed from other SDKs, as it did not send either x-amz-checksum-algorithm or x-amz-sdk-checksum and never included x-amz-decoded-content-length, despite AWS documentation requiring it. As a result, additional checksums were not recognized when sent, and some AWS-chunked requests failed an assertion check for decoded content length with an InvalidArgument error.

Shadow users for the AssumeRoleWithWebIdentity call are now created within the oidc namespace

Previously, an incorrect method was used to load the bucket stats, which caused the shadow users for AssumeRoleWithWebIdentity call to not be created within the oidc namespace. As a result, users were not able to differentiate between the shadow users and local rgw users.

Replicating metadata from earlier versions of Red Hat Ceph Storage no longer renders user access keys as “inactive”

Previously, when a secondary zone running Red Hat Ceph Storage 8.0 replicated user metadata from a pre-8.0 metadata master zone, the access keys of those users were erroneously marked as "inactive". Inactive keys cannot be used to authenticate requests, so those users are denied access to the secondary zone.

Invalid URL-encoded text from the client no longer creates errors

Previously, the system improperly handled scenarios where URL decoding resulted in an empty key.name. The empty key.name due to invalid URL-encoded text from the client. As a result, an assertion error during the copy operation would occur, and sometimes led to a crash later.

Network error code is now mapped correctly

Previously, when one or some of the Ceph Object Gateways in a target zone were down, the HTTP client in the Ceph Object Gateway in the source zone did not map the network connection error code correctly internally. As a result, the client kept attempting to connect to a downed Ceph Object Gateway instead of falling back to other active ones.

sync error trim now runs as expected with optional --shard-id input

Previously, the sync error trim command did not mark the --shard-id option as optional.

Objects restored from Cloud/Tape now synchronize correctly to remote locations

Previously, objects restored from Cloud/Tape retained their original mtime, making it insufficient for multisite sync checks. As a result, these restored objects were not synchronized to remote locations.

Sync rate now works as expected

Previously, in some cases, an incorrect internal error return caused sync operations to run slower than expected.

rgw daemons no longer crash due to stack overflows

Previously, large clusters of over 10,000 daemons with a stack-based allocation of variable-length arrays caused a stack overflow. As a result, the Ceph Object Gateway daemons crashed.

Pool removals now remove pg_upmap_primary mappings from the OSDMap

Previously, deleting pools did not remove pg_upmap_primary mappings from the OSDMap. As a result, pg_upmap_primary mappings were seen but could not be removed, since the pool, and pgid no longer existed.

Destroyed OSDs are no longer listed by the ceph node ls command

Previously, destroyed OSDs were listed without any indication of their status, leading to user confusion and causing cephadm to incorrectly report them as stray.

AVX512 support for the ISA-L erasure code plugin is now enabled

Previously, due to an issue in the build scripts, the plugin did not take advantage of the AVX512 instruction set—even on CPUs that supported it—resulting in reduced performance.

Multiple OSDs crashing while replaying bluefs -- ceph_assert(delta.offset == fnode.allocated)

Previously, a fix was implemented to prevent RocksDB’s SST files from expanding, but it contained a bug. As a result, the BlueFS log became corrupted, causing an error that prevented OSD bootup, even though it could be ignored.

BlueFS log no longer gets corrupted due to race conditions

Previously, a rare condition between truncate and unlink operations in BlueFS caused the log to reference deleted files. This corrupted the BlueFS log, triggering an assertion failure during OSD startup and resulting in a crash loop.

New Ceph Management gateway and the OAuth2 Proxy service for unified access and high availability

With this enhancement, the Ceph Dashboard introduces the Ceph Management gateway (mgmt-gateway) and the OAuth2 Proxy service (oauth2-proxy). With the Ceph Management gateway (mgmt-gateway) and the OAuth2 Proxy (oauth2-proxy) in place, nginx automatically directs the user through the oauth2-proxy to the configured Identity Provider (IdP), when single sign-on (SSO) is configured.

New Cephadm certificate lifecycle management for improved Ceph cluster security

With this enhancement, Cephadm now has certificate lifecycle management in the certmgr subsystem. This feature provides a unified mechanism to provision, rotate, and apply TLS certificates for Ceph services, supporting both user-provided and automatically generated cephadm-signed certificates. As part of this feature, certmgr periodically checks the status of all certificates managed by Cephadm and issues health warnings for any that are nearing expiration, misconfigured, or invalid. This improves Ceph cluster security and simplifies certificate management through automation and proactive alerts.

New OAuth2 SSO

OAuth2 SSO uses the oauth2-proxy service to work with the Ceph Management gateway (mgmt-gateway), providing unified access and improved user experience.

Bucket logging support for Ceph Object Gateway with bug fixes and enhancements

Bucket logging was introduced in Red Hat Ceph Storage 8.0. Bucket logging provides a mechanism for logging all access to a bucket. The log data can be used to monitor bucket activity, detect unauthorized access, get insights into the bucket usage and use the logs as a journal for bucket changes. The log records are stored in objects in a separate bucket and can be analyzed later. Logging configuration is done at the bucket level and can be enabled or disabled at any time. The log bucket can accumulate logs from multiple buckets. The configured prefix may be used to distinguish between logs from different buckets.

Restore objects transitioned to remote cloud endpoint back into Ceph Object gateway using the cloud-restore feature

With this release, the cloud-restore feature is implemented. This feature allows users to restore objects transitioned to remote cloud endpoint back into Ceph Object gateway, using either S3 restore-object API or by rehydrating using read-through options.