RGW services crashing during multipart upload in `RGWPutObj_Cksum` function.

Environment

• Red Hat Ceph Storage (RHCS) 8.0

Issue

• RGW services crashing during multipart upload with AWS CLI version 2.25 and above.

• The crashes contain the following crash signature:

  ceph version 19.2.0-84.el9cp (173d03a90c2ddc0bbfa9f23c3239ba19fe9e7c11) squid (stable)
  1: /lib64/libc.so.6(+0x3e730) [0x7f5fb4e44730]
  2: (rgw::putobj::RGWPutObj_Cksum::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x38) [0x55bdc8b16598]
  

Resolution

Until you are able to upgrade to the fixed versions you can use one of two following workarounds.

1. Use a lower version of the AWS CLI or AWS SDK. Versions aws-cli/1.36.0 and aws-cli/2.24.12 or below should work.
2. No crashes have been observed when request_checksum_calculation and response_checksum_validation configuration options are set to when_required. See the Artifacts section for the documentation about how to configure those parameters.

Root Cause

The issue is related to new checksum algorithms, new checksum construction and reporting behavior introduced in AWS S3 and AWS SDKs. A new CRC64NVME checksum algorithm was introduced, and simultaneously made the default behavior of some AWS SDKs, including at least awscliv2 as of 2.25.

Artifacts

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.