Rados Gateway degraded write performance after Red Hat Ceph Storage 1.3 upgrade

Environment

• Red Hat Ceph Storage 1.3
• Red Hat Enterprise Linux 7.1

Issue

• Rados Gateway degraded write performance after Red Hat Ceph Storage 1.3 upgrade
• Rados Gateway performance dip after upgrading Red Hat Ceph Storage to 1.3 from 1.2.3

Resolution

• Configure the correct pg_num value for all the rados gateway pool. Use the This content is not included.pgcalc tool for calculating the correct pg_num value. The below command can be used to set the pg_num

   #ceph osd set <pool_name> pg_num <value> 

Check the ceph -s output, to see the warning pg_num is greater than pgp_num. Use below command to set the pgp_num.

   #ceph osd set <pool_name> pgp_num <value> 

• Watch the Rados Gateway admin socket to check for blocked operations or other indications of throttled IOs as described Content from ceph.com is not included.here

• Use the This content is not included.Bucket Sharding feature of Red Hat Ceph Storage 1.3 to increase performance.

Diagnostic Steps

• Check the ceph osd dump output to see the pg_num is proper.

• Check the objecter_requests output from the Rados Gateway admin socket.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.