RHCS - osd_memory_target what it is and how ceph-ansible calculates the value

Solution Verified - Updated

Environment

  • Red Hat Ceph Storage 3.2 and newer
  • ceph-ansible-3.2 and above

Issue

  • osd_memory_target what it is?
  • How ceph-ansible calculates the value ?

Resolution

osd_memory_target

Description:	When tcmalloc is available and cache autotuning is enabled, try to keep this many bytes mapped in memory. Note: This may not exactly match the RSS memory usage of the process. While the total amount of heap memory mapped by the process should generally stay close to this target, there is no guarantee that the kernel will actually reclaim memory that has been unmapped. During initial developement, it was found that some kernels result in the OSD’s RSS Memory exceeding the mapped memory by up to 20%. It is hypothesised however, that the kernel generally may be more aggressive about reclaiming unmapped memory when there is a high amount of memory pressure. Your mileage may vary.

Type:	Unsigned Integer
Requered:	Yes
Default:	4294967296

ceph-ansible defaults is set in following file, and is used when nothing is defined in all.yml:

./roles/ceph-defaults/defaults/main.yml:357:osd_memory_target: 4294967296

To ensure the Ceph will get optimal osd_memory_target set on each node, the ceph-ansible is running script to find that value when building the ceph.conf.
The ceph.conf building script calculates the final osd_memory_target per node based on number of RAM and OSDs on the node - defined by number of devices in ceph-ansible configuration files, either osds.yml, inventory file or host_vars file for specific node. If the calculated value is lower then default 4GB , default is set anyway:

./roles/ceph-config/templates/ceph.conf.j2
...
[osd]
{% if is_hci and _num_osds > 0 %}
{# hci_safety_factor is the safety factor for HCI deployments #}
{% if ansible_memtotal_mb * 1048576 * hci_safety_factor / _num_osds > osd_memory_target %}
{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * hci_safety_factor / _num_osds) | int %}
{% endif %}
{% elif _num_osds > 0 %}
{# non_hci_safety_factor is the safety factor for dedicated nodes #}
{% if ansible_memtotal_mb * 1048576 * non_hci_safety_factor / _num_osds > osd_memory_target %} #<--------- if the (GB_RAM/OSDs > 4GB) use calculated value, otherwise default 4GB 
{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * non_hci_safety_factor / _num_osds) | int %}# <---------
{% endif %}
{% endif %}
osd memory target = {{ _osd_memory_target | default(osd_memory_target) }} <---------
{% endif %}
{% endif %}
...

In all.yml, you can define your own value for osd_memory_target as the default value:

osd_memory_target: 3221225472

Part of ceph.conf building script is looking if the calculated value is bigger then the osd_memory_target one:

{% if ansible_memtotal_mb * 1048576 * non_hci_safety_factor / _num_osds > osd_memory_target %} 
<--------- if the (GB_RAM/OSDs > 4GB) use calculated value, otherwise default 4GB

If this is true, the ceph-asnible will set the calculated value, lets say 256GB RAM, 10 OSDs:
256 × 1073741824 × 0.7 ÷ 10 = 19241453486 (18 GB)
but, if the calculated value is less then osd_memory_target=3GB(or 4GB in case default) this value is used as final "osd memory target"

{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * non_hci_safety_factor / _num_osds) | int %} <---------
{% endif %}
{% endif %}
osd memory target = {{ _osd_memory_target | default(osd_memory_target) }} <---------

In case you have nodes with different number of OSDs and RAM, you may end up with different "osd memory target" per node in ceph.conf,
but with lowest value "osd memory target" set equal to "osd_memory_target: 3221225472" (or 4GB in case of default) set in all.yml

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.