Update nova.conf on controllers to allow oversubscription of RAM and CPU
Environment
Red Hat Enterprise Linux OpenStack Platform 7.0
Red Hat OpenStack Platform 8
Issue
What is the best way to configure oversubscription for RAM in Red Hat OpenStack Platform and how can one configure it?
Resolution
The following instructions specifically talk about RAM allocation and RAM filters. The same would obviously apply to disk and CPU. Please test modifications against a lab or staging environment before applying them in production.
Making a temporary change
Please note that the change which is suggested in this section is temporaty. OpenStack Director will overwrite it when the openstack overcloud deploy is run again. E.g., during a scale-out of the overcloud. The following section will explain how to make this change permanently.
If one wants to increase the allocation ratio for RAM (e.g. due to a shortage), one can increase the ram_allocation_ratio.
First, verify that the cluster is healthy
[root@overcloud-controller-0 ~]# pcs status | grep -i stopp
[root@overcloud-controller-0 ~]#
nova.conf on the controller nodes contains the following default value for RAM allocation
# Virtual ram to physical ram allocation ratio which affects
# all ram filters. This configuration specifies a global ratio
# for RamFilter. For AggregateRamFilter, it will fall back to
# this configuration value if no per-aggregate setting found.
# (floating point value)
#ram_allocation_ratio=1.5
ram_allocation_ratio=1.0
Make the appropriate change to the above on all 3 controllers, and then restart nova scheduler.
[root@overcloud-controller-0 ~]# pcs resource restart openstack-nova-scheduler
Warning: using openstack-nova-scheduler-clone... (if a resource is a clone or master/slave one must use the clone or master/slave name
openstack-nova-scheduler-clone successfully restarted
Then, verify that all resources are up and running
[root@overcloud-controller-0 ~]# pcs status | grep -i stopp
[root@overcloud-controller-0 ~]#
Alternatively, one may disable the RamFilter in the scheduler_default_filters parameter. However, this is not recommended
# Which filter class names to use for filtering hosts when not
# specified in the request. (list value)
#scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
The above can be changed to
# Which filter class names to use for filtering hosts when not
# specified in the request. (list value)
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
Making a permanent change via Director
One needs to make the change to Director's templates in order to make it permanent. Once the change is applied to Director's templates, it is advisable to rerun a deployment with openstack overcloud deploy so that Director applies the change.
Please read How to customize Puppet Configuration Data with hiera in OSP Director to get a thorough understanding of how the configuration of the overcloud can be modified with puppet.
In particular, in order to apply a change to nova.conf on the compute nodes for the ram_allocation_ratio (the commented sections contain the parameters for cpu and disk allocation as well)
parameter_defaults:
controllerExtraConfig:
nova::scheduler::filter::ram_allocation_ratio: 2.0
# nova::scheduler::filter::cpu_allocation_ratio: 20.0
# nova::scheduler::filter::disk_allocation_ratio: 10.0
Run openstack overcloud deploy to update the stack - this needs to be done in the same way as for previous deployments.
After the stack update, verify /etc/nova/nova.conf
[root@overcloud-controller-0 ~]# grep allocation_ratio /etc/nova/nova.conf
#cpu_allocation_ratio=16.0
cpu_allocation_ratio=20
#disk_allocation_ratio=1.0
disk_allocation_ratio=10
#ram_allocation_ratio=1.5
ram_allocation_ratio=2
Verification
The currently is no simple way to verify that nova accepted these values as provided.
Spawn more instances than current allocation ratio and verify hypervisor
[stack@undercloud ~]$ nova hypervisor-show 2 | egrep 'vcpu|memory_mb|local_gb'
| local_gb | 59 |
| local_gb_used | 8 |
| memory_mb | 3791 |
| memory_mb_used | 4608 |
| vcpus | 2 |
| vcpus_used | 8
Verify nova scheduler logs
Set debug=True in `/etc/nova/nova.conf on all 3 controllers
[root@overcloud-controller-1 ~]# egrep '^debug' /etc/nova/nova.conf
debug=True
Restart nova-scheduler
[root@overcloud-controller-0 ~]# pcs resource restart openstack-nova-scheduler
Warning: using openstack-nova-scheduler-clone... (if a resource is a clone or master/slave you must use the clone or master/slave name
openstack-nova-scheduler-clone successfully restarted
Verify /var/log/nova/nova-scheduler.log
E.g. with RamFilter enabled
2016-08-10 23:08:08.111 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Starting with 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packa
ges/nova/filters.py:70
2016-08-10 23:08:08.112 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter RetryFilter returned 1 host(s) get_filtered_objects /usr/lib/python
2.7/site-packages/nova/filters.py:84
2016-08-10 23:08:08.112 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter AvailabilityZoneFilter returned 1 host(s) get_filtered_objects /usr
/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:08:08.112 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5
d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter RamFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:08:08.113 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ComputeFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:08:08.113 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ComputeCapabilitiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:08:08.113 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ImagePropertiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:08:08.113 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ServerGroupAntiAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:08:08.114 10355 DEBUG nova.filters [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ServerGroupAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:08:08.114 10355 DEBUG nova.scheduler.filter_scheduler [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filtered [(overcloud-compute-0.localdomain, overcloud-compute-0.localdomain) ram:-305 disk:52224 io_ops:2 instances:7] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:143
2016-08-10 23:08:08.115 10355 DEBUG nova.scheduler.filter_scheduler [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Weighed [WeighedHost [host: (overcloud-compute-0.localdomain, overcloud-compute-0.localdomain) ram:-305 disk:52224 io_ops:2 instances:7, weight: -1.0]] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:148
2016-08-10 23:08:08.115 10355 DEBUG nova.scheduler.filter_scheduler [req-6d1b790a-0180-4045-9ca1-36f49d5b5a96 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Selected host: WeighedHost [host: (overcloud-compute-0.localdomain, overcloud-compute-0.localdomain) ram:-305 disk:52224 io_ops:2 instances:7, weight: -1.0] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:158
With RamFilter disabled
2016-08-10 23:14:58.817 19765 DEBUG nova.filters [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Starting with 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packa
ges/nova/filters.py:70
2016-08-10 23:14:58.818 19765 DEBUG nova.filters [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter RetryFilter returned 1 host(s) get_filtered_objects /usr/lib/python
2.7/site-packages/nova/filters.py:84
2016-08-10 23:14:58.818 19765 DEBUG nova.filters [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter AvailabilityZoneFilter returned 1 host(s) get_filtered_objects /usr
/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:14:58.819 19765 DEBUG nova.filters [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ComputeFilter returned 1 host(s) get_filtered_objects /usr/lib/pyth
on2.7/site-packages/nova/filters.py:84
2016-08-10 23:14:58.819 19765 DEBUG nova.filters [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ComputeCapabilitiesFilter returned 1 host(s) get_filtered_objects /
usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:14:58.819 19765 DEBUG nova.filters [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ImagePropertiesFilter returned 1 host(s) get_filtered_objects /usr/
lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:14:58.819 19765 DEBUG nova.filters [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ServerGroupAntiAffinityFilter returned 1 host(s) get_filtered_objec
ts /usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:14:58.820 19765 DEBUG nova.filters [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filter ServerGroupAffinityFilter returned 1 host(s) get_filtered_objects /
usr/lib/python2.7/site-packages/nova/filters.py:84
2016-08-10 23:14:58.820 19765 DEBUG nova.scheduler.filter_scheduler [req-6032bd2e-897a-4856-90bc-c078c1f5af84 1f673e11af1f4aeebc340e647f5d4b72 bc383a8c9d6e443c9383e2cebe9f720a - - -] Filtered [(overcloud-compute-0.localdomain, overcloud-compute-0.localdomain) ram:-817 disk:49152 io_ops:0 instances:8] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:143
Once this is confirmed working, set debug=False if it used to be False and restart the scheduler again
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.