Upgrade to Red Hat Satellite 6.10 fails at db:seed step with error "ActiveRecord::RecordInvalid: Validation failed"

Solution Verified - Updated 13 Jun 2024

Environment

Red Hat Satellite 6.10
- being upgraded from 6.9

Issue

While upgrading to Red Hat Satellite 6.10, It fails at the following step.

2021-12-28 03:33:48 [NOTICE] [configure] 1250 configuration steps out of 2203 steps complete.
2021-12-28 03:34:08 [NOTICE] [configure] 1500 configuration steps out of 2209 steps complete.
2021-12-28 03:37:22 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-12-28 03:37:22 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2021-12-28 03:37:22 [NOTICE] [configure] 1750 configuration steps out of 2209 steps complete.
2021-12-28 03:37:23 [NOTICE] [configure] 2000 configuration steps out of 3013 steps complete.

Running the foreman-rake db:seed --trace command manually would result in reporting the following types of error messages.

   ActiveRecord::RecordInvalid: Validation failed: Domains expecting domains used by hosts or inherited (check mismatches report)
   
   ActiveRecord::RecordInvalid: Validation failed: Locations expecting locations used by hosts or inherited (check mismatches report)
   
   ActiveRecord::RecordInvalid: Validation failed: Environments expecting environments used by hosts or inherited (check mismatches report)

   ActiveRecord::RecordInvalid: Validation failed: Taxonomy|Capsules expecting capsules used by hosts or inherited (check mismatches report)

   ActiveRecord::RecordInvalid: Validation failed: Ptables expecting ptables used by hosts or inherited (check mismatches report).

Resolution

This issue is being investigated via Bugzilla This content is not included.2028205 by Red Hat Engineering team.
One very generic approach to identify the mismatches is to open WebUI, manage Organizations, click to Mismatches Report button; once those mismatches are fixed, repeat the same for Locations.
Another more fine-tuned approach (that might find nothing sometimes, though), is to identify the mismatches via rake and psql:
- Execute following command to fix the mismatches at organization level (which is the main source of the issue).
```
# echo "Organization.all.each {|org| org.import_missing_ids}" | foreman-rake console
```
- After the successful execution of the same, manually execute the db:seed command and it should not result in any of errors similar to what is mentioned in the Issue section.
```
# foreman-rake db:seed --trace
```
  - If no errors, then
    - Restart all services of the Red Hat Satellite server and retry the upgrade to Satellite 6.10.
    - If any post-upgrade issues related to pulp and pulp_auth components are showing up as failed as a part of ping call check or manual hammer ping execution, refer to this article to fix the problem.
  - But, if a similar error message occurred again, then
    - Execute this command and retry the db:seed command.
```
# echo "Taxonomy.all_import_missing_ids" | foreman-rake console
# foreman-rake db:seed --trace
```
    - If no further errors, then
      - Restart all services of the Red Hat Satellite server and retry the upgrade to Satellite 6.10.
      - If any post-upgrade issues related to pulp and pulp_auth components are showing up as failed as a part of ping call check or manual hammer ping execution, refer to this article to fix the problem.

Reach out to This content is not included.Red Hat Technical Support in case of any further assistance or clarification required.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

A mismatch in Organizations and any of the Locations/Hostgroup/Environment/Domain/Medium/Subnet/ComputeResource/SmartProxy/Realm/PartitionTable causes an attempt to update either that object fails on a validation check which is being performed for an Organization before saving it.

Diagnostic Steps

Following is an example of an error reported in /var/log/foreman-installer/satellite.log during the db:seed execution failure.

2021-12-28 03:37:22 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordInvalid: Validation failed: Domains expecting domains used by hosts or inherited (check mismatches report).

And re-execution of upgrade will skip the db:seed step entirely and then will throw an error like below:

[ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[satellite.example.com]: Could not evaluate: Error making GET request to Foreman at https://satellite.example.com/api/v2/smart_proxies: Response: 422 Unprocessable Entity

There are few independent ways to identify it:

To list individual mismatches (of either type) from individual Organizations, let invoke foreman-rake console and execute there:

Organization.all.map do |org|
  t = TaxHost.new(org, Host.all)
  puts("org=#{org.name} need_to_be_selected=#{t.need_to_be_selected_ids}")
end

The output like:

org=test_org need_to_be_selected={"location_ids"=>[2], "hostgroup_ids"=>[], "environment_ids"=>[1], "domain_ids"=>[1], "medium_ids"=>[], 
"subnet_ids"=>[], "compute_resource_ids"=>[], "smart_proxy_ids"=>[], "user_ids"=>[], "provisioning_template_ids"=>[], "realm_ids"=>[], "ptable_ids"=>[]}

It means that

test_org has some Host in Location ID 2, that is not assigned to the test_org
the Org has a host (potentially different from previous one) with assigned domain that belongs to Organization ID 1.

Optionally, to identify just mismatched Organization/Location assignment to Hosts, run:

su - postgres -c "psql foreman -c \"SELECT h.id,h.name,h.organization_id,h.location_id FROM hosts AS h INNER JOIN \
locations_organizations AS lo ON lo.location_id = h.location_id WHERE lo.organization_id != h.organization_id;\""

That will print output like:

 id |                    name                   | organization_id | location_id 
----+-------------------------------------------+-----------------+-------------
  1 |         myhost.example.xxxxxx.com         |               3 |           2

where the Host of given ID and name is assigned to that Organization ID and Location ID, but the Location is not assigned to that Organization.

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

foreman

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.