Reindexing of foreman database fails with error`could not create unique index "index_fact_names_on_name_and_type"`after the in-place OS upgrade of Satellite 6.11 to RHEL 8 or Satellite 6.16 to RHEL 9

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.11 and 6.16

    • Being converted from RHEL 7 to RHEL 8 or being migrated to RHEL 8.
    • Being converted from RHEL 8 to RHEL 9 or being migrated to RHEL 9.

  • Red Hat Satellite 6.17

Issue

  • After performing the in-place OS upgrade of the Red Hat Satellite 6 server from RHEL 7 to 8 or RHEL 8 to 9, the attempt of reindexing all the databases as per the upgrade guide, runs into one or both of the following errors:

    reindexdb: reindexing database "foreman" 
reindexdb: error: reindexing of database "foreman" failed: ERROR:  could not create unique index "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(augeasversion, PuppetFactName) is duplicated.   

    reindexdb: reindexing database "foreman"
reindexdb: error: reindexing of database "foreman" failed: ERROR:  could not create unique index "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(memory::swaptotal, Katello::RhsmFactName) is duplicated.

    reindexdb: reindexing database "foreman"
reindexdb: error: reindexing of database "foreman" failed: ERROR:  could not create unique index "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(ansible_python::version, ForemanAnsible::FactName) is duplicated.

    reindexdb: reindexing database "foreman"
reindexdb: error: reindexing of database "foreman" failed: ERROR:  could not create unique index "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(processorcount, DiscoveryFactName) is duplicated.

    reindexdb: reindexing database "foreman"
reindexdb: error: reindexing of database "foreman" failed: ERROR: could not create unique index "katello_available_module_streams_name_stream_context".
DETAIL: key (name, stream, context)=(pmdk, 1-fileformat-v6, b4937e53) is duplicated.

    reindexdb: reindexing database "foreman"
reindexdb: error: processing of database "foreman" failed: ERROR:  could not create unique index    "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(ssh::ecdsa::key, PuppetFactName) is duplicated.

  • The problem could also occur during the migration of the whole satellite instance to a new RHEL 8 system.

Resolution

This issue has been reported to the Red Hat Satellite engineering team via This content is not included.Bugzilla 2167984 and a viable immediate resolution of the problem has been identified as well.

Solution 1:

  • Create a backup/snapshot of the Satellite server, as we are going to merge the duplicate records.

  • It is recommended to stop all the services of the affected satellite server except the Postgresql server before proceeding to the next step, especially for a heavy\busy Red Hat Satellite server.
    This will ensure that no duplicate records will be re-created before the database indexes are being fixed.

    # satellite-maintain service stop --exclude postgresql

  • Execute the following rake script on the affected satellite server to fix those broken duplicate records.

    # cat << EOF | foreman-rake console
conf.echo = false
FactName.unscoped.all.pluck(:type).uniq.each do |type|
  type.constantize.unscoped.group_by(&:name).select{|k,v| v.size > 1}.values.each do |dup_names|
    existing = dup_names[0]
    dup_name_ids = dup_names[1..-1].map(&:id)
    p "Found duplicate fact names #{existing.type} #{existing.name} ids #{dup_name_ids}"
    values = FactValue.unscoped.where(fact_name_id: dup_name_ids)
    p "Merging #{values.size} fact values."
    values.in_batches.update_all(fact_name_id: existing.id)
    p "Deleting duplicate fact names."
    FactName.unscoped.where(id: dup_name_ids).destroy_all
  end
end
EOF

  • Once the cleanup has been completed, retry the reindexdb action.

    # runuser -u postgres -- reindexdb -a

  • Restart all satellite services:

    # satellite-maintain service restart

  • Once successfully executed and all services of the concerned satellite server are running fine, proceed with completing the post-upgrade steps from the RHEL upgrade guide.

Solution 2 ( If Solution 1 fails to resolve the problem ):

  • If the reindexdb action reports the duplication error with PuppetFactName, then execute the following command to clear all such types of facts and their corresponding values.

    # cat << EOF | foreman-rake console
conf.echo = false
fact_name_ids = FactName.unscoped.where(:type => 'PuppetFactName').ids
pp fact_name_ids.count
FactValue.unscoped.where(:fact_name_id => fact_name_ids).delete_all
FactName.unscoped.where(:id => fact_name_ids).delete_all
pp FactName.unscoped.where(:type => 'PuppetFactName').ids.count
EOF

    NOTE: These facts will come back on their own when the puppet agent is executed on concerned systems or the scheduled puppet execution will happen on them.

  • If the reindexdb action reports the duplication error with Katello::RhsmFactName, then execute the following command to clear all such types of facts and their corresponding values.

    # cat << EOF | foreman-rake console
conf.echo = false
fact_name_ids = FactName.unscoped.where(:type => 'Katello::RhsmFactName').ids
pp fact_name_ids.count
FactValue.unscoped.where(:fact_name_id => fact_name_ids).delete_all
FactName.unscoped.where(:id => fact_name_ids).delete_all
pp FactName.unscoped.where(:type => 'Katello::RhsmFactName').ids.count
EOF

    NOTE: These facts will come back on their own when the scheduled rhsm checkin happens for the individual client systems (every 4 hours ).

  • If the reindexdb action reports the duplication error with ForemanAnsible::FactName, then execute the following command to clear all such types of facts and their corresponding values.

    # cat << EOF | foreman-rake console
conf.echo = false
fact_name_ids = FactName.unscoped.where(:type => 'ForemanAnsible::FactName').ids
pp fact_name_ids.count
FactValue.unscoped.where(:fact_name_id => fact_name_ids).delete_all
FactName.unscoped.where(:id => fact_name_ids).delete_all
pp FactName.unscoped.where(:type => 'ForemanAnsible::FactName').ids.count
EOF

    NOTE: These facts will come back on their own when ansible roles will be executed on the client hosts.

  • If the reindexdb action reports the duplication error with DiscoveryFactName, then execute the following command to clear all such types of facts and their corresponding values.

    # cat << EOF | foreman-rake console
conf.echo = false
fact_name_ids = FactName.unscoped.where(:type => 'DiscoveryFactName').ids
pp fact_name_ids.count
FactValue.unscoped.where(:fact_name_id => fact_name_ids).delete_all
FactName.unscoped.where(:id => fact_name_ids).delete_all
pp FactName.unscoped.where(:type => 'DiscoveryFactName').ids.count
EOF

    NOTE: These facts will come back on their own when the system being built will be re-discovered via Foreman Discovery process.

  • If the reindexdb action reports the duplication error with key (name, stream, context)=(pmdk, 1-fileformat-v6, XXXXXXXX), then refer to this solution article to resolve the conflict.

  • Once the cleanup has been completed, retry the reindexdb action.

    # runuser -u postgres -- reindexdb -a

  • Once successfully executed and all services of the concerned satellite server are running fine, proceed with completing the post-upgrade steps from the RHEL upgrade guide.

 

Reach out to the This content is not included.Red Hat Technical Support should any further assistance be required.

For more KB articles/solutions related to Red Hat Satellite 6.x LEAPP Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x LEAPP Issues

For more KB articles/solutions related to Red Hat Satellite 6.x PostgreSQL-related Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x PostgreSQL-related Issues

Root Cause

The exact root cause is still under investigation\discussion but

  • After the conversion, The indexes were already broken for the DB tables due to the This content is not included.Bugzilla 2142270.

  • Once the system boots up after a successful in-place upgrade and satellite-installer has been successfully executed at the backend, There could be a short delay between that and when the reindexdb action will be performed and client systems have already started communicating with the satellite server by that time.

  • During that short timespan, if any new system facts ( RHSM or Puppet ) have come into the satellite database, Those facts might be what resulted in the issue reported in the This content is not included.Bugzilla 2167984.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.