Failing to install ruby or nginx package from RHEL9 AppStream to a Satellite 6 client

Solution Verified - Updated

Environment

  • Red Hat Satellite 6
    • with synced content from RHEL 9 AppStream and Oracle Linux 9 AppStream

Issue

  • Having a Satellite 6 client attached to a Content View (CV) with RHEL9 AppStream.
  • The CV contains any filters, on either packages/errata/modules and either include or exclude
  • The Satellite has synchronized both RHEL9 AppStream and Oracle Linux 9 AppStream repos
  • An attempt to install various ruby packages OR anything from nginx module fails - despite the client can even download all required packages

Resolution

Reactive workaround

  • Run Complete Sync of RHEL9 AppStream repo (ensure that RHEL9 AppStream repo got synchronized later than Oracle Linux 9 AppStream repo had got).
  • Once completed, republish repositories of affected CV version - provide your own CV version ID:
hammer content-view version republish-repositories --id 67 --force true

  • If Composite CV is affected, run the republish for the component CV first, and once it completes, repeat the same for the CCV

  • be aware that the above workaround will loose default modularity enabledness information - see This content is not included.this bugzilla. If that is an issue for you, do not republish repositories, but publish a new version of (C)CV.

Proactive workaround

Before publishing a CV with filters which contains RHEL9 AppStream repo, ensure that RHEL9 AppStream repo got synchronized some content later than Oracle Linux 9 AppStream repo had got. By "synchronized some content", we mean either Complete Sync or a regular Optimised Sync that fetched some new content.

Definitive solution

One possible permanent solution is to turn to the provider of Oracle Linux 9 AppStream repo with a request to fix the modularity metadata. See Root Cause for details.

Another possible permanent solution is to wait when the This content is not included.raised bugzilla - making Satellite more robust - will be fixed.

For more KB articles/solutions related to Red Hat Satellite 6.x Content View Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Content View Issues

Root Cause

Red Hat released in RHEL 9 AppStream ruby-3.1.2-141.module+el9.1.0+15737+76195479.x86_64.rpm (and various other ruby-* packages of the same version+release), within ruby 3.1 module version 9010020220623124214. Oracle Linux has later also released in OL 9 AppStream ruby-3.1.2-141.module+el9.1.0+20815+286161bd.x86_64.rpm (slightly higher release) within the same ruby module (the same name+stream+version+context+arch, a unique identification of a module). So a different operating system released the same module but with different content. That breaks the requirement of a module, that a unique module delivers the same content, in either repository, world-wide.

Pulp in Satellite works on that natural requirement. So when some new content is synced (say, for OL9 AppStream repo), it processes the module and creates or updates a record in DB for it. So When syncing RHEL9 first, pulp writes down module snippet telling "this module enables 15737+76195479 packages" - but further OL9 repo sync overwrites this information to "this module enables 20815+286161bd packages".

Now, (only) Content Views with filters do rely on that snippet when generating metadata for their repositories - which happens when publishing a new CV version also. So OL9-related snippet would be used even for RHEL9 modularity metadata in these case. That will happen until one synchronizes a new content to RHEL9 repo, which overrides the snippet back to RHEL9 original one.

The above is applicable to any mirroring policy of either RHEL9 or OL9 repo. An attempt of preventive workaround to set either repo with some alternate mirroring policy does not work.

Diagnostic Steps

  • Various packages can be successfully downloaded to a client:
# dnf install rubygem-foreman_scap_client --downloadonly
Updating Subscription Management repositories.
..
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                                            20 MB/s |  20 MB     00:01
..
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                          Architecture                Version                                                         Repository                                                      Size
===================================================================================================================================================================================================================
Installing:
 rubygem-foreman_scap_client                      noarch                      0.5.1-1.1.el9sat                                                satellite-client-6-for-rhel-9-x86_64-rpms                       26 k
Installing dependencies:
 bzip2                                            x86_64                      1.0.8-8.el9                                                     rhel-9-for-x86_64-baseos-rpms                                   60 k
 ruby                                             x86_64                      3.1.2-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                                44 k
 ruby-default-gems                                noarch                      3.1.2-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                                44 k
 ruby-libs                                        x86_64                      3.1.2-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                               3.3 M
 rubygem-io-console                               x86_64                      0.5.11-141.module+el9.1.0+15737+76195479                        rhel-9-for-x86_64-appstream-rpms                                28 k
 rubygem-psych                                    x86_64                      4.0.3-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                                62 k
 rubygems                                         noarch                      3.3.7-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                               301 k

Transaction Summary
===================================================================================================================================================================================================================
Install  8 Packages

Total download size: 3.9 M
Installed size: 13 M
DNF will only download packages for the transaction.
Is this ok [y/N]: y
Downloading Packages:
(1/8): bzip2-1.0.8-8.el9.x86_64.rpm                                                                                                                                                864 kB/s |  60 kB     00:00
(2/8): ruby-3.1.2-141.module+el9.1.0+15737+76195479.x86_64.rpm                                                                                                                     567 kB/s |  44 kB     00:00
(3/8): rubygem-foreman_scap_client-0.5.1-1.1.el9sat.noarch.rpm                                                                                                                     303 kB/s |  26 kB     00:00
(4/8): ruby-default-gems-3.1.2-141.module+el9.1.0+15737+76195479.noarch.rpm                                                                                                        1.7 MB/s |  44 kB     00:00
(5/8): rubygem-io-console-0.5.11-141.module+el9.1.0+15737+76195479.x86_64.rpm                                                                                                      660 kB/s |  28 kB     00:00
(6/8): rubygem-psych-4.0.3-141.module+el9.1.0+15737+76195479.x86_64.rpm                                                                                                            1.4 MB/s |  62 kB     00:00
(7/8): ruby-libs-3.1.2-141.module+el9.1.0+15737+76195479.x86_64.rpm                                                                                                                 31 MB/s | 3.3 MB     00:00
(8/8): rubygems-3.3.7-141.module+el9.1.0+15737+76195479.noarch.rpm                                                                                                                 2.1 MB/s | 301 kB     00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                               16 MB/s | 3.9 MB     00:00
Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
#
  • But an attempt to install them fails on No available modular metadata for modular package error:
# dnf install rubygem-foreman_scap_client
Updating Subscription Management repositories.
..
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                          Architecture                Version                                                         Repository                                                      Size
===================================================================================================================================================================================================================
Installing:
 rubygem-foreman_scap_client                      noarch                      0.5.1-1.1.el9sat                                                satellite-client-6-for-rhel-9-x86_64-rpms                       26 k
Installing dependencies:
 bzip2                                            x86_64                      1.0.8-8.el9                                                     rhel-9-for-x86_64-baseos-rpms                                   60 k
 ruby                                             x86_64                      3.1.2-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                                44 k
 ruby-default-gems                                noarch                      3.1.2-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                                44 k
 ruby-libs                                        x86_64                      3.1.2-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                               3.3 M
 rubygem-io-console                               x86_64                      0.5.11-141.module+el9.1.0+15737+76195479                        rhel-9-for-x86_64-appstream-rpms                                28 k
 rubygem-psych                                    x86_64                      4.0.3-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                                62 k
 rubygems                                         noarch                      3.3.7-141.module+el9.1.0+15737+76195479                         rhel-9-for-x86_64-appstream-rpms                               301 k

Transaction Summary
===================================================================================================================================================================================================================
Install  8 Packages

Total size: 3.9 M
Installed size: 13 M
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] bzip2-1.0.8-8.el9.x86_64.rpm: Already downloaded
[SKIPPED] rubygem-foreman_scap_client-0.5.1-1.1.el9sat.noarch.rpm: Already downloaded
[SKIPPED] ruby-3.1.2-141.module+el9.1.0+15737+76195479.x86_64.rpm: Already downloaded
[SKIPPED] ruby-default-gems-3.1.2-141.module+el9.1.0+15737+76195479.noarch.rpm: Already downloaded
[SKIPPED] rubygem-io-console-0.5.11-141.module+el9.1.0+15737+76195479.x86_64.rpm: Already downloaded
[SKIPPED] rubygem-psych-4.0.3-141.module+el9.1.0+15737+76195479.x86_64.rpm: Already downloaded
[SKIPPED] rubygems-3.3.7-141.module+el9.1.0+15737+76195479.noarch.rpm: Already downloaded
[SKIPPED] ruby-libs-3.1.2-141.module+el9.1.0+15737+76195479.x86_64.rpm: Already downloaded
Running transaction check
No available modular metadata for modular package 'ruby-3.1.2-141.module+el9.1.0+15737+76195479.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'ruby-default-gems-3.1.2-141.module+el9.1.0+15737+76195479.noarch', it cannot be installed on the system
No available modular metadata for modular package 'rubygem-io-console-0.5.11-141.module+el9.1.0+15737+76195479.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'rubygem-psych-4.0.3-141.module+el9.1.0+15737+76195479.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'rubygems-3.3.7-141.module+el9.1.0+15737+76195479.noarch', it cannot be installed on the system
No available modular metadata for modular package 'ruby-libs-3.1.2-141.module+el9.1.0+15737+76195479.x86_64', it cannot be installed on the system
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: No available modular metadata for modular package
#
  • RHEL9 AppStream repository metadata for modularity contains different version of the package (cf 15737+76195479 above and 20815+286161bd below):
# vim /var/cache/dnf/rhel-9-for-x86_64-appstream-rpms-*/repodata/*-modules.yaml
..
document: modulemd
version: 2
data:
  name: ruby
  stream: "3.1"
  version: 9010020220623124214
  context: 9
  static_context: true
  arch: x86_64
  summary: An interpreter of object-oriented scripting language
..
  artifacts:
    rpms:
    - ruby-0:3.1.2-141.module+el9.1.0+20815+286161bd.x86_64
..
#
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.