How to solve users unable to authenticate to IPA/IDM with PAC issues - S4U2PROXY_EVIDENCE_TKT_WITHOUT_PAC error

Solution Verified - Updated

Environment

RHEL 8.9, ipa-server-4.9.12-11 +
RHEL 9.3, ipa-server-4.10.2-5 +

Issue

After upgrading to RHEL 8.9 or RHEL 9.3, most IPA users are unable to log into WebUI or kinit, with errors like

 GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credential cache is empty)

Subsequently, any ipa commands start to fail with

# ipa -d user-show
...
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 730, in single_request response.msg)
xmlrpc.client.ProtocolError: ... 401 Unauthorized>

In krb5kdc.log of IPA we can see staple error

S4U2PROXY_EVIDENCE_TKT_WITHOUT_PAC

Resolution

In order to resolve this issue, we need to generate SIDs (ipaNTSecurityIdentifier attribute in LDAP) for all IPA users and groups.

1. Enabling SID generation and triggering SID generation task

If admin can run IPA commands

If any of IPA's admin users can authenticate, we can enable the SID generation and trigger the SIDgen task via

# ipa config-mod --enable-sid --add-sids

Note! This task is resource-intensive, as it goes through all the users and groups in IPA installation!

If admin users can't authenticate

If the default admin user and any other privileged users don't have ipaNTSecurityIdentifier, this will fail, as admin will fail to authenticate as well. We can force this via

# /usr/libexec/ipa/oddjob/org.freeipa.server.config-enable-sid --add-sids

which is essentially the backend of the command above. Note that this command needs to be run through oddjob mechanism and if run directly, it is best to run it as specified above to make sure that right SELinux context is used to execute it.

Note! Even though it is a python script, do not run it through python interpreter, like 

python3 /usr/libexec/ipa/oddjob/org.freeipa.server.config-enable-sid --add-sids

Doing so can break the SELinux fcontext and prevent command execution of the SIDgen task.

If the IPA installation doesn't have NetBIOS name and the domain SID set up, this will fail with

DEBUG The ipa-enable-sid command failed, exception: TypeError: ('Tuple_to_LDAPMod(): expected a byte string in the list', None)
ERROR ('Tuple_to_LDAPMod(): expected a byte string in the list', None)
ERROR The ipa-enable-sid command failed. See /var/log/ipaserver-enable-sid.log for more information

In this case, customer should choose its IPA domain NetBIOS name (15 max characters, NOT the same as the realm, more about limitations Content from learn.microsoft.com is not included.here), then run

# /usr/libexec/ipa/oddjob/org.freeipa.server.config-enable-sid --add-sids --netbios-name CHOSENNAME

2. Results of SIDgen task

After the task is triggered, it will write into /var/log/dirsrv/slapd-INSTANCE/errors log.

Success

If you can see this:

ERR - sidgen_task_thread - [file ipa_sidgen_task.c, line 199]: Sidgen task finished [0].

you are good to go, and your users should be able to authenticate. Try

# kinit username
# ipa user-find username --all | grep -i ipantsecurityidentifier

for any previously faulty username.

However, if something is not fine, SIDgen task will fail on the first issue it finds, and you have to fix it before proceeding further.

Range issues

If you see errors like

ERR - get_ranges - [file ipa_sidgen_common.c, line 276]: Failed to convert LDAP entry to range struct.
ERR - ipa_sidgen_add_post_op - [file ipa_sidgen.c, line 140]: Failed to get ID ranges.

this means that something is off with ID ranges in the IPA. You can see 4. Fixing ID range issues manually down the document.

Incorrect ID issues

If you see errors like

ERR - find_sid_for_ldap_entry - [file ipa_sidgen_common.c, line 522]: Cannot convert Posix ID [10022] into an unused SID.

this means that something is off with the users and/or groups, there are either duplicate IDs or some of the entities are out of IPA-managed ranges. See 5. Fixing user issues manually.

3. Automatic fix for ranges and IDs out of range issues

NOTE: If you have RHEL 8.10, ipa-server-4.9.13-14+, or RHEL 9.6+, please use ipa-idrange-fix tool for fixing the ranges. It will automatically detect any issues related to SID generation and propose automatic solutions. To start, please run from root:

# ipa-idrange-fix

The tool doesn't write any changes until explicit confirmation from the user, and allows customization of the proposed solutions, so please refer to man ipa-idrange-fix for more information. Once the changes are applied, re-run the SIDgen task to generate SIDs with corrected ID ranges.

If the proposed options don't fit you, please fall back for the manual approach.

4. Fixing ID range issues manually

First, we need to list currently available ranges. If your admin user works, run:

# ipa idrange-find --all --raw

or, if this doesn't work, run ldapsearch (change $SUFFIX according to your domain suffix):

# ldapsearch -xLLL -D "cn=Directory Manager" -W -b "cn=ranges,cn=etc,$SUFFIX" "(objectClass=ipaIDrange)"

All of your ipa-local ranges should have both ipabaserid and ipasecondarybaserid attributes filled in. If that's not the case, you should choose base RID and secondary base RID as described in this article, see Handling multiple IDRanges for IPA. When chosen and carefully calculated to not intersect with any others, add them via (change the numbers accordingly!):

# ldapmodify -D "cn=Directory Manager" -W -x << EOF
dn: cn=IDRANGENAME,cn=ranges,cn=etc,$SUFFIX
changetype: modify
add: ipabaserid
ipabaserid: 2000
-
add: ipasecondarybaserid
ipasecondarybaserid: 5001
EOF

If the command fails with constraint violation, please check your calculations again.

After the ranges are fixed, please re-run the SIDgen task again, as described in 1. Enabling SID generation and triggering SID generation task, and observe the results again.

5. Fixing user issues manually

Out of range

If your user is out of IPA-managed range, it won't get any RID, as those are tied to ID ranges, and SIDgen task won't be able to assign SID for this user. First, list the available ranges as described in 4. Fixing ID range issues manually and check if the error ID Cannot convert Posix ID [10022] into an unused SID. is out of the ranges specified.
If that's the case, chances are, this user is not alone. You can search for users and groups that are out of ranges with searches like these:

# ldapsearch -xLLL -D "cn=Directory Manager" -W -b "cn=users,cn=accounts,$SUFFIX" "(&(objectClass=posixaccount)(|(&(uidNumber>=1)(uidNumber<=IDrange1Start))(&(uidNumber>=IDrange1End)(uidNumber<=IDrange2Start))))" dn uidNumber > outofranges.ldif

# ldapsearch -xLLL -D "cn=Directory Manager" -W -b "cn=groups,cn=accounts,$SUFFIX" "(&(objectClass=posixgroup)(|(&(gidNumber>=1)(gidNumber<=IDrange1Start))(&(gidNumber>=IDrange1End)(gidNumber<=IDrange2Start))))" dn gidNumber >> outofranges.ldif

You can expand this search with additional (&(gidNumber>=IDrangeNEnd)(gidNumber<=IDrangeN+1Start)) block if you have multiple ranges. Make sure to replace entries like IDrangeN and IDrangeNEnd with actual integer numbers for IDs, and $SUFFIX with your domain default suffix!

Once you get the result, sort IDs and evaluate the scope of the problem:

# grep Number outofranges.ldif | awk '{print $2}' | sort -n | less

If there are a lot of users and groups, if might be easier to create a new IPA range to accommodate them. Determine the limits of this new range(s) to fit all the IDs as well as calculate ipabaserid and ipasecondarybaserid attributes, and add it via

# ipa idrange-add new_range_name --base-id=10000 --range-size=1000000 --rid-base=1000000 --secondary-rid-base=2000001

Change the numbers accordingly!

If there are a few users/groups, you may consider moving them to existing ID ranges via

# ipa user-mod username --uid=469000103 --gid=469000103         (if the user has private group)
# ipa user-mod username --uid=469000103                         (if the user doesn't have a private group)
# ipa group-mod groupname --gid=469000104

Note! File permissions on Linux systems are ID-driven, so changing user UID will render user unable to access previously created files, as they will have permissions assigned according to old UID. Make sure to re-chown necessary files afterwards. SSSD doesn't support ID change too, so client's cache have to be removed in order to see immediate result of ID change.

Duplicate IDs

If the user/group is in a valid range, but still gets the same error, chances are that user/group has duplicate ID with another user/group. You will see errors like this:

ERR - rid_to_sid_with_check - [file ipa_sidgen_common.c, line 384]: SID [S-1-5-21-123456789-123456789-123456789-1006] is already used.

If two users share UID, they would get same RID, but SIDs must be unique for the domain. This also poses serious security risk permission-wise, so generally ID collisions are not supported by IPA. Normally, IPA won't allow that, but manual ID change can force this behavior. To check this, dump list of all users and groups into files like this:

# ldapsearch -xLLL -D "cn=Directory Manager" -W -b "cn=users,cn=accounts,$SUFFIX" "(objectClass=posixaccount)" dn uidNumber > allUsers.ldif
# ldapsearch -xLLL -D "cn=Directory Manager" -W -b "cn=groups,cn=accounts,$SUFFIX" "(objectClass=posixgroup)" dn gidNumber > allGroups.ldif

Then you can list list duplicating IDs and the amount of duplicates like this:

# grep uidNumber allUsers.ldif | sort | uniq --count --repeated
# grep gidNumber allGroups.ldif | sort | uniq --count --repeated

If the commands above do return some of the IDs - there is no other option, but to move all the duplicates to its own unique ID via

# ipa user-mod username --uid=469000103 --gid=469000103         (if the user has private group)
# ipa user-mod username --uid=469000103                         (if the user doesn't have a private group)
# ipa group-mod groupname --gid=469000104

Make sure that you're choosing IDs that are not occupied, but still are in the IPA-managed ranges!

After the entities are moved, check all the users/groups if they ended up with incorrect RID in the end of the SID:

# ipa user-show username --all
# ipa group-show groupname --all

If that's the case, remove the incorrect SID with

# ipa user-mod username --delattr=ipantsecurityidentifier=S-1-5-21-123456789-123456789-123456789-1006 --delattr=objectclass=ipantuserattrs
# ipa group-mod groupname --delattr=ipantsecurityidentifier=S-1-5-21-123456789-123456789-123456789-1007 --delattr=objectclass=ipaNTGroupAttrs

Same can be done with ldapmodify command (change $SUFFIX and username accordingly):

# ldapmodify -D "cn=Directory Manager" -W << EOF
dn: uid=username,cn=users,cn=accounts,$SUFFIX
changetype: modify
delete: ipaNTSecurityIdentifier
-
delete: objectClass
objectClass: ipantuserattrs
EOF

or the same for groups:

# ldapmodify -D "cn=Directory Manager" -W << EOF
dn: cn=groupname,cn=groups,cn=accounts,$SUFFIX
changetype: modify
delete: ipaNTSecurityIdentifier
-
delete: objectClass
objectClass: ipaNTGroupAttrs
EOF

To ensure everything runs smoothly, it might be a good idea to clear up all SIDs of all the users and groups that were participating in duplication, we will be regenerating them anyway, but regeneration will not change already generated SIDs!

After the user issues are fixed, please re-run the SIDgen task again, as described in 1. Enabling SID generation and triggering SID generation task, and observe the results again.

Root Cause

Root cause of the issue - IPA users don't have SIDs (ipaNTSecurityIdentifier attribute in LDAP) generated for them, and user's PAC validation fails.

For more info on the root cause, please refer to:

Diagnostic Steps

You can get a file with all users and groups that don't have SIDs via

# ldapsearch -xLLL -D "cn=Directory Manager" -W -b "cn=users,cn=accounts,$SUFFIX" "(&(objectClass=posixaccount)(!(ipaNTSecurityIdentifier=*)))" dn uidNumber >  noSID.ldif
# ldapsearch -xLLL -D "cn=Directory Manager" -W -b "cn=groups,cn=accounts,$SUFFIX" "(&(objectClass=posixgroup)(objectClass=ipausergroup)(!(ipaNTSecurityIdentifier=*)))" dn gidNumber >> noSID.ldif

Don't forget to change $SUFFIX with your actual domain suffix!

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.