[IPA][SSSD][AD Trust] Trusted AD users with Enterprise UPN cannot login to IPA clients

Environment

RHEL 7.0/7.1/7.2
sssd-1.13.0-40 and earlier
workaround available for sssd-1.13 and above (RHEL7.2 and later)
fix available for sssd-1.14 and above (coming with RHEL7.3)

Issue

Trusted AD users with Enterprise UPN cannot login to IPA clients

Resolution

Current versions of SSSD do not support Enterprise UPNs from AD. This feature is coming with RHEL7.3 release and sssd-1.14.x versions.

Effective workarounds for sssd versions 1.13 and above are

1 - Set in the domain section of sssd.conf of the IPA server the following options and restart sssd to apply the changes.

    subdomain_inherit = ldap_user_principal
    ldap_user_principal = nosuchattr

2 - Correct the UPN in Active Directory

Root Cause

Current versions of SSSD do not support Enterprise UPNs from AD. This feature is coming with RHEL7.3 release and sssd-1.14.x versions.

Diagnostic Steps

When trying to access an IPA client with an AD trusted user, the user is getting permission denied on command line.
In krb5_child.log we can see sssd trying to find a KDC for a realm completely different to the trusted setup.
As it turns out and verified from AD administrators, the suffix of the UPN of the user is a completely different domain.
This is a normal AD setup, however we do not yet have the capability to make use of it since currently SSSD is grabbing the UPN of the user and uses the suffix of it as a realm and tries to find a KDC for this realm to request/validate a TGT for the incoming user.

• IPA realm = ipa.com
• AD domain = ad.example.com
• AD user = aduser@ad.example.com
• UPN = aduser@example.com

/var/log/sssd/krb5_child.log
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [unpack_buffer] (0x0100): cmd [241] uid [2363320234] gid [2363320234] validate [true] enterprise principal [false] offline [false] UPN [aduser@EXAMPLE.COM]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [unpack_buffer] (0x0100): ccname: [FILE:/tmp/krb5cc_2363320234_XXXXXX] old_ccname: [not set] keytab: [/etc/krb5.keytab]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [k5c_setup_fast] (0x0100): SSSD_KRB5_FAST_PRINCIPAL is set to [host/ipaclient.ipa.com@IPA.COM]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [match_principal] (0x1000): Principal matched to the sample (host/ipaclient.ipa.com@IPA.COM).
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [check_fast_ccache] (0x0200): FAST TGT is still valid.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [become_user] (0x0200): Trying to become user [2363320234][2363320234].
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_RENEWABLE_LIFETIME] from environment.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_LIFETIME] from environment.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [set_canonicalize_option] (0x0100): SSSD_KRB5_CANONICALIZE is set to [true]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [main] (0x0400): Will perform online auth
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [tgt_req_child] (0x1000): Attempting to get a TGT
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [get_and_save_tgt] (0x0400): Attempting kinit for realm [EXAMPLE.COM]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [get_and_save_tgt] (0x0020): 1232: [-1765328164][Cannot resolve servers for KDC in realm "EXAMPLE.COM"]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [map_krb5_error] (0x0020): 1301: [-1765328164][Cannot resolve servers for KDC in realm "EXAMPLE.COM"]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [k5c_send_data] (0x0200): Received error code 1432158222
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19025]]]] [main] (0x0400): krb5_child completed successfully
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [main] (0x0400): krb5_child started.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [unpack_buffer] (0x1000): total buffer size: [130]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [unpack_buffer] (0x0100): cmd [241] uid [2363320234] gid [2363320234] validate [true] enterprise principal [false] offline [false] UPN [aduser@EXAMPLE.COM]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [unpack_buffer] (0x0100): ccname: [FILE:/tmp/krb5cc_2363320234_XXXXXX] old_ccname: [not set] keytab: [/etc/krb5.keytab]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [k5c_setup_fast] (0x0100): SSSD_KRB5_FAST_PRINCIPAL is set to [host/ipaclient.ipa.com@IPA.COM]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [match_principal] (0x1000): Principal matched to the sample (host/ipaclient.ipa.com@IPA.COM).
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [check_fast_ccache] (0x0200): FAST TGT is still valid.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [become_user] (0x0200): Trying to become user [2363320234][2363320234].
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_RENEWABLE_LIFETIME] from environment.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_LIFETIME] from environment.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [set_canonicalize_option] (0x0100): SSSD_KRB5_CANONICALIZE is set to [true]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [main] (0x0400): Will perform online auth
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [tgt_req_child] (0x1000): Attempting to get a TGT
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [get_and_save_tgt] (0x0400): Attempting kinit for realm [EXAMPLE.COM]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [get_and_save_tgt] (0x0020): 1232: [-1765328164][Cannot resolve servers for KDC in realm "EXAMPLE.COM"]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [map_krb5_error] (0x0020): 1301: [-1765328164][Cannot resolve servers for KDC in realm "EXAMPLE.COM"]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [k5c_send_data] (0x0200): Received error code 1432158222
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19026]]]] [main] (0x0400): krb5_child completed successfully
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [main] (0x0400): krb5_child started.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [unpack_buffer] (0x1000): total buffer size: [130]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [unpack_buffer] (0x0100): cmd [241] uid [2363320234] gid [2363320234] validate [true] enterprise principal [false] offline [true] UPN [aduser@EXAMPLE.COM]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [unpack_buffer] (0x0100): ccname: [FILE:/tmp/krb5cc_2363320234_XXXXXX] old_ccname: [not set] keytab: [/etc/krb5.keytab]
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [become_user] (0x0200): Trying to become user [2363320234][2363320234].
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [become_user] (0x0200): Trying to become user [2363320234][2363320234].
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [become_user] (0x0200): Already user [2363320234].
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_RENEWABLE_LIFETIME] from environment.
(Thu Nov 10 10:46:11 2016) [[sssd[krb5_child[19027]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_LIFETIME] from environment.

/var/log/secure
Nov 10 10:46:11 dmnanlx7210 sshd[19020]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=addc1.ad.example.com  user=AD\aduser
Nov 10 10:46:11 dmnanlx7210 sshd[19020]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= addc1.ad.example.com user=AD\aduser
Nov 10 10:46:11 dmnanlx7210 sshd[19020]: pam_sss(sshd:auth): received for user AD\aduser: 6 (Permission denied)

/var/log/sssd/sssd_ipa.com.log
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [sysdb_update_members_ex] (0x0020): Could not add member [aduser@ad.example.com] to group name=aduser@ad.example.com,cn=groups,cn=ad.example.com,cn=sysdb]. Skipping.
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [acctinfo_callback] (0x0100): Request processed. Returned 0,0,Success
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [be_req_set_domain] (0x0400): Changing request domain from [ipa.com] to [ad.example.com]
...
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [pam_print_data] (0x0100): command: SSS_PAM_AUTHENTICATE
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [pam_print_data] (0x0100): domain: ad.example.com
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [pam_print_data] (0x0100): user: aduser@ad.example.com
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [pam_print_data] (0x0100): service: sshd
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [pam_print_data] (0x0100): tty: ssh
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [pam_print_data] (0x0100): ruser: 
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [pam_print_data] (0x0100): rhost: addc1.ad.example.com
...
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [krb5_auth_queue_send] (0x1000): Wait queue of user [aduser@ad.example.com] is empty, running request [0x20076e0] immediately.
(Thu Nov 10 10:46:11 2016) [sssd[be[ipa.com]]] [krb5_auth_prepare_ccache_name] (0x1000): No ccache file for user [aduser@ad.example.com] found.

/var/log/messages
Nov 10 10:46:11 dmnanlx7210 [sssd[krb5_child[19025]]]: Cannot resolve servers for KDC in realm "EXAMPLE.COM"
Nov 10 10:46:11 dmnanlx7210 [sssd[krb5_child[19025]]]: Cannot resolve servers for KDC in realm "EXAMPLE.COM"
Nov 10 10:46:11 dmnanlx7210 [sssd[krb5_child[19026]]]: Cannot resolve servers for KDC in realm "EXAMPLE.COM"
Nov 10 10:46:11 dmnanlx7210 [sssd[krb5_child[19026]]]: Cannot resolve servers for KDC in realm "EXAMPLE.COM"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.