ID range management in IPA

POSIX IDs and IPA

Local domain range

IPA is an identity management solution that allows to centrally define users and groups. As it operates in Linux environments, the users and groups are defined with a name and a POSIX ID (a numeric value), that are used for access control: each file belongs to a user and a group identified by their POSIX ID, sudo rules evaluation relies on POSIX IDs and so on.

When users and groups are created inside IPA, they can either be created with a provided id or the admin can let IPA assign an id. The range from which the ids can be picked is defined at the IPA level and is represented by a local ID range visible with the ipa idrange-find command:

[root@server ~]# ipa idrange-find
---------------
1 range matched
---------------
  Range name: IPA.TEST_id_range
  First Posix ID of the range: 936000000
  Number of IDs in the range: 200000
  Range type: local domain range
----------------------------
Number of entries returned 1
----------------------------

During the first IPA server installation, this ID range can either be provided through a base ID number (—idstart) and a max ID (—id max), or be picked randomly by the installer.

In the above example, the users and groups will get assigned IDs starting from 936000000, up to 936000000 + 200000 (not included).

When the ID range is depleted, the admin can extend the possible ID numbers by defining additional local ranges (ipa idrange-add --type ipa-local).

Other types of ranges

When IPA has configured a trust with an Active Directory forest, the users and groups defined in AD also need to have a POSIX ID for access control performed on IPA side. There are 2 different situations:

• if AD has been extended with Identity Management for UNIX, and is also storing a uidNumber and a gidNumber for each user, and a gidNumber for each group, then IPA can directly re-use the POSIX IDs stored on AD side. The trust is established with a range type “Active Directory domain with POSIX attributes” (ipa-ad-trust-posix).
• If AD doesn’t store POSIX IDs, IPA maps the unique SID value from AD users and groups to a POSIX ID based on settings defined for the trust. The trust is established with a range type “Active Directory domain” (ipa-ad-trust) and sets the first POSIX ID for the mapping.

As a summary, IPA manages 3 different types of ID ranges:

• local domain
• active directory domain
• active directory domain with POSIX attributes

In any case, the various ranges must not overlap in order to guarantee that an ID corresponds to a unique user or group.

Next available ID

DNA range plugin

During the first server installation, the local ID range is set and defines the range from which IDs get picked for new IPA users and groups. This local ID range is used for the whole IPA deployment.

The server is then configured with the DNA range plugin (Dynamic Number Assignment), that provides a mechanism to automatically assign IDs but also ensure existing IDs are not re-used. The implementation is simple to understand: the DNA plugin stores the next available ID and assigns the IDs sequentially.

The existing DNA ranges can be seen using ipa-replica-manage dnarange-show. If the command is run when there is a single server, the output looks like the following:

[root@server ~]# ipa-replica-manage  dnarange-show
server.ipa.test: 936000000-936199999

The DNA range on the first server starts from the same value as the local domain ID range , and ends on first POSIX ID + number of IDs – 1. When the first user gets created, he is assigned a POSIX ID, and the DNA range plugin increments the next available value:

[root@server ~]# ipa user-add idmuser1 --first idm --last user1 
...
[root@server ~]# ipa-replica-manage  dnarange-show
server.ipa.test: 936000002-936199999

The above output shows that the available range has shrinked due to the new user taking one ID. Note that the first ID (936000000) is reserved for the admin user.

Splitting ranges between replicas

When additional servers (replicas) are set up, it is important to make sure that a given ID is not assigned at the same time by 2 different replicas to different users/groups. In order to avoid this type of conflict, each replica is also configured with the DNA range plugin (dynamic number assignment).

[root@server ~]# ipa-replica-manage  dnarange-show
server.ipa.test: 936000002-936199999
replica.ipa.test: No range set

The above command shows that the replica hasn’t created yet any user or group and didn’t request any DNA range. But as soon as the replica needs to assign an ID, it communicates with the server and grabs a sub-range:

[root@replica ~]# ipa user-add idmuser2 --first idm --last user2
...
[root@replica ~]# ipa-replica-manage dnarange-show
server.ipa.test: 936000002-936100499
replica.ipa.test: 936100501-936199999

The output shows that server.ipa.test abandoned part of its range to replica.ipa.test. From now on, server.ipa.test will be able to assign IDs up to 936100499, while replica.ipa.test will assign IDs starting from 936100501 (he assigned 936100500 to the new user).

The DNA range is configured locally to each replica, in a part of the configuration that is not replicated. The DNA range records the next available ID and the max available ID. Remember, the DNA range is constantly evolving: as soon as a replica assigns an ID to a new user, the next available ID is incremented and the range shrinks over time.

It is important to make sure that DNA ranges are not overlapping, and that each DNA range fits into local ID ranges.

Frequent issues

Issues can happen when a replica doesn’t have any assigned DNA range and fails to grab a range when it needs to (for instance because the only other server he knows is down).

In this case, manual remediation is possible using ldapmodify, by editing the DNA range plugin configuration on the replica. The DNA range plugin configuration can be seen within the following LDAP entry:

[root@replica ~]# ldapsearch -LLL -D "cn=directory manager" -W -o ldif-wrap=no -b "cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config" dnaMaxValue dnaNextValue 
Enter LDAP Password: 
dn: cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
dnaMaxValue: 936199999
dnaNextValue: 936100501

Note that the DNA range plugin has different values on each replica, and the ranges must not overlap and must not include already assigned IDs.

Another type of issue arises when the ID ranges are depleted. In this case, a new local id range can be added with a new range of values.

The official documentation contains a whole chapter related to Adjusting ID ranges manually and is a recommended read. I would also like to point to Rob’s blog “FreeIPA and no DNA range” for more details.

Advertisement

Troubleshooting ipa commands: read Apache error log and 389-ds access log

IdM includes an internal LDAP directory server instance where all the IdM information is stored: 389-ds. For instance the user and host entries reside in 389-ds as LDAP entries..

Most of the ipa commands are either reading or writing data in 389-ds. When an ipa command fails, 389-ds access log is really useful in order to understand if the failure is related to an issue in the LDAP server. This post will show how to used both Apache log and 389-ds access log to troubleshoot an issue.

Issue description

A Bugzilla reports that the command ipa domainlevel set behaves badly when the user tries to change the domain level but specifies the current level by mistake:

$ ipa domainlevel-get
-----------------------
Current domain level: 1
-----------------------
$ ipa domainlevel-set 0
ipa: ERROR: Domain Level cannot be lowered.

This result is expected as it is not possible to lower the current domain level.

$ ipa domainlevel-set 1
ipa: ERROR: Domain Level cannot be raised to 1, server master.example.com does not support it.

This result is wrong, as the domain level is already 1 and the following output is expected:

$ ipa domainlevel-set 1
ipa: ERROR: no modifications to be performed

Reading httpd error logs

By default, httpd error logs do not provide a lot of information (only messages with INFO or above). The logging level needs to be raised in order to have a hint of the root cause:

Create a file /etc/ipa/server.conf with the following content:

$ cat /etc/ipa/server.conf
[global]
debug=True

Restart FreeIPA stack:

$ ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
Starting pki-tomcatd Service
ipa: INFO: The ipactl command was successful

Re-run the ipa domainlevel-set 1 command, more information will get written in httpd error log:

$ tail -f /var/log/httpd/error_log
[...] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[...] ipa: DEBUG: WSGI jsonserver.__call__:
[...] ipa: DEBUG: KerberosWSGIExecutioner.__call__:
[...] ipa: DEBUG: Created connection context.ldap2_139798041384160
[...] ipa: DEBUG: WSGI WSGIExecutioner.__call__:
[...] ipa: DEBUG: raw: domainlevel_set(1, version='2.230')
[...] ipa: DEBUG: domainlevel_set(1, version='2.230')
[...] ipa: DEBUG: retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-DOMAIN-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7f25440f1208>
[...] ipa: DEBUG: WSGI wsgi_execute PublicError: Traceback (most recent call last):
[...] File "/usr/lib/python3.7/site-packages/ipaserver/rpcserver.py", line 370, in wsgi_execute
[...] result = command(*args, **options)
[...] File "/usr/lib/python3.7/site-packages/ipalib/frontend.py", line 450, in __call__
[...] return self.__do_call(*args, **options)
[...] File "/usr/lib/python3.7/site-packages/ipalib/frontend.py", line 478, in __do_call
[...] ret = self.run(*args, **options)
[...] File "/usr/lib/python3.7/site-packages/ipalib/frontend.py", line 800, in run
[...] return self.execute(*args, **options)
[...] File "/usr/lib/python3.7/site-packages/ipaserver/plugins/domainlevel.py", line 156, in execute
[...] raise errors.InvalidDomainLevelError(reason=message)
[...] ipalib.errors.InvalidDomainLevelError: Domain Level cannot be raised to 1, server master.domain.com does not support it.
[...]
[...] ipa: INFO: [jsonserver_kerb] host/master.domain.com@DOMAIN.COM: domainlevel_set/1(1, version='2.230'): InvalidDomainLevelError
[...] ipa: DEBUG: Destroyed connection context.ldap2_139798041384160

The log displays the command that was run and its parameters:

[...] ipa: DEBUG: raw: domainlevel_set(1, version='2.230')
[...] ipa: DEBUG: domainlevel_set(1, version='2.230')

and also the result:

[...] ipa: INFO: [jsonserver_kerb] host/master.domain.com@DOMAIN.COM: domainlevel_set/1(1, version='2.230'): InvalidDomainLevelError

The debug log shows a stack trace that can be examined, with the exact line where the failure occurred:

[...] File "/usr/lib/python3.7/site-packages/ipaserver/plugins/domainlevel.py", line 156, in execute
[...] raise errors.InvalidDomainLevelError(reason=message)

The code around this line looks like the following:

        # Check if every master supports the desired level
        for master in get_master_entries(ldap, self.api):
            supported = get_domainlevel_range(master)

            if supported.min > desired_value or supported.max < desired_value:
                message = _("Domain Level cannot be raised to {0}, server {1} "
                            "does not support it."
                            ).format(desired_value, master['cn'][0])
                raise errors.InvalidDomainLevelError(reason=message)

The get_domainlevel_range function is defined in the same file:

def get_domainlevel_range(master_entry):
    try:
        return DomainLevelRange(
            int(master_entry['ipaMinDomainLevel'][0]),
            int(master_entry['ipaMaxDomainLevel'][0])
        )
    except KeyError:
        return DomainLevelRange(0, 0)

get_domainlevel_range is reading ipaMinDomainLevel and ipaMaxDomainLevel from the master_entry. If the attributes are missing, the default values are set to 0.

What is this master_entry, and how is it built? The get_master_entries function is also defined in the same file:

def get_master_entries(ldap, api):
    """
    Returns list of LDAPEntries representing IPA masters.
    """

    container_masters = DN(
        ('cn', 'masters'),
        ('cn', 'ipa'),
        ('cn', 'etc'),
        api.env.basedn
    )

    masters, _dummy = ldap.find_entries(
        filter="(cn=*)",
        base_dn=container_masters,
        scope=ldap.SCOPE_ONELEVEL,
        paged_search=True,  # we need to make sure to get all of them
    )

    return masters

The function is simply performing a one-level search below cn=masters,cn=ipa,cn=etc,$BASEDN with a search filter “(cn=*)”. From there it’s easy to understand that the entries below cn=masters are supposed to contain the attributes ipaMinDomainLevel and ipaMaxDomainLevel.

389-ds logs will also help understand what happened for this search.

Reading 389-ds access logs

In a default IdM installation, 389-ds logs LDAP operations in /var/log/dirsv/slapd-DOMAIN-COM/access (the access logs is buffered).
To troubleshoot the problem, tail the access log and re-launch the command ipa domainlevel-set 1:

$ tail -f /var/log/dirsrv/slapd-DOMAIN-COM/access
[...] conn=23 fd=102 slot=102 connection from 10.37.171.82 to 10.37.171.82
[...] conn=23 op=0 BIND dn="" method=sasl version=3 mech=GSS-SPNEGO
[...] conn=23 op=0 RESULT err=0 tag=97 nentries=0 etime=0.0009926457 dn="fqdn=master.domain.com,cn=computers,cn=accounts,dc=domain,dc=com"
[...] conn=23 op=1 SRCH base="cn=ipaconfig,cn=etc,dc=domain,dc=com" scope=0 filter="(objectClass=*)" attrs=ALL
[...] conn=23 op=1 RESULT err=0 tag=101 nentries=1 etime=0.0002818162
[...] conn=23 op=2 SRCH base="cn=schema" scope=0 filter="(objectClass=*)" attrs="attributeTypes objectClasses"
[...] conn=23 op=2 RESULT err=0 tag=101 nentries=1 etime=0.0134877660
[...] conn=23 op=3 SRCH base="cn=Domain Level,cn=ipa,cn=etc,dc=domain,dc=com" scope=0 filter="(objectClass=*)" attrs=ALL
[...] conn=23 op=3 RESULT err=0 tag=101 nentries=1 etime=0.0001469752
[...] conn=23 op=4 SRCH base="cn=masters,cn=ipa,cn=etc,dc=domain,dc=com" scope=1 filter="(cn=*)" attrs=ALL
[...] conn=23 op=4 RESULT err=0 tag=101 nentries=1 etime=0.0002024975 notes=P pr_idx=0 pr_cookie=-1
[...] conn=23 op=5 UNBIND
[...] conn=23 op=5 fd=102 closed - U1

Let’s analyse the log from the beginning:

[...] conn=23 fd=102 slot=102 connection from 10.37.171.82 to 10.37.171.82

A LDAP client has opened a new connection (the client is running on the same machine as the server because the IP addresses are identical). In the following lines, the logs with an identical connection id (conn=23) will refer to operations sent using the same connection, meaning the same client.

[...] conn=23 op=0 BIND dn="" method=sasl version=3 mech=GSS-SPNEGO

The first operation (with op=0) executed by the LDAP client is a BIND using Kerberos authentication (mech=GSS-SPNEGO). In order to see the operation’s result, one needs to find in the log a line with the same tuple (connection id, op id):

[...] conn=23 op=0 RESULT err=0 tag=97 nentries=0 etime=0.0009926457 dn="fqdn=master.domain.com,cn=computers,cn=accounts,dc=domain,dc=com"

The logs show that the BIND was successful (RESULT err=0) and authenticated the connection as fqdn=master.domain.com,cn=computers,cn=accounts,dc=domain,dc=com. This is the host entry associated to the master. The kinit was probably done using kinit -kt /etc/krb5.keytab as this keytab contains the key for the local host.

Then the client performed a few searches, and it’s easy to spot the search performed by Apache server:

[...] conn=23 op=4 SRCH base="cn=masters,cn=ipa,cn=etc,dc=domain,dc=com" scope=1 filter="(cn=*)" attrs=ALL

This log corresponds to a one-level search (scope=1) on cn=masters,cn=ipa,cn=etc,$BASEDN (base=…) with the filter “(cn=*)”. The search returned success (RESULT err=0) and found a single entry (nentries=1):

[...] conn=23 op=4 RESULT err=0 tag=101 nentries=1 etime=0.0002024975 notes=P pr_idx=0 pr_cookie=-1

The log doesn’t show the content that was returned… But it’s possible to manually reproduce the search with a simple ldapsearch command:

$ kinit -kt /etc/krb5.keytab
$ ldapsearch -LLL -o ldif-wrap=no -Y GSSAPI -b cn=masters,cn=ipa,cn=etc,dc=domain,dc=com -s one "(cn=*)"
SASL/GSSAPI authentication started
SASL username: host/master.domain.com@DOMAIN.COM
SASL SSF: 256
SASL data security layer installed.
dn: cn=master.domain.com,cn=masters,cn=ipa,cn=etc,dc=domain,dc=com
objectClass: top
objectClass: nsContainer
objectClass: ipaReplTopoManagedServer
objectClass: ipaConfigObject
objectClass: ipaSupportedDomainLevelConfig
cn: master.domain.com

The above content is exactly the same as the content returned to Apache server. There is no ipaMinDomainLevel or ipaMaxDomainLevel in this entry, meaning that the code considered instead the default value of (0,0) for (supported.min, supported.max):

            if supported.min > desired_value or supported.max < desired_value:
                message = _("Domain Level cannot be raised to {0}, server {1} "
                            "does not support it."
                            ).format(desired_value, master['cn'][0])
                raise errors.InvalidDomainLevelError(reason=message)

With supported.max = 0, we have supported.max < desired_value and raise the Exception.

But why didn’t the ldap search return these attributes? The same request but authenticated as cn=Directory Manager will show the full entry content (access control do not apply to this user):

$ ldapsearch -D "cn=directory manager" -W -LLL -o ldif-wrap=no -b cn=masters,cn=ipa,cn=etc,dc=domain,dc=com -s one "(cn=*)"
Enter LDAP Password:
dn: cn=master.domain.com,cn=masters,cn=ipa,cn=etc,dc=domain,dc=com
objectClass: top
objectClass: nsContainer
objectClass: ipaReplTopoManagedServer
objectClass: ipaConfigObject
objectClass: ipaSupportedDomainLevelConfig
cn: master.domain.com
ipaReplTopoManagedSuffix: dc=domain,dc=com
ipaReplTopoManagedSuffix: o=ipaca
ipaMinDomainLevel: 1
ipaMaxDomainLevel: 1

The LDAP entry contains these attributes but they are filtered out from the entry returned if the search is authenticated as the local host. This is probably a simple ACI issue. To prove this assumption, repeat the ipa domainlevel-set 1 command, but this time authenticated as admin, who has more read permissions than the local host:

$ kinit admin
Password for admin@DOMAIN.COM:
$ ipa domainlevel-set 1
ipa: ERROR: no modifications to be performed

The mystery is solved! The operation was performed with the wrong user and did not have enough permissions to execute successfully.

Conclusion

When one is able to read both Apache error log and 389-ds access log, it is often easy to spot the exact cause for an issue within ipa commands. 389-ds logs display the command and the authenticated user, allowing to replay the same operation in order to see the actual result returned by 389-ds.

Demystifying the Certificate Authority component in FreeIPA

When I joined the FreeIPA team, I wanted to start by getting more familiar with the product from a user perspective and the first step was to install FreeIPA server.

I opened the Linux Domain Identity, Authentication, and Policy Guide, tried to figure out which options would be useful and… I froze when I reached the section Determining which CA configuration to use. I had literally no idea what the documentation meant by “Server with an integrated IdM CA” or “Server without a CA“. But I had to choose something to start with.
In this blog post, I will explain what this choice really means, and what are the consequences of picking one over the other.

Basic requirement: HTTP and LDAP server certificates

First of all, FreeIPA is composed of many services accessed through the network, among which a LDAP server and an HTTP server. These 2 services can be accessed through a standard port (in clear) or through a SSL port, meaning that they both need a server certificate.

The HTTP and LDAP server certificates are needed during the installation, because the installer will put them in the right NSS database and configure their nickname and location for the HTTP and LDAP servers to find them. There are multiple ways to obtain server certificates, but one needs to understand first the basic notions around Public Key Infrastructure (PKI).

I will use a comparison with the delivery of a passport: in order to have a passport issued with your name, you need to provide official documents (for instance a birth certificate and a photo) to the government agency that will validate the documents, make sure that you are who you claim to be, and then issue the passport. The server certificate can be compared to the passport, that will later prove your identity to whoever trusts the government agency, and the government agency can be compared to the Certificate Authority.

So in order to obtain server certificates, it is possible to:

• request certificates to an official Certificate Authority. Many commercial or non-profit companies provide this type of service (Verisign, Let’sEncrypt, GoDaddy etc…)
• request certificates to a home-made Certificate Authority. It is possible to create a home-made self-signed Certificate Authority with tools such as certutil or openssl. The main difference with the previous method is that people are less likely to trust your home-made CA (it’s like asking them to accept a passport that was issued by a newly founded country not recognized yet by the rest of the world). Self-signed here means that the Certificate Authority passport  is delivered by… the Certificate Authority itself!
• install your own Certificate Authority with FreeIPA, that will sign the certificates needed by the HTTP and LDAP server.

The last option corresponds to a “Server with an integrated IdM CA” and has many advantages over the first options:

• certificates have a limited lifetime and need to be renewed before they expire (otherwise the HTTP/LDAP servers stop working). The renewal process is time-consuming but also can be forgotten if the sysadmin does not carefully track the dates, leaving the deployment in a state where some services become unavailable.
  When FreeIPA is installed with an embedded Certificate Authority, FreeIPA automatically monitors the expiry dates of the certificates and triggers a renewal a few weeks before expiration, ensuring service continuity. This is the first advantage of the embedded CA, but not the only one.
• FreeIPA with an embedded CA is also able to deliver certificates for the users, the hosts or the services managed by FreeIPA. Various certificates profiles can be defined, populating specific fields in each type of certificate (for instance extensions with the OCSP responder URL…)
  For more information on the embedded Certificate Authority, you can refer to Dogtag Certificate System documentation (the embedded CA in FreeIPA is a tailored version of Dogtag).

At this point, if you decide to install FreeIPA with an embedded Certificate Authority, there are 2 possible choices. The embedded CA can either be:

• a self-signed CA: the Certificate Authority is created “from scratch” without the need for any external authority. It is the root CA, meaning that its own certificate was not delivered by anyone else but signed by itself.
• a CA subordinate to an external CA. This means that FreeIPA CA certificate was signed by another CA, a sort of parent CA.

 

Corresponding installation options

CA-less installation

As said above, we need one certificate for the HTTP server and one for the LDAP server.  They have to be provided to ipa-server-install or ipa-replica-install with the options:

• –http-cert-file / –http-pin: file containing the HTTP server certificate + private key and password protecting the file
• –dirsrv-cert-file / –dirsrv-pin: file containing the LDAP server certificate + private key and password protecting the file

Installation with an embedded self-signed CA

FreeIPA CA is created during FreeIPA installation, and generates the HTTP and LDAP certificates. There is no need to provide any cert file! No options!

Installation with an externally-signed embedded CA

The installation is a 2-step process. In the first step, ipa-server-install must be called with –external-ca and generates a CSR file (Certificate Signing Request). This CSR file needs to be sent to the external CA that will perform a bunch of validations to authenticate the recipient of the certificate and issue a certificate for FreeIPA Certificate Authority.

In the second step, ipa-server-install is called with –external-cert-file to provide the certificate obtained from the external CA. The installer then configures FreeIPA certificate authority as a subCA of the external CA, and FreeIPA CA can issue the HTTP and LDAP server certificates.

 

What if…

I installed FreeIPA without any embedded CA but I change my mind?

FreeIPA allows to install an embedded CA at a later time, using ipa-ca-install. The tool provides the same options as ipa-server-install: you can either install a self-signed CA or an externally signed CA.

Important: installing an embedded CA with ipa-ca-install does not replace the HTTP and LDAP server certificates. If they were initially delivered by an external CA, they will not be automatically renewed.

I installed FreeIPA with a self-signed CA but I’d rather have an externally-signed CA?

FreeIPA allows to switch from self-signed CA to externally-signed CA using ipa-cacert-manage renew –external-ca. This is a 2-step process similar to ipa-server-install –external-ca, where the 1st step produces a CSR that needs to be supplied to an external CA. The external CA then issues a CA cert that is provided back to ipa-cacert-manage renew through the –external-cert-file option.

I installed FreeIPA with an externally signed CA but I’d rather have a self-signed CA?

FreeIPA allows to switch from externally signed CA to self-signed CA using ipa-cacert-manage renew –self-signed.

I want to replace HTTP and LDAP certificates with certificates signed by a third-party Certificate Authority?

FreeIPA provides the ipa-server-certinstall tool that will replace the current HTTP or LDAP certificates with the certs provided in the file.
Important: ipa-server-certinstall can be called to install externally signed certificates for HTTP and LDAP even if FreeIPA is installed with an embedded CA. In this case, FreeIPA CA still has the capability to issue certificates for users, hosts or services. The only difference is that HTTP and LDAP certificates are not issued by IPA CA.

 

Other FreeIPA commands related to certificates

When an embedded CA is installed, its certificate must be present in various files or NSS databases on all the FreeIPA hosts (master, replicas and clients) so that any FreeIPA machine trusts the certificates delivered by the embedded CA.

In addition to that, the HTTP and LDAP server certificates can be issued either by IPA CA or by an external CA, and the issuer can even be changed over time. In the external CA case, this means that the external CA needs to be trusted by all the FreeIPA machines for the IPA commands to work (the CLI communicates with the HTTP server using the https port, and this requires to trust the CA that issued the HTTP server certificate). Yet another CA certificate to add to files and databases on all the FreeIPA machines…

To ease this configuration, the tool ipa-certupdate is able to retrieve the CA certificates stored in LDAP (the embedded FreeIPA CA or the external CA certs), and install them in all the relevant files and NSS databases. It needs to be called when the CA cert is manually renewed or when a new external CA cert is added.

ipa-cacert-manage install is used to add a new external CA certificate in the LDAP store. It does not replace FreeIPA embedded CA but rather declares another certificate authority as trusted. This is useful when the HTTP and LDAP server certificates need to be replaced by certs signed by a new CA, not yet known by FreeIPA. After calling ipa-cacert-manage install (that puts the new CA in LDAP store), you need to call ipa-certupdate on all FreeIPA machines (to get the CA from the LDAP store and put it in the local NSS databases).

ipa cert-request is used to request new certificates for users, hosts or services. The certificate is signed by FreeIPA embedded CA (meaning that this command is available only when an embedded CA is configured).

Conclusion

By now you should be able to pick a deployment option and understand the differences between CA-less or with embedded CA, and self-signed CA or externally-signed CA.

You should also be aware that your choice is not definitive and that it is possible to install a CA at a later time or change your certificate chain from self-signed to externally-signed and vice-versa.

Troubleshooting FreeIPA: pki-tomcatd fails to start

When performing the upgrade of FreeIPA, you may encounter an issue with pki-tomcatd failing to start. At first this issue looks related to the upgrade, but often reveals a latent problem and gets detected only because the upgrade triggers a restart of pki-tomcatd.

So how to troubleshoot this type of issue?

 

Upgrade logs

The upgrade is using /var/log/ipaupgrade.log and may contain a lot of useful information. In this specific case, I could see:

[...] DEBUG The ipa-server-upgrade command failed,
exception: ScriptError: CA did not start in 300.0s
[...] ERROR CA did not start in 300.0s
[...] ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information

 

CA debug logs

The first step is to understand why pki-tomcatd refuses to start. This process is launched inside Tomcat and corresponds to the CA component of FreeIPA. It is logging into /var/log/pki/pki-tomcat/ca/debug:

[...][localhost-startStop-2]: ============================================
[...][localhost-startStop-2]: ===== DEBUG SUBSYSTEM INITIALIZED =======
[...][localhost-startStop-2]: ============================================
[...][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false
[...][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[...][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: done init id=debug
[...][localhost-startStop-2]: CMSEngine: initialized debug
[...][localhost-startStop-2]: CMSEngine: initSubsystem id=log
[...][localhost-startStop-2]: CMSEngine: ready to init id=log
[...][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
[...][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
[...][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
[...][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false
[...][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[...][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: done init id=log
[...][localhost-startStop-2]: CMSEngine: initialized log
[...][localhost-startStop-2]: CMSEngine: initSubsystem id=jss
[...][localhost-startStop-2]: CMSEngine: ready to init id=jss
[...][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false
[...][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[...][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: done init id=jss
[...][localhost-startStop-2]: CMSEngine: initialized jss
[...][localhost-startStop-2]: CMSEngine: initSubsystem id=dbs
[...][localhost-startStop-2]: CMSEngine: ready to init id=dbs
[...][localhost-startStop-2]: DBSubsystem: init() mEnableSerialMgmt=true
[...][localhost-startStop-2]: Creating LdapBoundConnFactor(DBSubsystem)
[...][localhost-startStop-2]: LdapBoundConnFactory: init
[...][localhost-startStop-2]: LdapBoundConnFactory:doCloning true
[...][localhost-startStop-2]: LdapAuthInfo: init()
[...][localhost-startStop-2]: LdapAuthInfo: init begins
[...][localhost-startStop-2]: LdapAuthInfo: init ends
[...][localhost-startStop-2]: init: before makeConnection errorIfDown is true
[...][localhost-startStop-2]: makeConnection: errorIfDown true
[...][localhost-startStop-2]: TCP Keep-Alive: true
[...][localhost-startStop-2]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca
[...][localhost-startStop-2]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca
[...][localhost-startStop-2]: SSL handshake happened
Could not connect to LDAP server host ipaserver.ipadomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
 at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
 at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
 at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
 at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
 at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172)
 at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078)
 at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570)
 at com.netscape.certsrv.apps.CMS.init(CMS.java:188)
 at com.netscape.certsrv.apps.CMS.start(CMS.java:1621)
 at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
 at javax.servlet.GenericServlet.init(GenericServlet.java:158)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
 at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
 at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
 at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
 at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
 at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
 at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
 at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
 at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
 at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
 at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
 at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)

The exception shows that LDAP authentication failed with return code 49: invalid credentials.

 

Communication between pki-tomcatd and the LDAP server

We can see that pki-tomcatd is trying to open a LDAP connection through SSL but fails to authenticate. Within FreeIPA, pki-tomcat is storing data in the 389-ds LDAP server and needs to communicate through LDAP with this server.

The configuration of this communication channel can be read in /etc/pki/pki-tomcat/ca/CS.cfg:

internaldb.ldapauth.authtype=SslClientAuth
internaldb.ldapauth.bindDN=cn=Directory Manager
internaldb.ldapauth.bindPWPrompt=internaldb
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
internaldb.ldapconn.host=ipaserver.ipadomain.com
internaldb.ldapconn.port=636
internaldb.ldapconn.secureConn

The connection is using port 636 (SSL port) with SSL Client authentication (authtype=SslClientAuth). This means that pki-tomcatd provides a user certificate to the LDAP server, and the LDAP server maps a user to this certificate in order to authenticate the communications.

Note: Authtype can either be SslClientAuth or BasicAuth (authentication with username and password).

In this case, the SSL client authentication is done with the certificate named ‘subsystemCert cert-pki-ca‘ that is stored in /etc/pki/pki-tomcat/alias. So what could be causing the authentication to fail? We need to check that the certificate is available in /etc/pki/pki-tomcat/alias and that pki-tomcat is able to use the associated private key, and that the LDAP server is able to map this certificate to a user.

 

Check the subsystemCert cert-pki-ca

The first step consists in making sure that this certificate is present in /etc/pki/pki-tomcat/alias:

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
Certificate:
 Data:
 Version: 3 (0x2)
...

 

Then make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=…)

$ sudo grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt
$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa 86a7fe00cc2a01ad085f35d4ed3e84e7b82ab4f5 subsystemCert cert-pki-ca

At this point we know that pki-tomcat is able to access the certificate and the private key. So the issue is likely to be on the LDAP server side.

Note: depending on your version, the nickname of the certificate may also be prefixed with ‘NSS Certificate DB:’. In this case, run the following command:

$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'NSS Certificate DB: subsystemCert cert-pki-ca'

 

LDAP server configuration

The LDAP configuration describes how a certificate can be mapped to a user in /etc/dirsrv/slapd-IPADOMAIN-COM/certmap.conf:

$ sudo cat /etc/dirsrv/slapd-IPADOMAIN-COM/certmap.conf 
[...]
certmap default default
[...]
default:DNComps
default:FilterComps uid
certmap ipaca CN=Certificate Authority,O=IPADOMAIN.COM
ipaca:CmapLdapAttr seeAlso
ipaca:verifycert on

This means that when the LDAP server receives an authentication request with a certificate issued by the CA CN=Certificate Authority,O=IPADOMAIN.COM, it will look for users that contain a seeAlso attribute equal to the subject of the certificate, and the user entry must contain the certificate in the usercertificate attribute (verifycert: on).

With a default config, the ‘subsystemCert cert-pki-ca‘ is mapped to the user uid=pkidbuser,ou=people,o=ipaca. So let’s compare the user entry and the certificate:

$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password: 
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIID...uwab3
description: 2;4;CN=Certificate Authority,O=IPADOMAIN.COM;CN=CA Subsystem,O=IPADOMAIN.COM
seeAlso: CN=CA Subsystem,O=IPADOMAIN.COM


$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIID...e5QAR
-----END CERTIFICATE-----

The certificate in the userCertificate attribute is different from the one in the NSS database! This can also be seen by comparing the serial number with the value from the ldap entry (in description: 2;<serial>;<issuer>;<subject>):

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial
 Serial Number: 1341718541 (0x4ff9000d

This explains why pki-tomcat could not authenticate to the LDAP server. The fix consists in updating the LDAP entry with the right certificate (and do not forget to update the description attribute with the right serial number!):

$ cat updatecert.ldif
dn: uid=pkidbuser,ou=people,o=ipaca
changetype: modify
replace: usercertificate
usercertificate::MII... <here put the new cert in a single line. The double colon :: is important as the format is binary>
-
replace: description
description: 2;<here put the new serial number>;CN=Certificate Authority,O=IPADOMAIN.COM;CN=CA Subsystem,O=IPADOMAIN.COM


$ ldapmodify -D "cn=directory manager" -W -f updatecert.ldif

But we still do not know the root cause for the inconsistency between the NSS database /etc/pki/pki-tomcat/alias and the user entry for uid=pkidbuser. This could be described in another blog post (for the impatients, the automatic renewal of the certificate failed to update the LDAP server entry…)

Troubleshooting: mapping between a SmartCard certificate and an IdM user

Authentication with a SmartCard may fail when the SmartCard certificate is not linked to any IdM user, or to a user different from the one specified on the console.

In order to find which user is associated to a given SmartCard certificate, you can run the following command:

ipaclient$ ipa certmap-match cert.pem
--------------
1 user matched
--------------
 Domain: IPADOMAIN.COM
 User logins: demosc1
----------------------------
Number of entries returned 1
----------------------------

If the result is not what you were expecting, you need first to check how certificates are mapped to users.

By default, a certificate is associated to a user when the user entry contains the full certificate in its usercertificate attribute. But this behavior can be modified by defining certificate mapping rules:

ipaclient$ ipa certmaprule-find
-------------------------------------------
1 Certificate Identity Mapping Rule matched
-------------------------------------------
 Rule name: rulesmartcard
 Mapping rule: (ipacertmapdata=X509:<I>{issuer_dn}<S>{subject_dn})
 Matching rule: <ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG
 Enabled: FALSE
----------------------------
Number of entries returned 1
----------------------------

 

Mapping with full certificate content

When the mapping is based on the full certificate content, you can check if the user entry contains the certificate:

root@ipaclient$ ipa user-show demosc1
 User login: demosc1
[...]
Certificate: MIIC...

If it is not the case, associate the certificate with the user entry using:

ipaclient$ CERT=`cat cert.pem | tail -n +2 | head -n -1 | tr -d '\r\n'`
ipaclient$ ipa user-add-cert demosc1 --certificate $CERT

Once this is done, you may need to clear sssd cache to force SSSD to reload the entries before retrying ipa certmap-match:

ipaclient$ sudo sss_cache -E

 

Flexible mapping with certificate identity mapping rule

When the mapping is based on certificate mapping rules, the same tool ipa certmap-match can be used to check which user entry is associated to a certificate. When the result is not what you expect, you can enable sssd domain logs by adding the following in /etc/sssd/sssd.conf on IdM master:

[domain/ipadomain.com]
...
debug_level = 9

then restart sssd with

root@ipaserver$ systemctl restart sssd

The logs will be located in /var/log/sssd/sssd_ipadomain.log.

 

Check that the certificate identity mapping rules are properly loaded

When sssd is restarted, it reads the mapping rules and should print the following in /var/log/sssd/sssd_ipadomain.log:

[sssd[be[ipadomain.com]]] [sss_certmap_init] (0x0040): sss_certmap initialized.
[sssd[be[ipadomain.com]]] [ipa_certmap_parse_results] (0x4000): Trying to add rule [rulesmartcard][-1][<ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG][(ipacertmapdata=X509:<I>{issuer_dn!nss_x500}<S>{subject_dn!nss_x500})].

If the rule has an invalid syntax, you will see instead:

[sssd[be[ipadomain.com]]] [sss_certmap_init] (0x0040): sss_certmap initialized.
[sssd[be[ipadomain.com]]] [ipa_certmap_parse_results] (0x4000): Trying to add rule [rulesmartcard][-1][<ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG][(ipacertmapdata=X509:<I>{issuer_dn!x509}<S>{subject_dn})].
[sssd[be[ipadomain.com]]] [parse_template] (0x0040): Parse template invalid.
[sssd[be[ipadomain.com]]] [parse_ldap_mapping_rule] (0x0040): Failed to add template.
[sssd[be[ipadomain.com]]] [parse_mapping_rule] (0x0040): Failed to parse LDAP mapping rule.
[sssd[be[ipadomain.com]]] [ipa_certmap_parse_results] (0x0020): sss_certmap_add_rule failed for rule [rulesmartcard], skipping. Please check for typos and if rule syntax is supported.
[sssd[be[ipadomain.com]]] [ipa_subdomains_certmap_done] (0x0040): Unable to parse certmap results [22]: Invalid argument
[sssd[be[ipadomain.com]]] [ipa_subdomains_refresh_certmap_done] (0x0020): Failed to read certificate mapping rules [22]: Invalid argument

The log shows that the rule named rulesmartcard is invalid. Check the rule (see man page for sss-certmap for the supported syntax) and fix if needed:

ipaclient$ ipa certmaprule-show rulesmartcard
 Rule name: rulesmartcard
 Mapping rule: (ipacertmapdata=X509:<I>{issuer_dn!x509}<S>{subject_dn})
 Matching rule: <ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG
 Enabled: TRUE
ipaclient$ ipa certmaprule-mod rulesmartcard --maprule '(ipacertmapdata=X509:<I>{issuer_dn}<S>{subject_dn})'

 

Check that the matching rule corresponds to the certificate

When SSSD tries to associate the certificate to a user, it starts by finding which rule can be applied based on the matching rule (for instance rulesmartcard applies to all certificates issued by CN=Smart Card CA,O=EXAMPLE.ORG because its matching rule is <ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG).

If no matching rule applies to the certificate, SSSD will not be able to associate the certificate with a user, and will display the following in /var/log/sssd/sssd_ipadomain.log:

[sssd[be[ipadomain.com]]] [sss_cert_derb64_to_ldap_filter] (0x0040): Certificate does not match matching-rules.

In this case, you need to create or modify an identity mapping rule, so that the match rule applies to your certificate. See sss-certmap man page for the supported syntax of the –matchrule option of ipa certmaprule-add command.

Check that the expected certificate identity mapping rule is used

When SSSD tries to find the user associated to the certificate, you will see the following logs in /var/log/sssd/sssd_ipadomain.log:

[sssd[be[ipadomain.com]]] [dp_get_account_info_handler] (0x0200): Got request for [0x14][BE_REQ_BY_CERT][cert=MII..]
...
[sssd[be[ipadomain.com]]] [sdap_search_user_next_base] (0x0400): Searching for users with base [cn=accounts,dc=ipadomain,dc=com]
[sssd[be[ipadomain.com]]] [sdap_print_server] (0x2000): Searching 10.34.58.20:389
[sssd[be[ipadomain.com]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(ipacertmapdata=X509:<I>O=EXAMPLE.ORG,CN=Smart Card CA<S>CN=test,O=EXAMPLE.ORG)(objectclass=posixAccount)(uid=*)(&(uidNumber=*)(!(uidNumber=0))))][cn=accounts,dc=ipadomain,dc=com].
...
[sssd[be[ipadomain.com]]] [sdap_search_user_process] (0x0400): Search for users, returned 0 results.

The logs show the LDAP search performed by SSSD: IP address of the LDAP server, base and search filter. Carefully review this information and compare with what you would expect.

Check that the mapping rule defines a valid search filter

If the rule cannot be transformed to a valid search filter, you will see in /var/log/sssd/sssd_ipadomain.log:

[sssd[be[ipadomain.com]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(ipacertmapdata=X509:<I>O=EXAMPLE.ORG,CN=Smart Card CA<S>CN=test,O=EXAMPLE.ORG(objectclass=posixAccount)(uid=*)(&(uidNumber=*)(!(uidNumber=0))))][cn=accounts,dc=ipadomain,dc=com].
[...]
[sssd[be[ipadomain.com]]] [sdap_get_generic_ext_step] (0x0080): ldap_search_ext failed: Bad search filter

If it is the case, you need to fix the certmap rule using

ipaclient$ ipa certmaprule-mod rulesmartcard –maprule …

 

 

Troubleshooting: authentication to the system console or Gnome Desktop Manager of an IdM host with a SmartCard

IdM allows to authenticate to an IdM enrolled-host by providing a SmartCard certificate instead of a username/password. The below steps are based on system console authentication but the process is similar when using Gnome desktop  login authentication.

When the authentication fails, the issue usually comes from a wrong configuration of the IdM system for SmartCard, or of PKINIT.

 

Configuration of the IdM host for SmartCard authentication

If the console does not even prompt for the SmartCard PIN, chances are high that the system was not properly configured for SmartCard authentication.

SSSD configuration for smart card

Check that /etc/sssd.conf contains

[pam]
pam_cert_auth = True

If you need to update the file, do not forget to restart sssd with

root@ipaclient$ systemctl restart sssd

 

SmartCard CA must be trusted

Check that the SmartCard CA is trusted in the /etc/pki/nssdb database:

root@ipaclient$ certutil -L -d /etc/pki/nssdb/

Certificate Nickname Trust Attributes     SSL,S/MIME,JAR/XPI

SmartCardCA                               CT,C,C

If the CA is not present, add it using:

root@ipaclient$ certutil -A -d /etc/pki/nssdb -n SmartCardCA -t CT,C,C -i ca.pem

 

IdM host PKCS#11 module

Check that the IdM host is properly configured for Smart Cards. The opensc package must be installed, the the SmartCard daemon must be running, and the PKCS#11 module must be loaded

root@ipaclient$ dnf install opensc
root@ipaclient$ systemctl start pcscd.service pcscd.socket
root@ipaclient$ modutil -dbdir /etc/pki/nssdb -add "OpenSC" -libfile /usr/lib64/opensc-pkcs11.so

 

Configuration for PKINIT

If the console prompts for the SmartCard PIN but displays

ipaclient login: demosc1
Pin for PIV Card:
Login incorrect

it is possible that the authentication fails trying to acquire a Kerberos ticket with PKINIT. In this case, login with username/password to the IdM host and try to manually perform kinit in order to get more information:

root@ipaclient$ kinit -X X509_user_identity='PKCS11:opensc-pkcs11.so' demosc1

 

If  the command outputs the following:

kinit: Pre-authentication failed: Failed to verify own certificate (depth 1): self signed certificate in certificate chain while getting initial credentials

then check the content of /etc/krb5.conf on the IdM host. The realms section must contain a configuration for ipadomain with pkinit_anchors:

[realms]
 IPADOMAIN.COM = {
 pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
 pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem

}

The file defined in pkinit_anchors must exist, be readable and contain the certificate of the CA which signed the SmartCard certificate. If it is not the case, run the following commands on any IPA server:

root@ipaserver$ ipa-cacert-manage install -n SmartCardCA -t CT,C,C -p $DM_PWD ca.pem
root@ipaserver$ ipa-certupdate

And run the ipa-certupdate command on all IdM hosts in order to download the certificate.

If the kinit command output the following:

kinit: Certificate mismatch while getting initial credentials

check that the SmartCard certificate is associated to the username provided in the console (see mapping between a SmartCard certificate and an IdM user).

Troubleshooting: ssh to an IdM host with a SmartCard

IdM allows to perform ssh from a non-enrolled host into an IdM enrolled host, using Smart Card authentication instead of ssh authorized keys. The ssh command would be the following to log as demosc1 into the host ipaclient.ipadomain.com:

localuser@localhost$ ssh -I /usr/lib64/opensc-pkcs11.so -l demosc1 ipaclient.ipadomain.com
Enter PIN for 'PIV_II (PIV Card Holder pin)':

The -I option specifies a PKCS#11 shared library, and -l the username on the remote host.

 

Configuration of the local host

First check that the local host is properly configured for Smart Cards. The opensc package must be installed, and the the SmartCard daemon must be running.

localuser@localhost$ sudo dnf install opensc
localuser@localhost$ sudo systemctl start pcscd.service pcscd.socket

 

Configuration of the remote (IdM) host

When IdM is properly configured, ssh will prompt for the SmartCard PIN and authenticate the user. If there is an issue with the certificate, ssh will revert to another authentication type (private keys or username/password).

In this case, enable debug logs for ssh authentication on IdM host. Edit /etc/sssd/sssd.conf and add the following line in [ssh] section:

[ssh]
debug_level = 9

then restart sssd using

root@ipaclient$ systemctl restart sssd

The logs will be located on the IdM host in /var/log/sssd/sssd_ssh.log.

 

The Smart Card CA is not trusted by SSSD

If you see the following in /var/log/sssd/sssd_ssh.log:

[sssd[ssh]] [cert_to_ssh_key] (0x0020): CERT_VerifyCertificateNow failed [-8179].
[sssd[ssh]] [get_valid_certs_keys] (0x0040): cert_to_ssh_key failed, ignoring.

then it means that the CA that signed the Smart Card certificate is not trusted. The trusted certs are stored in /etc/pki/nssdb and can be found using:

root@ipaclient$ certutil -L -d /etc/pki/nssdb

Certificate Nickname Trust Attributes     SSL,S/MIME,JAR/XPI

SmartCardCA                               CT,C,C

 

If the CA cert is missing, add it using

root@ipaclient$ certutil -A -d /etc/pki/nssdb -n SmartCardCA -t CT,C,C -i ca.pem

 

The user is not an IdM user

If the ssh operation does not log any line in /var/log/sssd/sssd_ssh.log, it probably means that the supplied user name is not a user defined in IdM. You can check with:

root@ipaclient$ ipa user-find demosc1
---------------
0 users matched
---------------
----------------------------
Number of entries returned 0
----------------------------

Check that you provided the right user name, or define an IdM user and associate the SmartCard certificate with this user.

 

The certificate is not mapped to the IdM user

If you see the following in /var/log/sssd/sssd_ssh.log:

Found 1 entries in domain ipadomain.com

but the authentication fails, check that the SmartCard certificate is associated to the provided username (refer to mapping between a SmartCard certificate and an IdM user)

Troubleshooting: Authentication to IdM WebGUI with a SmartCard

IdM web GUI can be accessed at the following url: https://ipaserver.ipadomain.com/ipa/ui

The authentication can be done either through Kerberos, by providing a username and password, or with a certificate. If the certificate authentication fails, the issue can either come from your local browser configuration, or from IdM configuration.

The browser does not prompt for the SmartCard PIN

In this case, it is likely that your browser is not properly configured for Smart Card authentication. Please refer to the browser documentation.

 

The browser prompts for the SmartCard PIN but authentication fails

The issue is likely to be a configuration problem on IdM server. The file /var/log/http/error_log will be your best friend to diagnose and fix the issue.

The root CA is not trusted by IdM server

If you see the following in /var/log/http/error_log:

[:error] [pid 50892] Re-negotiation handshake failed: Not accepted by client!?
[:error] [pid 50892] SSL Library Error: -12285 Unable to find the certificate or key necessary for authentication

then it means that the CA that issued your Smart Card certificate is not trusted by Apache httpd server. You need to run the following command on IdM master:

root@ipaserver$ ipa-cacert-manage install -n SmartCardCA -t CT,C,C -p $DM_PWD ca.pem
root@ipaserver$ ipa-certupdate
root@ipaserver$ systemctl restart httpd

The first command adds the Smart Card CA as a trusted CA into IdM. The second command installs the certificate in the various databases used by IdM, especially in /etc/httpd/alias used by Apache. The third command restarts Apache Web server.

The ipa-certupdate step must be run on all IdM hosts, and Apache must be restarted on all IdM servers.

 

Apache is not allowed to authenticate on behalf of a client

If you see the following log in /var/log/http/error_log:

[auth_gssapi:error] [pid 55807] [client 10.40.204.99:60644] GSS ERROR gss_init_sec_context(): [Unspecified GSS failure. Minor code may provide more information (Credential cache is empty)], referer: https://ipaserver.ipadomain.com/ipa/ui/
[:error] [pid 55743] ipa: INFO: 401 Unauthorized: KRB5CCNAME not set

then it means that you forgot to configure apache for delegation. You need to run the following command:

root@ipaserver$ ipa service-mod --ok-to-auth-as-delegate=True HTTP/$(hostname)
root@ipaserver$ systemctl restart httpd

on each IdM server.

 

The SmartCard certificate does not match any IdM user

If you see the following log in /var/log/http/error_log:

[lookup_identity:error] [pid 69382] [client 10.40.204.99:39400] lookup_user_by_certificate failed [dbus_connection_send_with_reply_and_block(org.freedesktop.sssd.infopipe.Users.FindByNameAndCertificate)]: [User not found], referer: https://ipaserver.ipadomain.com/ipa/ui/
[lookup_identity:error] [pid 69382] [client 10.40.204.99:39400] lookup_user_by_certificate cleared r->user, referer: https://ipaserver.ipadomain.com/ipa/ui/
[core:error] [pid 69382] [client 10.40.204.99:39400] AH00027: No authentication done but request not allowed without authentication for /ipa/session/login_x509. Authentication not configured?, referer: https://ipaserver.ipadomin.com/ipa/ui/

then it means that IPA did not find any user associated to the provided certificate. Check  mapping between a SmartCard certificate and an IdM user.

 

The SmartCard certificate matches more than one user

If you see the following log in /var/log/http/error_log:

[lookup_identity:error] [pid 70244] [client 10.40.204.99:39458] lookup_user_by_certificate failed [dbus_connection_send_with_reply_and_block(org.freedesktop.sssd.infopipe.Users.FindByNameAndCertificate)]: [More than one user found. Use ListByCertificate to get all.], referer: https://ipaserver.ipadomain.com/ipa/ui/
[lookup_identity:error] [pid 70244] [client 10.40.204.99:39458] lookup_user_by_certificate cleared r->user, referer: https://ipaserver.ipadomain.com/ipa/ui/
[core:error] [pid 70244] [client 10.40.204.99:39458] AH00027: No authentication done but request not allowed without authentication for /ipa/session/login_x509. Authentication not configured?, referer: https://ipaserver.ipadomain.com/ipa/ui/

then check if the certificate is mapped to a single user using

root@ipaserver$ ipa certmap-match cert.pem
--------------
1 user matched
--------------
 Domain: IPADOMAIN.COM
 User logins: demosc1
----------------------------
Number of entries returned 1
----------------------------

If the certificate matches multiple users, you need to specify the username in the Web GUI.

Apache mod_lookup_identity badly configured

If you see the following in /var/log/http/error_log:

[lookup_identity:error] [pid 70069] [client 10.40.204.99:39444] lookup_user_by_certificate failed [dbus_connection_send_with_reply_and_block(org.freedesktop.sssd.infopipe.Users.FindByNameAndCertificate)]: [More than one user found. Use ListByCertificate to get all.], referer: https://ipaserver.ipadomain/ipa/ui/
[lookup_identity:error] [pid 70069] [client 10.40.204.99:39444] lookup_user_by_certificate cleared r->user, referer: https://ipaserver.ipadomain.com/ipa/ui/
[core:error] [pid 70069] [client 10.40.204.99:39444] AH00027: No authentication done but request not allowed without authentication for /ipa/session/login_x509. Authentication not configured?, referer: https://ipaserver.ipadomain.com/ipa/ui/

then check that /etc/httpd/conf.d/ipa.conf contains the following:

<Location "/ipa/session/login_x509">
...
 LookupUserByCertificate On
 LookupUserByCertificateParamName "username"
...

 

If you see the following in /var/log/http/error_log:

[auth_gssapi:error] [pid 70144] [client 10.40.204.99:39454] GSS ERROR In S4U2Self: gss_acquire_cred_impersonate_name(): [A required input parameter could not be read, No credentials were supplied, or the credentials were unavailable or inaccessible (Unknown error)], referer: https://ipaserver.ipadomain.com/ipa/ui/
[:error] [pid 70141] ipa: INFO: 401 Unauthorized: KRB5CCNAME not set

then check that /etc/httpd/conf.d/ipa.conf contains the following:

<Location "/ipa/session/login_x509">
...
 LookupUserByCertificate On
 LookupUserByCertificateParamName "username"
...

FreeIPA: troubleshooting SmartCard authentication

RHEL 7.4 beta is now available, delivering a new version of IPA which contains the support for Smart Card authentication. This feature allows to use a certificate contained in a SmartCard in order to login to IdM WebUI, to ssh to an IdM-enrolled host, or to login to the console or Gnome Desktop Manager of an IdM-enrolled host.

This feature is really powerful but may also seem difficult to troubleshoot. I will explain where to look for additional information when authentication fails, and how to fix the most common issues.

The information is split into posts specific to each authentication method:

• authentication to IdM WebGUI with a SmartCard
• ssh to an IdM host with a SmartCard
• authentication to the system console or Gnome Desktop Manager of an IdM host with a SmartCard
• mapping between a SmartCard certificate and an IdM user

 

 

Debug ipasam plugin for Samba

Last week I was working on an issue involving ipasam plugin for Samba. I started by enabling more logs:

$ cat /etc/ipa/default.conf
[global]
...
debug=True
$ ipactl restart
$ net conf setparm global "log level" 20

Note: if you add the debug=True parameter in /etc/ipa/default.conf, the settings applies to both IPA server and IPA client, meaning that ipa command line is also set into debug mode (and can be quite verbose…). To set only the server in debug mode, create or edit /etc/ipa/server.conf:

$ cat /etc/ipa/server.conf
[global]
...
debug=True

The logs were then in /var/log/samba. I could see traces corresponding to my issue logged in log.smbd.lsasd.<id> and decided to attach my debugger. The problem was that samba had forked a set of processes and it was impossible to know in advance which one would be used to process the requests.

A simple way to solve this issue is to configure the maximum number of processes forked for lsasd with the following command:

$ net conf setparm global "lsasd:prefork_max_children" 1
$ net conf setparm global "lsasd:prefork_min_children" 1
$ systemctl restart smb

This way, only one process will be used for lsasd. Its pid can be seen in /var/log/samba/log.smbd.lsasd.1:

[2017/03/22 10:19:56.973864, 10, pid=3982, effective(0, 0), real(0, 0)] ../source3/rpc_server/lsasd.c:237(parent_ping)
 Got message that the parent changed status.

Once the pid is known, it is possible to attach the debugger with

$ gdb -p 3982

At that point, it is easy to add breakpoints and debug with the usual methods!