Category Archives: FreeIPA

Troubleshooting FreeIPA: pki-tomcatd fails to start

When performing the upgrade of FreeIPA, you may encounter an issue with pki-tomcatd failing to start. At first this issue looks related to the upgrade, but often reveals a latent problem and gets detected only because the upgrade triggers a restart of pki-tomcatd.

So how to troubleshoot this type of issue?


Upgrade logs

The upgrade is using /var/log/ipaupgrade.log and may contain a lot of useful information. In this specific case, I could see:

[...] DEBUG The ipa-server-upgrade command failed,
exception: ScriptError: CA did not start in 300.0s
[...] ERROR CA did not start in 300.0s
[...] ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information


CA debug logs

The first step is to understand why pki-tomcatd refuses to start. This process is launched inside Tomcat and corresponds to the CA component of FreeIPA. It is logging into /var/log/pki/pki-tomcat/ca/debug:

[...][localhost-startStop-2]: ============================================
[...][localhost-startStop-2]: ===== DEBUG SUBSYSTEM INITIALIZED =======
[...][localhost-startStop-2]: ============================================
[...][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false
[...][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[...][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: done init id=debug
[...][localhost-startStop-2]: CMSEngine: initialized debug
[...][localhost-startStop-2]: CMSEngine: initSubsystem id=log
[...][localhost-startStop-2]: CMSEngine: ready to init id=log
[...][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
[...][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
[...][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
[...][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false
[...][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[...][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: done init id=log
[...][localhost-startStop-2]: CMSEngine: initialized log
[...][localhost-startStop-2]: CMSEngine: initSubsystem id=jss
[...][localhost-startStop-2]: CMSEngine: ready to init id=jss
[...][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false
[...][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[...][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[...][localhost-startStop-2]: CMSEngine: done init id=jss
[...][localhost-startStop-2]: CMSEngine: initialized jss
[...][localhost-startStop-2]: CMSEngine: initSubsystem id=dbs
[...][localhost-startStop-2]: CMSEngine: ready to init id=dbs
[...][localhost-startStop-2]: DBSubsystem: init() mEnableSerialMgmt=true
[...][localhost-startStop-2]: Creating LdapBoundConnFactor(DBSubsystem)
[...][localhost-startStop-2]: LdapBoundConnFactory: init
[...][localhost-startStop-2]: LdapBoundConnFactory:doCloning true
[...][localhost-startStop-2]: LdapAuthInfo: init()
[...][localhost-startStop-2]: LdapAuthInfo: init begins
[...][localhost-startStop-2]: LdapAuthInfo: init ends
[...][localhost-startStop-2]: init: before makeConnection errorIfDown is true
[...][localhost-startStop-2]: makeConnection: errorIfDown true
[...][localhost-startStop-2]: TCP Keep-Alive: true
[...][localhost-startStop-2]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca
[...][localhost-startStop-2]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca
[...][localhost-startStop-2]: SSL handshake happened
Could not connect to LDAP server host port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
 at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(
 at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(
 at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(
 at com.netscape.cmscore.dbs.DBSubsystem.init(
 at com.netscape.cmscore.apps.CMSEngine.initSubsystem(
 at com.netscape.cmscore.apps.CMSEngine.initSubsystems(
 at com.netscape.cmscore.apps.CMSEngine.init(
 at com.netscape.certsrv.apps.CMS.init(
 at com.netscape.certsrv.apps.CMS.start(
 at com.netscape.cms.servlet.base.CMSStartServlet.init(
 at javax.servlet.GenericServlet.init(
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(
 at java.lang.reflect.Method.invoke(
 at Method)
 at org.apache.catalina.core.StandardWrapper.initServlet(
 at org.apache.catalina.core.StandardWrapper.loadServlet(
 at org.apache.catalina.core.StandardWrapper.load(
 at org.apache.catalina.core.StandardContext.loadOnStartup(
 at org.apache.catalina.core.StandardContext.startInternal(
 at org.apache.catalina.util.LifecycleBase.start(
 at org.apache.catalina.core.ContainerBase.addChildInternal(
 at org.apache.catalina.core.ContainerBase.access$000(
 at org.apache.catalina.core.ContainerBase$

The exception shows that LDAP authentication failed with return code 49: invalid credentials.


Communication between pki-tomcatd and the LDAP server

We can see that pki-tomcatd is trying to open a LDAP connection through SSL but fails to authenticate. Within FreeIPA, pki-tomcat is storing data in the 389-ds LDAP server and needs to communicate through LDAP with this server.

The configuration of this communication channel can be read in /etc/pki/pki-tomcat/ca/CS.cfg:

internaldb.ldapauth.bindDN=cn=Directory Manager
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca

The connection is using port 636 (SSL port) with SSL Client authentication (authtype=SslClientAuth). This means that pki-tomcatd provides a user certificate to the LDAP server, and the LDAP server maps a user to this certificate in order to authenticate the communications.

Note: Authtype can either be SslClientAuth or BasicAuth (authentication with username and password).

In this case, the SSL client authentication is done with the certificate named ‘subsystemCert cert-pki-ca‘ that is stored in /etc/pki/pki-tomcat/alias. So what could be causing the authentication to fail? We need to check that the certificate is available in /etc/pki/pki-tomcat/alias and that pki-tomcat is able to use the associated private key, and that the LDAP server is able to map this certificate to a user.


Check the subsystemCert cert-pki-ca

The first step consists in making sure that this certificate is present in /etc/pki/pki-tomcat/alias:

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
 Version: 3 (0x2)


Then make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=…)

$ sudo grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt
$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa 86a7fe00cc2a01ad085f35d4ed3e84e7b82ab4f5 subsystemCert cert-pki-ca

At this point we know that pki-tomcat is able to access the certificate and the private key. So the issue is likely to be on the LDAP server side.


LDAP server configuration

The LDAP configuration describes how a certificate can be mapped to a user in /etc/dirsrv/slapd-IPADOMAIN-COM/certmap.conf:

$ sudo cat /etc/dirsrv/slapd-IPADOMAIN-COM/certmap.conf 
certmap default default
default:FilterComps uid
certmap ipaca CN=Certificate Authority,O=IPADOMAIN.COM
ipaca:CmapLdapAttr seeAlso
ipaca:verifycert on

This means that when the LDAP server receives an authentication request with a certificate issued by the CA CN=Certificate Authority,O=IPADOMAIN.COM, it will look for users that contain a seeAlso attribute equal to the subject of the certificate, and the user entry must contain the certificate in the usercertificate attribute (verifycert: on).

With a default config, the ‘subsystemCert cert-pki-ca‘ is mapped to the user uid=pkidbuser,ou=people,o=ipaca. So let’s compare the user entry and the certificate:

$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password: 
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIID...uwab3
description: 2;4;CN=Certificate Authority,O=IPADOMAIN.COM;CN=CA Subsystem,O=IPADOMAIN.COM
seeAlso: CN=CA Subsystem,O=IPADOMAIN.COM

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a

The certificate in the userCertificate attribute is different from the one in the NSS database! This can also be seen by comparing the serial number with the value from the ldap entry (in description: 2;<serial>;<issuer>;<subject>):

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial
 Serial Number: 1341718541 (0x4ff9000d

This explains why pki-tomcat could not authenticate to the LDAP server. The fix consists in updating the LDAP entry with the right certificate (and do not forget to update the description attribute with the right serial number!)

But we still do not know the root cause for the inconsistency between the NSS database /etc/pki/pki-tomcat/alias and the user entry for uid=pkidbuser. This could be described in another blog post (for the impatients, the automatic renewal of the certificate failed to update the LDAP server entry…)


Troubleshooting: mapping between a SmartCard certificate and an IdM user

Authentication with a SmartCard may fail when the SmartCard certificate is not linked to any IdM user, or to a user different from the one specified on the console.

In order to find which user is associated to a given SmartCard certificate, you can run the following command:

ipaclient$ ipa certmap-match cert.pem
1 user matched
 User logins: demosc1
Number of entries returned 1

If the result is not what you were expecting, you need first to check how certificates are mapped to users.

By default, a certificate is associated to a user when the user entry contains the full certificate in its usercertificate attribute. But this behavior can be modified by defining certificate mapping rules:

ipaclient$ ipa certmaprule-find
1 Certificate Identity Mapping Rule matched
 Rule name: rulesmartcard
 Mapping rule: (ipacertmapdata=X509:<I>{issuer_dn}<S>{subject_dn})
 Matching rule: <ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG
 Enabled: FALSE
Number of entries returned 1


Mapping with full certificate content

When the mapping is based on the full certificate content, you can check if the user entry contains the certificate:

root@ipaclient$ ipa user-show demosc1
 User login: demosc1
Certificate: MIIC...

If it is not the case, associate the certificate with the user entry using:

ipaclient$ CERT=`cat cert.pem | tail -n +2 | head -n -1 | tr -d '\r\n'`
ipaclient$ ipa user-add-cert demosc1 --certificate $CERT

Once this is done, you may need to clear sssd cache to force SSSD to reload the entries before retrying ipa certmap-match:

ipaclient$ sudo sss_cache -E


Flexible mapping with certificate identity mapping rule

When the mapping is based on certificate mapping rules, the same tool ipa certmap-match can be used to check which user entry is associated to a certificate. When the result is not what you expect, you can enable sssd domain logs by adding the following in /etc/sssd/sssd.conf on IdM master:

debug_level = 9

then restart sssd with

root@ipaserver$ systemctl restart sssd

The logs will be located in /var/log/sssd/sssd_ipadomain.log.


Check that the certificate identity mapping rules are properly loaded

When sssd is restarted, it reads the mapping rules and should print the following in /var/log/sssd/sssd_ipadomain.log:

[sssd[be[]]] [sss_certmap_init] (0x0040): sss_certmap initialized.
[sssd[be[]]] [ipa_certmap_parse_results] (0x4000): Trying to add rule [rulesmartcard][-1][<ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG][(ipacertmapdata=X509:<I>{issuer_dn!nss_x500}<S>{subject_dn!nss_x500})].

If the rule has an invalid syntax, you will see instead:

[sssd[be[]]] [sss_certmap_init] (0x0040): sss_certmap initialized.
[sssd[be[]]] [ipa_certmap_parse_results] (0x4000): Trying to add rule [rulesmartcard][-1][<ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG][(ipacertmapdata=X509:<I>{issuer_dn!x509}<S>{subject_dn})].
[sssd[be[]]] [parse_template] (0x0040): Parse template invalid.
[sssd[be[]]] [parse_ldap_mapping_rule] (0x0040): Failed to add template.
[sssd[be[]]] [parse_mapping_rule] (0x0040): Failed to parse LDAP mapping rule.
[sssd[be[]]] [ipa_certmap_parse_results] (0x0020): sss_certmap_add_rule failed for rule [rulesmartcard], skipping. Please check for typos and if rule syntax is supported.
[sssd[be[]]] [ipa_subdomains_certmap_done] (0x0040): Unable to parse certmap results [22]: Invalid argument
[sssd[be[]]] [ipa_subdomains_refresh_certmap_done] (0x0020): Failed to read certificate mapping rules [22]: Invalid argument

The log shows that the rule named rulesmartcard is invalid. Check the rule (see man page for sss-certmap for the supported syntax) and fix if needed:

ipaclient$ ipa certmaprule-show rulesmartcard
 Rule name: rulesmartcard
 Mapping rule: (ipacertmapdata=X509:<I>{issuer_dn!x509}<S>{subject_dn})
 Matching rule: <ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG
 Enabled: TRUE
ipaclient$ ipa certmaprule-mod rulesmartcard --maprule '(ipacertmapdata=X509:<I>{issuer_dn}<S>{subject_dn})'


Check that the matching rule corresponds to the certificate

When SSSD tries to associate the certificate to a user, it starts by finding which rule can be applied based on the matching rule (for instance rulesmartcard applies to all certificates issued by CN=Smart Card CA,O=EXAMPLE.ORG because its matching rule is <ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG).

If no matching rule applies to the certificate, SSSD will not be able to associate the certificate with a user, and will display the following in /var/log/sssd/sssd_ipadomain.log:

[sssd[be[]]] [sss_cert_derb64_to_ldap_filter] (0x0040): Certificate does not match matching-rules.

In this case, you need to create or modify an identity mapping rule, so that the match rule applies to your certificate. See sss-certmap man page for the supported syntax of the –matchrule option of ipa certmaprule-add command.

Check that the expected certificate identity mapping rule is used

When SSSD tries to find the user associated to the certificate, you will see the following logs in /var/log/sssd/sssd_ipadomain.log:

[sssd[be[]]] [dp_get_account_info_handler] (0x0200): Got request for [0x14][BE_REQ_BY_CERT][cert=MII..]
[sssd[be[]]] [sdap_search_user_next_base] (0x0400): Searching for users with base [cn=accounts,dc=ipadomain,dc=com]
[sssd[be[]]] [sdap_print_server] (0x2000): Searching
[sssd[be[]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(ipacertmapdata=X509:<I>O=EXAMPLE.ORG,CN=Smart Card CA<S>CN=test,O=EXAMPLE.ORG)(objectclass=posixAccount)(uid=*)(&(uidNumber=*)(!(uidNumber=0))))][cn=accounts,dc=ipadomain,dc=com].
[sssd[be[]]] [sdap_search_user_process] (0x0400): Search for users, returned 0 results.

The logs show the LDAP search performed by SSSD: IP address of the LDAP server, base and search filter. Carefully review this information and compare with what you would expect.

Check that the mapping rule defines a valid search filter

If the rule cannot be transformed to a valid search filter, you will see in /var/log/sssd/sssd_ipadomain.log:

[sssd[be[]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(ipacertmapdata=X509:<I>O=EXAMPLE.ORG,CN=Smart Card CA<S>CN=test,O=EXAMPLE.ORG(objectclass=posixAccount)(uid=*)(&(uidNumber=*)(!(uidNumber=0))))][cn=accounts,dc=ipadomain,dc=com].
[sssd[be[]]] [sdap_get_generic_ext_step] (0x0080): ldap_search_ext failed: Bad search filter

If it is the case, you need to fix the certmap rule using

ipaclient$ ipa certmaprule-mod rulesmartcard –maprule …



Troubleshooting: authentication to the system console or Gnome Desktop Manager of an IdM host with a SmartCard

IdM allows to authenticate to an IdM enrolled-host by providing a SmartCard certificate instead of a username/password. The below steps are based on system console authentication but the process is similar when using Gnome desktop  login authentication.

When the authentication fails, the issue usually comes from a wrong configuration of the IdM system for SmartCard, or of PKINIT.


Configuration of the IdM host for SmartCard authentication

If the console does not even prompt for the SmartCard PIN, chances are high that the system was not properly configured for SmartCard authentication.

SSSD configuration for smart card

Check that /etc/sssd.conf contains

pam_cert_auth = True

If you need to update the file, do not forget to restart sssd with

root@ipaclient$ systemctl restart sssd


SmartCard CA must be trusted

Check that the SmartCard CA is trusted in the /etc/pki/nssdb database:

root@ipaclient$ certutil -L -d /etc/pki/nssdb/

Certificate Nickname Trust Attributes     SSL,S/MIME,JAR/XPI

SmartCardCA                               CT,C,C

If the CA is not present, add it using:

root@ipaclient$ certutil -A -d /etc/pki/nssdb -n SmartCardCA -t CT,C,C -i ca.pem


IdM host PKCS#11 module

Check that the IdM host is properly configured for Smart Cards. The opensc package must be installed, the the SmartCard daemon must be running, and the PKCS#11 module must be loaded

root@ipaclient$ dnf install opensc
root@ipaclient$ systemctl start pcscd.service pcscd.socket
root@ipaclient$ modutil -dbdir /etc/pki/nssdb -add "OpenSC" -libfile /usr/lib64/


Configuration for PKINIT

If the console prompts for the SmartCard PIN but displays

ipaclient login: demosc1
Pin for PIV Card:
Login incorrect

it is possible that the authentication fails trying to acquire a Kerberos ticket with PKINIT. In this case, login with username/password to the IdM host and try to manually perform kinit in order to get more information:

root@ipaclient$ kinit -X X509_user_identity='' demosc1


If  the command outputs the following:

kinit: Pre-authentication failed: Failed to verify own certificate (depth 1): self signed certificate in certificate chain while getting initial credentials

then check the content of /etc/krb5.conf on the IdM host. The realms section must contain a configuration for ipadomain with pkinit_anchors:

 pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
 pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem


The file defined in pkinit_anchors must exist, be readable and contain the certificate of the CA which signed the SmartCard certificate. If it is not the case, run the following commands on any IPA server:

root@ipaserver$ ipa-cacert-manage install -n SmartCardCA -t CT,C,C -p $DM_PWD ca.pem
root@ipaserver$ ipa-certupdate

And run the ipa-certupdate command on all IdM hosts in order to download the certificate.

If the kinit command output the following:

kinit: Certificate mismatch while getting initial credentials

check that the SmartCard certificate is associated to the username provided in the console (see mapping between a SmartCard certificate and an IdM user).

Troubleshooting: ssh to an IdM host with a SmartCard

IdM allows to perform ssh from a non-enrolled host into an IdM enrolled host, using Smart Card authentication instead of ssh authorized keys. The ssh command would be the following to log as demosc1 into the host

localuser@localhost$ ssh -I /usr/lib64/ -l demosc1
Enter PIN for 'PIV_II (PIV Card Holder pin)':

The -I option specifies a PKCS#11 shared library, and -l the username on the remote host.


Configuration of the local host

First check that the local host is properly configured for Smart Cards. The opensc package must be installed, and the the SmartCard daemon must be running.

localuser@localhost$ sudo dnf install opensc
localuser@localhost$ sudo systemctl start pcscd.service pcscd.socket


Configuration of the remote (IdM) host

When IdM is properly configured, ssh will prompt for the SmartCard PIN and authenticate the user. If there is an issue with the certificate, ssh will revert to another authentication type (private keys or username/password).

In this case, enable debug logs for ssh authentication on IdM host. Edit /etc/sssd/sssd.conf and add the following line in [ssh] section:

debug_level = 9

then restart sssd using

root@ipaclient$ systemctl restart sssd

The logs will be located on the IdM host in /var/log/sssd/sssd_ssh.log.


The Smart Card CA is not trusted by SSSD

If you see the following in /var/log/sssd/sssd_ssh.log:

[sssd[ssh]] [cert_to_ssh_key] (0x0020): CERT_VerifyCertificateNow failed [-8179].
[sssd[ssh]] [get_valid_certs_keys] (0x0040): cert_to_ssh_key failed, ignoring.

then it means that the CA that signed the Smart Card certificate is not trusted. The trusted certs are stored in /etc/pki/nssdb and can be found using:

root@ipaclient$ certutil -L -d /etc/pki/nssdb

Certificate Nickname Trust Attributes     SSL,S/MIME,JAR/XPI

SmartCardCA                               CT,C,C


If the CA cert is missing, add it using

root@ipaclient$ certutil -A -d /etc/pki/nssdb -n SmartCardCA -t CT,C,C -i ca.pem


The user is not an IdM user

If the ssh operation does not log any line in /var/log/sssd/sssd_ssh.log, it probably means that the supplied user name is not a user defined in IdM. You can check with:

root@ipaclient$ ipa user-find demosc1
0 users matched
Number of entries returned 0

Check that you provided the right user name, or define an IdM user and associate the SmartCard certificate with this user.


The certificate is not mapped to the IdM user

If you see the following in /var/log/sssd/sssd_ssh.log:

Found 1 entries in domain

but the authentication fails, check that the SmartCard certificate is associated to the provided username (refer to mapping between a SmartCard certificate and an IdM user)

FreeIPA: troubleshooting SmartCard authentication

RHEL 7.4 beta is now available, delivering a new version of IPA which contains the support for Smart Card authentication. This feature allows to use a certificate contained in a SmartCard in order to login to IdM WebUI, to ssh to an IdM-enrolled host, or to login to the console or Gnome Desktop Manager of an IdM-enrolled host.

This feature is really powerful but may also seem difficult to troubleshoot. I will explain where to look for additional information when authentication fails, and how to fix the most common issues.

The information is split into posts specific to each authentication method:



Troubleshooting Certmonger issues with FreeIPA

In a previous post I explained the basics for certmonger. This post will focus on troubleshooting the issues that certmonger can have with FreeIPA deployments.

Certificate Authorities used by FreeIPA

When certmonger is installed on a machine, it comes with pre-defined Certificate Authorities (that can be listed using certmonger list-cas): SelfSign, IPA, certmaster, and local.

The installation of FreeIPA configures additional Certificate Authorities: dogtag-ipa-renew-agent and dogtag-ipa-ca-renew-agent. FreeIPA is using 3 of those CAs for its certificates:

  • IPA
  • dogtag-ipa-renew-agent
  • dogtag-ipa-ca-renew-agent

The table below summarizes the tracked certificates and their associated CA:

Certificate nickname Location CA Purpose
auditSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
ocspSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
subsystemCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
caSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI Certificate Authority
Server-Cert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-renew-agent PKI Server
ipaCert /etc/httpd/alias dogtag-ipa-ca-renew-agent Renew Agent – authenticates renewal requests
Server-Cert /etc/httpd/alias IPA HTTPd server
Server-Cert /etc/dirsrv/slapd-DOMAIN-COM IPA LDAP server

Note: all the certificates tracked by dogtag-ipa-ca-renew-agent are identical on all the replicas. This means for instance that when ‘ocspSigningCert cert-pki-ca’ is renewed on the master, the new certificate must also be installed on the replicas.

The behavior is different for the Server-Cert certificates (for PKI, HTTPd and LDAP servers): each replica has its own certificate, which contains the hostname in its subject.

Potential issues with dogtag-ipa-ca-renew-agent helper

As stated in the previous post, it is possible to check the status of a certificate using getcert list. When each certificate is valid and properly tracked, the output should display status: MONITORING. If it is not the case, you may see an additional error message:

$ getcert list -n ipaCert
Number of certificates and requests being tracked: 10.
Request ID '20161214165502':
 ca-error: Error 7 connecting to Couldn't connect to server.

So let’s try to understand what is happening. The helper dogtag-ipa-ca-renew-agent has a different behavior, whether it is running on the renewal master or not.

On the renewal master

Under normal circumstances, the helper dogtag-ipa-ca-renew-agent is contacting Dogtag server through http or https, issues the renewal command and then puts the new certificate in LDAP so that they can be retrieved by other replicas . Several issues can happen:

  • Dogtag server may be down. Easy to check with
    $ systemctl status pki-tomcatd@pki-tomcat.service
  • When renewing a certificate, the helper needs to authenticate through SSL. In order to do that, it uses the ipaCert certificate stored in /etc/httpd/alias and needs to read the PIN in /etc/httpd/alias/pwdfile.txt. You can check that the certificate is present and the PIN is up-to-date with
    $ certutil -L -d /etc/httpd/alias/ -n ipaCert
    $ certutil -K -d /etc/httpd/alias/ -f /etc/httpd/alias/pwdfile.txt
  • The helper needs to trust IPA CA in order to establish a SSL connection to Dogtag server. It is using /etc/ipa/ca.crt as certificate trust anchor. Check that this file is up-to-date.
  • When the certificate is renewed, the helper puts the certificate in LDAP below cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN.

In order to find which step failed, you can first examine the journal:

$ journalctl -u certmonger --since today
[...] dogtag-ipa-ca-renew-agent-submit[14237]: Forwarding request to dogtag-ipa-renew-agent
[...] dogtag-ipa-ca-renew-agent-submit[14237]: dogtag-ipa-renew-agent returned 3
[...] certmonger[69651]: 2016-12-19 11:19:02 [69651] Error 7 connecting to Couldn't connect to server.

You can see that the request is forwarded to another helper (dogtag-ipa-renew-agent) and the logs display the result code returned by this other helper. The result codes are defined in certmonger API:

  • 0: certificate issued, success
  • 1: the client should wait. The helper outputs a cookie value that will be reused in the next call.
  • 2: request rejected
  • 3: error connecting to the CA
  • 4: the helper requires additional configuration data
  • 5: the client should wait. The helper outputs a delay and a cookie.
  • 6: the CA does not understand the request
  • 16: the helper needs SCEP data
  • 17: the client needs to try again using a different key pair

The log also displays when the pre-save and post-save commands are called, and finally the success of the operation.

When the operation fails, you can increase the debug level to get more information: edit (or create) /etc/ipa/server.conf and add the following:

debug = True

and modify the helper to put it in debug mode with the -vv option:

$ getcert modify-ca -c dogtag-ipa-ca-renew-agent -e '/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -vv'

At this point, the helper will log information in /var/log/ipa/renew.log that may help diagnose the issue:

[...] ipa DEBUG Starting external process
[...] ipa DEBUG args=/usr/libexec/certmonger/dogtag-ipa-renew-agent-submit -vv --submit-option requestor_name=IPA
[...] ipa DEBUG Process finished, return code=3
[...] ipa DEBUG stdout=Error 7 connecting to Couldn't connect to server.

[...] ipa DEBUG stderr=* Trying 2620:52:0:224e:21a:4aff:fe23:14c7...
* connect to 2620:52:0:224e:21a:4aff:fe23:14c7 port 8080 failed: Connection refused
* Trying
* connect to port 8080 failed: Connection refused
* Failed to connect to port 8080: Connection refused
* Closing connection 0
GET ""
code = 7
code_text = "Couldn't connect to server"
results = "(null)"

In this case the issue was simply that pki server was stopped. The solution was to restart and resubmit the certificate request.


On non-renewal master

The helper has a different behavior when it does not run on the renewal master. It connects to the local LDAP server and reads the renewed certificate below cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN, then copies the certificate into the right location. It is easy to check the access log of the directory server and make sure that the LDAP search succeeded (/var/log/dirsrv/slapd-DOMAIN-COM/access):

[...] conn=1274 op=1 SRCH base="cn=subsystemCert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=domain,dc=com" scope=0 filter="(objectClass=*)" attrs="userCertificate"
[...] conn=1274 op=1 RESULT err=0 tag=101 nentries=1 etime=0

In this log we can see that the renewed certificate ‘subsystemCert cert-pki-ca’ was found (nentries=1).

The certificate is not found if:

  • there are replication issues and the replica did not get cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN updated
  • the renewal on the master did not succeed to upload the new certificate in LDAP


Potential issues with IPA helper

This helper is used to renew the server certificate for HTTPd, LDAP and PKI. The helper is communicating with FreeIPA server using XML-RPC (with the URI defined in /etc/ipa/default.conf as xmlrpc_uri), trusting the CA certificate in /etc/ipa/ca.crt. Note that the helper may connect to FreeIPA server running on a different host.

When FreeIPA server receives the request, it handles the operation by submitting the renewal to the PKI server through CA REST API, using the certificate ipaCert located in /etc/httpd/alias for authentication.

If the certificates are not up-to-date, the communication will fail with the following error:

Request ID '20161214165647':
 ca-error: Server at failed request, will retry: 4016 (RPC failed at server. Failed to authenticate to CA REST API)

Sometimes just running $ ipa-certupdate and resubmitting the request will fix the issue. On a non-renewal master you may also need to update ipaCert using $ getcert resubmit -i requestID_for_ipaCert.

The log files that may help troubleshoot are the following:

  • /var/log/httpd/access_log and error_log: they should display the communication between the CA helper and FreeIPA server
  • /var/log/pki/pki-tomcat/ca/debug: this log will show PKI server handling the renewal requests, in multiple steps (profileSubmit, profileReview, profileProcess).
  • the journal


Potential issues with dogtag-ipa-renew-agent

This helper is used to renew PKI Server certificate. It is also using ipaCert to communicate with the PKI server and may fail on a replica when ipaCert is not up-to-date:

Request ID '20161219133932':
 ca-error: Server at "" replied: 1: Invalid Credential.

In this case, start by renewing ipaCert. This command will download the new ipaCert from LDAP:

$ getcert list -n ipaCert | grep Request
Request ID '20161214210956':
$ getcert resubmit -i 20161214210956

When ipaCert has been updated, you can try to re-submit the request for PKI Server-Cert.


Using Certmonger to track certificates

When FreeIPA is installed with an integrated IdM CA, it is using certmonger to track and renew its certificates. But what does this exactly mean?

When the certificates are reaching their expiration date, certmonger detects that it needs to renew them and takes care of the renewal (request a renewed certificate, then install the new certificate at the right location and finally restart the service so that it picks up the new certificate). It means that the system administrator does not need to bother anymore with renewals!

Well… When everything works well it is really a great functionality. But sometimes a small problem can prevent the renewal and FreeIPA ends up with expired certificates and HTTP or LDAP services refusing to start. In this case, it is really difficult to understand what has gone wrong, and how to fix the issue.

In this post, I will explain what is happening behind the scene with certmonger, so that you understand where to look for if you need to troubleshoot.

Certmonger concepts

Certmonger daemon and CLI

Certmonger provides 2 main components:

  • the certmonger daemon that is the “engine” tracking the list of certificates and launching renewal commands
  • the command-line interface: getcert, that allows to send commands to the certmonger daemon (for instance request a new certificate, list the tracked certificates, start or stop tracking a certificate, renew a certificate…)

Certificate Authority

Certmonger provides a generic interface allowing to communicate with various certificate systems, such as Dogtag, FreeIPA… A simple definition for Certificate System would be a software solution able to deliver certificates. This allows to use the same certmonger command independently of the Certificate System that will actually handle the request. The getcert command just reads the additional argument -c to know with which Certificate authority to interface.

Then certmonger needs to know how to interface with each type of Certificate System. This is done by defining Certificate Authorities that can be listed with:

$ getcert list-cas
CA 'SelfSign':
 is-default: no
 next-serial-number: 01
 is-default: no
 ca-type: EXTERNAL
 helper-location: /usr/libexec/certmonger/ipa-submit

Each section starting with ‘CA’ defines a type of Certificate Authority that certmonger knows to handle. The output of the command also shows a helper-location, which is the command that certmonger will call to discuss with the Certificate Authority. For instance:

$ getcert list-cas -c IPA
 is-default: no
 ca-type: EXTERNAL
 helper-location: /usr/libexec/certmonger/ipa-submit

shows that certmonger will run the command “/usr/libexec/certmonger/ipa-submit” when interfacing with IPA certificate authority.

Each helper command is following an interface imposed by certmonger. For instance, environment variables are set by certmonger to provide the operation to execute, the CSR etc…

Certificate tracking

List of tracked certificates

In order to know the list of certificates currently tracked by certmonger, the command getcert list can be used. It shows a lot of information:

  • the certificate location (for instance HTTP server cert is stored in the NSS database /etc/httpd/alias)
  • the certificate nickname
  • the file storing the pin
  • the Certificate Authority that will be used to renew the certificate
  • the expiration date
  • the status of the certificate (MONITORING when it is tracked and not expired)

For instance, to list all the tracking requests for certificates with a nickname “Server-Cert” stored in the NSS db /etc/httpd/alias:

$ getcert list -n Server-Cert -d /etc/httpd/alias/
Number of certificates and requests being tracked: 8.
Request ID '20161122101308':
 stuck: no
 key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
 certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
 issuer: CN=Certificate Authority,O=DOMAIN.COM
 expires: 2018-11-23 10:09:34 UTC
 key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
 eku: id-kp-serverAuth,id-kp-clientAuth
 pre-save command: 
 post-save command: /usr/lib64/ipa/certmonger/restart_httpd
 track: yes
 auto-renew: yes

Certificate renewal

When a certification is near its expiration date, certmonger daemon will automatically issue a renewal command using the CA helper, obtain a renewed certificate and replace the previous cert with the new one.

It is also possible to manually renew a certificate in advance by using the command getcert resubmit -i <id>, where <id> is the Request ID displayed by getcert list for the targetted certificate. This command will renew the certificate using the right helper command.

Start/Stop tracking a certificate

The commands getcert start-tracking and getcert stop-tracking enable or disable the monitoring of a certificate. It is important to understand that they do not manipulate the certificate (stop-tracking does not delete it or remove it from the NSS database) but simply add/remove the certificate to/from the list of monitored certificates.

Pre and post-save commands

When a certificate is tracked by certmonger, it can be useful to define pre-save and post-save commands that certmonger will call during the renewal process. For instance:

$ getcert list -n Server-Cert -d /etc/httpd/alias/
Number of certificates and requests being tracked: 8.
Request ID '20161122101308':
 stuck: no
 key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
 certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
 pre-save command: 
 post-save command: /usr/lib64/ipa/certmonger/restart_httpd
 track: yes
 auto-renew: yes

shows that the renewal of HTTPd Server Cert:

  • will be handled by IPA Certificate Authority. Remember, we can find the associated helper using getcert list-cas -c IPA
  • will also launch the command restart_httpd

This is useful when a service needs to be restarted in order to pick up the new certificate.


 Certmonger logs

Certmonger uses the journal log. For instance, when a certificate is near its expiration date, the journal will show:

$ sudo journalctl -xe -t certmonger | more
Nov 05 11:35:47 certmonger[59223]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" will not be valid after 20161115150822.

And when the certificate has been automatically renewed, the journal will show:

$ journalctl -t certmonger | more
Nov 24 12:23:15 certmonger[36674]: Certificate named "ipaCert" in token "NSS Certificate DB" in database "/etc/httpd/alias" issued by CA and saved.

Output of getcert list

It is possible to check the status for each certificate using getcert list:

  • when the certificate is still valid, the status should be MONITORING.
  • when the certificate is near its expiration date, certmonger will request its renewal and the status will change from MONITORING to SUBMITTING and finally back to MONITORING (you may also see intermediate status PRE_SAVE_CERT and POST_SAVE_CERT).

When the renewal fails, getcert list will also show an error message. It will help determine which phase failed, and from there you will need to check the logs specific to the CA helper or to the pre-save or post-save commands.

In the next post, I will detail the errors that can arise with the helpers used with FreeIPA.