Troubleshooting Certmonger issues with FreeIPA

In a previous post I explained the basics for certmonger. This post will focus on troubleshooting the issues that certmonger can have with FreeIPA deployments.

Certificate Authorities used by FreeIPA

When certmonger is installed on a machine, it comes with pre-defined Certificate Authorities (that can be listed using certmonger list-cas): SelfSign, IPA, certmaster, and local.

The installation of FreeIPA configures additional Certificate Authorities: dogtag-ipa-renew-agent and dogtag-ipa-ca-renew-agent. FreeIPA is using 3 of those CAs for its certificates:

  • IPA
  • dogtag-ipa-renew-agent
  • dogtag-ipa-ca-renew-agent

The table below summarizes the tracked certificates and their associated CA:

Certificate nickname Location CA Purpose
auditSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
ocspSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
subsystemCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
caSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI Certificate Authority
Server-Cert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-renew-agent (IPA < 4.5)
dogtag-ipa-ca-renew-agent (IPA 4.5+)
PKI Server
ipaCert /etc/httpd/alias (IPA < 4.5)
dogtag-ipa-ca-renew-agent Renew Agent – authenticates renewal requests
Server-Cert /etc/httpd/alias IPA HTTPd server
Server-Cert /etc/dirsrv/slapd-DOMAIN-COM IPA LDAP server

Note: all the certificates tracked by dogtag-ipa-ca-renew-agent are identical on all the replicas. This means for instance that when ‘ocspSigningCert cert-pki-ca’ is renewed on the master, the new certificate must also be installed on the replicas.

The behavior is different for the Server-Cert certificates (for PKI, HTTPd and LDAP servers): each replica has its own certificate, which contains the hostname in its subject.

Potential issues with dogtag-ipa-ca-renew-agent helper

As stated in the previous post, it is possible to check the status of a certificate using getcert list. When each certificate is valid and properly tracked, the output should display status: MONITORING. If it is not the case, you may see an additional error message:

$ getcert list -n ipaCert
Number of certificates and requests being tracked: 10.
Request ID '20161214165502':
 ca-error: Error 7 connecting to Couldn't connect to server.

So let’s try to understand what is happening. The helper dogtag-ipa-ca-renew-agent has a different behavior, whether it is running on the renewal master or not.

On the renewal master

Under normal circumstances, the helper dogtag-ipa-ca-renew-agent is contacting Dogtag server through http or https, issues the renewal command and then puts the new certificate in LDAP so that they can be retrieved by other replicas . Several issues can happen:

  • Dogtag server may be down. Easy to check with
    $ systemctl status pki-tomcatd@pki-tomcat.service
  • When renewing a certificate, the helper needs to authenticate through SSL. In order to do that, it uses the ipaCert certificate stored in /etc/httpd/alias and needs to read the PIN in /etc/httpd/alias/pwdfile.txt. You can check that the certificate is present and the PIN is up-to-date with
    $ certutil -L -d /etc/httpd/alias/ -n ipaCert
    $ certutil -K -d /etc/httpd/alias/ -f /etc/httpd/alias/pwdfile.txt
  • The helper needs to trust IPA CA in order to establish a SSL connection to Dogtag server. It is using /etc/ipa/ca.crt as certificate trust anchor. Check that this file is up-to-date.
  • When the certificate is renewed, the helper puts the certificate in LDAP below cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN.

In order to find which step failed, you can first examine the journal:

$ journalctl -u certmonger --since today
[...] dogtag-ipa-ca-renew-agent-submit[14237]: Forwarding request to dogtag-ipa-renew-agent
[...] dogtag-ipa-ca-renew-agent-submit[14237]: dogtag-ipa-renew-agent returned 3
[...] certmonger[69651]: 2016-12-19 11:19:02 [69651] Error 7 connecting to Couldn't connect to server.

You can see that the request is forwarded to another helper (dogtag-ipa-renew-agent) and the logs display the result code returned by this other helper. The result codes are defined in certmonger API:

  • 0: certificate issued, success
  • 1: the client should wait. The helper outputs a cookie value that will be reused in the next call.
  • 2: request rejected
  • 3: error connecting to the CA
  • 4: the helper requires additional configuration data
  • 5: the client should wait. The helper outputs a delay and a cookie.
  • 6: the CA does not understand the request
  • 16: the helper needs SCEP data
  • 17: the client needs to try again using a different key pair

The log also displays when the pre-save and post-save commands are called, and finally the success of the operation.

When the operation fails, you can increase the debug level to get more information: edit (or create) /etc/ipa/server.conf and add the following:

debug = True

and modify the helper to put it in debug mode with the -vv option:

$ getcert modify-ca -c dogtag-ipa-ca-renew-agent -e '/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -vv'

At this point, the helper will log information in /var/log/ipa/renew.log that may help diagnose the issue:

[...] ipa DEBUG Starting external process
[...] ipa DEBUG args=/usr/libexec/certmonger/dogtag-ipa-renew-agent-submit -vv --submit-option requestor_name=IPA
[...] ipa DEBUG Process finished, return code=3
[...] ipa DEBUG stdout=Error 7 connecting to Couldn't connect to server.

[...] ipa DEBUG stderr=* Trying 2620:52:0:224e:21a:4aff:fe23:14c7...
* connect to 2620:52:0:224e:21a:4aff:fe23:14c7 port 8080 failed: Connection refused
* Trying
* connect to port 8080 failed: Connection refused
* Failed to connect to port 8080: Connection refused
* Closing connection 0
GET ""
code = 7
code_text = "Couldn't connect to server"
results = "(null)"

In this case the issue was simply that pki server was stopped. The solution was to restart and resubmit the certificate request.


On non-renewal master

The helper has a different behavior when it does not run on the renewal master. It connects to the local LDAP server and reads the renewed certificate below cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN, then copies the certificate into the right location. It is easy to check the access log of the directory server and make sure that the LDAP search succeeded (/var/log/dirsrv/slapd-DOMAIN-COM/access):

[...] conn=1274 op=1 SRCH base="cn=subsystemCert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=domain,dc=com" scope=0 filter="(objectClass=*)" attrs="userCertificate"
[...] conn=1274 op=1 RESULT err=0 tag=101 nentries=1 etime=0

In this log we can see that the renewed certificate ‘subsystemCert cert-pki-ca’ was found (nentries=1).

The certificate is not found if:

  • there are replication issues and the replica did not get cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN updated
  • the renewal on the master did not succeed to upload the new certificate in LDAP


Potential issues with IPA helper

This helper is used to renew the server certificate for HTTPd, LDAP and PKI. The helper is communicating with FreeIPA server using XML-RPC (with the URI defined in /etc/ipa/default.conf as xmlrpc_uri), trusting the CA certificate in /etc/ipa/ca.crt. Note that the helper may connect to FreeIPA server running on a different host.

When FreeIPA server receives the request, it handles the operation by submitting the renewal to the PKI server through CA REST API, using the certificate ipaCert located in /etc/httpd/alias for authentication.

If the certificates are not up-to-date, the communication will fail with the following error:

Request ID '20161214165647':
 ca-error: Server at failed request, will retry: 4016 (RPC failed at server. Failed to authenticate to CA REST API)

Sometimes just running $ ipa-certupdate and resubmitting the request will fix the issue. On a non-renewal master you may also need to update ipaCert using $ getcert resubmit -i requestID_for_ipaCert.

The log files that may help troubleshoot are the following:

  • /var/log/httpd/access_log and error_log: they should display the communication between the CA helper and FreeIPA server
  • /var/log/pki/pki-tomcat/ca/debug: this log will show PKI server handling the renewal requests, in multiple steps (profileSubmit, profileReview, profileProcess).
  • the journal


Potential issues with dogtag-ipa-renew-agent

This helper is used to renew PKI Server certificate. It is also using ipaCert to communicate with the PKI server and may fail on a replica when ipaCert is not up-to-date:

Request ID '20161219133932':
 ca-error: Server at "" replied: 1: Invalid Credential.

In this case, start by renewing ipaCert. This command will download the new ipaCert from LDAP:

$ getcert list -n ipaCert | grep Request
Request ID '20161214210956':
$ getcert resubmit -i 20161214210956

When ipaCert has been updated, you can try to re-submit the request for PKI Server-Cert.



11 thoughts on “Troubleshooting Certmonger issues with FreeIPA”

  1. Hello,

    great article, the explanation is very well done.
    I have a question: some of my certificates are going to expire in few days, and certmonger did not renew it;
    if i run getcert list, both the certificates with nickname “Server-Cert” return the following error:

    Request ID ‘20170320134855’:
    status: CA_UNREACHABLE
    ca-error: Server at https://MY_SITE.COM/ipa/xml failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: FAILURE (CA not found: 516f847a-ddc0-4da6-b15d-6703742b74be)).
    stuck: no

    while all the other certificates have been correctly updated.
    Do you have any suggestion? Any feedback would be greatly welcome.
    Thank you in advance,


  2. Hi,
    this issue reminds me of a similar problem. Can you check the content of the entry cn=ipa,cn=cas,cn=ca,$BASEDN (especially the attribute ipacaid)?
    Then list the entries below ou=authorities,ou=ca,o=ipaca. You should find an entry with dn: cn=$ipacaid,ou=authorities,ou=ca,o=ipaca.

    If it is not the case, the entry cn=ipa,cn=cas,cn=ca,$BASEDN probably somehow got inconsistent with the content of ou=authorities,ou=ca,o=ipaca. Check if there are any replication conflicts, and if not you can try to edit cn=ipa,cn=cas,cn=ca,$BASEDN and replace its ipacaid with the value read from cn=$ipacaid,ou=authorities,ou=ca,o=ipaca.



    1. Hello,

      thank you very much for you quick and detailed answer!
      You found the problem… if I do

      # ldapsearch -H ldap://$HOSTNAME -D ‘cn=Directory Manager’ -W -b ‘cn=ipa,cn=cas,cn=ca,dc=my_site,dc=com’

      # extended LDIF

      objectClass: ipaca
      ipaCaSubjectDN: CN=Certificate Authority,O=MY_SITE.COM
      ipaCaId: 516f847a-ddc0-4da6-b15d-6703742b74be
      cn: ipa

      # search result
      search: 2
      result: 0 Success

      so, according to to output, i should find an entry

      but there wasn’t :

      ldapsearch -H ldap://$HOSTNAME -D ‘cn=Directory Manager’ -W -b ‘cn=516f847a-ddc0-4da6-b15d-6703742b74be,ou=authorities,ou=ca,o=ipaca’

      # base with scope subtree
      # search result
      search: 2
      result: 32 No such object
      matchedDN: ou=authorities,ou=ca,o=ipaca

      I found, instead, another entry under ou=authorities,ou=ca,o=ipaca :

      # 072b0716-4cc3-45d4-86ce-7fcd9f37f911, authorities, ca, ipaca
      dn: cn=072b0716-4cc3-45d4-86ce-7fcd9f37f911,ou=authorities,ou=ca,o=ipaca
      authoritySerial: 267649026
      objectClass: authority
      objectClass: top
      cn: 072b0716-4cc3-45d4-86ce-7fcd9f37f911
      authorityID: 072b0716-4cc3-45d4-86ce-7fcd9f37f911
      authorityKeyNickname: caSigningCert cert-pki-ca
      authorityEnabled: TRUE
      authorityDN: CN=Certificate Authority,O=MY_SITE.COM
      description: Host authority

      I changed the entry as you suggested, then did a
      and a
      ipa-getcert resubmit -i $ID

      and everything worked as a charm, the certificates now are renewed.

      You really saved my day 🙂

      Thank you very much,



  3. Hi,

    I’m having a difficult issue with renewing expired Server certs for httpd and LDAP server. I’ve dialed time back to try and renew them but when I try to renew on the CA master, I keep getting this error:

    status: CA_UNREACHABLE
    ca-error: Server at https:///ipa/xml failed request, will retry: 4002 (RPC failed at server. service with name “HTTP/@DOMAIN.COM” already exists).

    Any chance you’ve encountered something like it before, or could point me in the right direction on which “service” its referring to?



    1. Hi,
      which command are you running to renew the HTTP cert? When the date is set back in time, the procedure is the following:
      – make sure that all services (except ntp or chrony) are running
      – sudo getcert list -d /etc/httpd/alias -n ‘Server-Cert’
      => will provide you with a Request ID
      – sudo getcert resubmit -i
      – wait for a few minutes and check the status with getcert list, should become MONITORING

      The error you’re seeing seems to indicate that IPA is somehow trying to create a new service entry for the HTTP service (but the entry is already there, as expected). This is why I’m interested in your workflow.


      1. Hi,
        With regards to the workflow,

        I set the date back in time via ($ date -s)
        Attempt to start all services, ($ ipactl start –ignore-service-failure).
        Sometimes pki-tomcatd would fail (can’t remember if this was the same case if I set time back).
        stop ntpd ($ systemctl stop ntpd).
        Restart certmonger ($ systemctl restart certmonger)
        Monitor the status via ($ ipa-getcert list)
        Get failure error from above.

        I inherited the freeipa system/s and I’m not quite sure how it was configured in the first place. From what I can tell all the freeipa nodes are CA servers and the current one I’m working on is the original CA master.

        The other nodes also have expired HTTP and LDAP certs but I saw that certmonger was also trying to renew them before expiry but they were also getting the same “service already exists” error.=/

        Thank you very much.


  4. $ getcert modify-ca -c dogtag-ipa-ca-renew-agent -e ‘/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -vv’

    seems above cmd is incorrect and According to the RH solution

    you misplaced -v option that should be after the last single quote :
    # getcert modify-ca -c dogtag-ipa-ca-renew-agent -e ‘/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit ‘ -v


    1. Hi,
      the command from the blog post modifies the arguments passed to the helper in order to put the helper in verbose mode. The command you provided would set ‘modify-ca’ in verbose mode, but wouldn’t have any impact on the helper logging. Hope this clarifies.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s