When FreeIPA is installed with an integrated IdM CA, it is using certmonger to track and renew its certificates. But what does this exactly mean?
When the certificates are reaching their expiration date, certmonger detects that it needs to renew them and takes care of the renewal (request a renewed certificate, then install the new certificate at the right location and finally restart the service so that it picks up the new certificate). It means that the system administrator does not need to bother anymore with renewals!
Well… When everything works well it is really a great functionality. But sometimes a small problem can prevent the renewal and FreeIPA ends up with expired certificates and HTTP or LDAP services refusing to start. In this case, it is really difficult to understand what has gone wrong, and how to fix the issue.
In this post, I will explain what is happening behind the scene with certmonger, so that you understand where to look for if you need to troubleshoot.
Certmonger concepts
Certmonger daemon and CLI
Certmonger provides 2 main components:
- the certmonger daemon that is the “engine” tracking the list of certificates and launching renewal commands
- the command-line interface: getcert, that allows to send commands to the certmonger daemon (for instance request a new certificate, list the tracked certificates, start or stop tracking a certificate, renew a certificate…)
Certificate Authority
Certmonger provides a generic interface allowing to communicate with various certificate systems, such as Dogtag, FreeIPA… A simple definition for Certificate System would be a software solution able to deliver certificates. This allows to use the same certmonger command independently of the Certificate System that will actually handle the request. The getcert command just reads the additional argument -c to know with which Certificate authority to interface.
Then certmonger needs to know how to interface with each type of Certificate System. This is done by defining Certificate Authorities that can be listed with:
$ getcert list-cas CA 'SelfSign': is-default: no ca-type: INTERNAL:SELF next-serial-number: 01 CA 'IPA': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/ipa-submit [...]
Each section starting with ‘CA’ defines a type of Certificate Authority that certmonger knows to handle. The output of the command also shows a helper-location, which is the command that certmonger will call to discuss with the Certificate Authority. For instance:
$ getcert list-cas -c IPA CA 'IPA': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/ipa-submit
shows that certmonger will run the command “/usr/libexec/certmonger/ipa-submit” when interfacing with IPA certificate authority.
Each helper command is following an interface imposed by certmonger. For instance, environment variables are set by certmonger to provide the operation to execute, the CSR etc…
Certificate tracking
List of tracked certificates
In order to know the list of certificates currently tracked by certmonger, the command getcert list can be used. It shows a lot of information:
- the certificate location (for instance HTTP server cert is stored in the NSS database /etc/httpd/alias)
- the certificate nickname
- the file storing the pin
- the Certificate Authority that will be used to renew the certificate
- the expiration date
- the status of the certificate (MONITORING when it is tracked and not expired)
For instance, to list all the tracking requests for certificates with a nickname “Server-Cert” stored in the NSS db /etc/httpd/alias:
$ getcert list -n Server-Cert -d /etc/httpd/alias/ Number of certificates and requests being tracked: 8. Request ID '20161122101308': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=ipaserver.domain.com,O=DOMAIN.COM expires: 2018-11-23 10:09:34 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes
Certificate renewal
When a certification is near its expiration date, certmonger daemon will automatically issue a renewal command using the CA helper, obtain a renewed certificate and replace the previous cert with the new one.
It is also possible to manually renew a certificate in advance by using the command getcert resubmit -i <id>, where <id> is the Request ID displayed by getcert list for the targetted certificate. This command will renew the certificate using the right helper command.
Start/Stop tracking a certificate
The commands getcert start-tracking and getcert stop-tracking enable or disable the monitoring of a certificate. It is important to understand that they do not manipulate the certificate (stop-tracking does not delete it or remove it from the NSS database) but simply add/remove the certificate to/from the list of monitored certificates.
Pre and post-save commands
When a certificate is tracked by certmonger, it can be useful to define pre-save and post-save commands that certmonger will call during the renewal process. For instance:
$ getcert list -n Server-Cert -d /etc/httpd/alias/ Number of certificates and requests being tracked: 8. Request ID '20161122101308': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' [...] pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes
shows that the renewal of HTTPd Server Cert:
- will be handled by IPA Certificate Authority. Remember, we can find the associated helper using getcert list-cas -c IPA
- will also launch the command restart_httpd
This is useful when a service needs to be restarted in order to pick up the new certificate.
Troubleshooting
Certmonger logs
Certmonger uses the journal log. For instance, when a certificate is near its expiration date, the journal will show:
$ sudo journalctl -xe -t certmonger | more Nov 05 11:35:47 ipaserver.domain.com certmonger[59223]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" will not be valid after 20161115150822.
And when the certificate has been automatically renewed, the journal will show:
$ journalctl -t certmonger | more Nov 24 12:23:15 ipaserver.domain.com certmonger[36674]: Certificate named "ipaCert" in token "NSS Certificate DB" in database "/etc/httpd/alias" issued by CA and saved.
Output of getcert list
It is possible to check the status for each certificate using getcert list:
- when the certificate is still valid, the status should be MONITORING.
- when the certificate is near its expiration date, certmonger will request its renewal and the status will change from MONITORING to SUBMITTING and finally back to MONITORING (you may also see intermediate status PRE_SAVE_CERT and POST_SAVE_CERT).
When the renewal fails, getcert list will also show an error message. It will help determine which phase failed, and from there you will need to check the logs specific to the CA helper or to the pre-save or post-save commands.
In the next post, I will detail the errors that can arise with the helpers used with FreeIPA.
Great content Florence! Very well explained.
LikeLike
Excellent article indeed. Question though, is it possible to change the post-save command once certmonger has started tracking?
LikeLike
Hi! if you want to modify an existing request (for instance to update the post-save command), you can use:
# getcert start-tracking -i -C
For more information, please see man getcert-start-tracking:
-i NAME
Modify the request which has this nickname. If this option is
not specified, and a tracking entry which matches the key and
certificate storage options which are specified already exists,
that entry will be modified. Otherwise, a new tracking entry
will be added.
LikeLike
Hi!
That’s great info again.
Let’s say I wanted to edit that information directly in the database that certmonger uses. Could I do that? I realize it’s not recommended, but just out of curiosity.
It looks like getcert start-tracking can also handle specifying an alternative DNS name (the -D switch) if I forgot to specify it originally. Which is great if it’s true.
Thanks again for your contribution to the project and educating us.
LikeLike
Hi,
it is certainly not supported to directly modify the requests in /var/lib/certmonger/requests/, you should use the getcert tool instead. The tool supports the -D option, as you found out, to specify a subject alternate name.
LikeLike
Is it possible to track Let’s Encrypt certificates expiration with certmonger?
LikeLike
There are open source implementations of a certmonger helper for Let’s Encrypt but AFAIK none officially supported with IPA.
On the other hand, IPA has implemented support for the ACME protocol (please see https://pagure.io/freeipa/issue/4751).
LikeLike
Hey, unfortuanatly my Certificate expired, when i want to renew with “ipa-cacert-manage renew” i only get CA Certificate is not tracked by certmonger.
i have vers. 4.6 installed.
Maybe someone once had the same problem?
LikeLike
Which Certificate expired? If the deployment is CA-less (HTTP and LDAP certificates provided by an external CA), they need to be renewed outside of IPA and installed with ipa-server-certinstall. ipa-cacert-manage renew is able to renew IPA CA when the deployment includes an embedded CA.
LikeLike
Very well explained. Thank you.
LikeLike
Hi certmonger start-tracking commands removes the IP addresses from SAN field if it has combination of IP & DNS like SAN filed has below data
test.com,test,100.40.60.209 ,fd11:63fc:e9a0:de13:1a66:daff:feb0:c30a
LikeLike
Hi,
start-tracking has 2 different options for SAN fields, one to provide DNS names and the other to provide IP addresses:
-D DNSNAME, –dns=DNSNAME
-A ADDRESS, –ip-address=ADDRESS
Did you use the -A option to specify the IP addresses?
LikeLike