Troubleshooting Certmonger issues with FreeIPA

In a previous post I explained the basics for certmonger. This post will focus on troubleshooting the issues that certmonger can have with FreeIPA deployments.

Certificate Authorities used by FreeIPA

When certmonger is installed on a machine, it comes with pre-defined Certificate Authorities (that can be listed using certmonger list-cas): SelfSign, IPA, certmaster, and local.

The installation of FreeIPA configures additional Certificate Authorities: dogtag-ipa-renew-agent and dogtag-ipa-ca-renew-agent. FreeIPA is using 3 of those CAs for its certificates:

  • IPA
  • dogtag-ipa-renew-agent
  • dogtag-ipa-ca-renew-agent

The table below summarizes the tracked certificates and their associated CA:

Certificate nickname Location CA Purpose
auditSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
ocspSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
subsystemCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI
caSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-ca-renew-agent PKI Certificate Authority
Server-Cert cert-pki-ca /etc/pki/pki-tomcat/alias dogtag-ipa-renew-agent (IPA < 4.5)
dogtag-ipa-ca-renew-agent (IPA 4.5+)
PKI Server
ipaCert /etc/httpd/alias (IPA < 4.5)
/var/lib.ipa/ra-agent.{key|pem}
dogtag-ipa-ca-renew-agent Renew Agent – authenticates renewal requests
Server-Cert /etc/httpd/alias IPA HTTPd server
Server-Cert /etc/dirsrv/slapd-DOMAIN-COM IPA LDAP server

Note: all the certificates tracked by dogtag-ipa-ca-renew-agent are identical on all the replicas. This means for instance that when ‘ocspSigningCert cert-pki-ca’ is renewed on the master, the new certificate must also be installed on the replicas.

The behavior is different for the Server-Cert certificates (for PKI, HTTPd and LDAP servers): each replica has its own certificate, which contains the hostname in its subject.

Potential issues with dogtag-ipa-ca-renew-agent helper

As stated in the previous post, it is possible to check the status of a certificate using getcert list. When each certificate is valid and properly tracked, the output should display status: MONITORING. If it is not the case, you may see an additional error message:

$ getcert list -n ipaCert
Number of certificates and requests being tracked: 10.
Request ID '20161214165502':
 status: CA_UNREACHABLE
 ca-error: Error 7 connecting to http://ipaserver.domain.com:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.

So let’s try to understand what is happening. The helper dogtag-ipa-ca-renew-agent has a different behavior, whether it is running on the renewal master or not.

On the renewal master

Under normal circumstances, the helper dogtag-ipa-ca-renew-agent is contacting Dogtag server through http or https, issues the renewal command and then puts the new certificate in LDAP so that they can be retrieved by other replicas . Several issues can happen:

  • Dogtag server may be down. Easy to check with
    $ systemctl status pki-tomcatd@pki-tomcat.service
  • When renewing a certificate, the helper needs to authenticate through SSL. In order to do that, it uses the ipaCert certificate stored in /etc/httpd/alias and needs to read the PIN in /etc/httpd/alias/pwdfile.txt. You can check that the certificate is present and the PIN is up-to-date with
    $ certutil -L -d /etc/httpd/alias/ -n ipaCert
    $ certutil -K -d /etc/httpd/alias/ -f /etc/httpd/alias/pwdfile.txt
  • The helper needs to trust IPA CA in order to establish a SSL connection to Dogtag server. It is using /etc/ipa/ca.crt as certificate trust anchor. Check that this file is up-to-date.
  • When the certificate is renewed, the helper puts the certificate in LDAP below cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN.

In order to find which step failed, you can first examine the journal:

$ journalctl -u certmonger --since today
[...] dogtag-ipa-ca-renew-agent-submit[14237]: Forwarding request to dogtag-ipa-renew-agent
[...] dogtag-ipa-ca-renew-agent-submit[14237]: dogtag-ipa-renew-agent returned 3
[...] certmonger[69651]: 2016-12-19 11:19:02 [69651] Error 7 connecting to http://ipaserver.domain.com:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.

You can see that the request is forwarded to another helper (dogtag-ipa-renew-agent) and the logs display the result code returned by this other helper. The result codes are defined in certmonger API:

  • 0: certificate issued, success
  • 1: the client should wait. The helper outputs a cookie value that will be reused in the next call.
  • 2: request rejected
  • 3: error connecting to the CA
  • 4: the helper requires additional configuration data
  • 5: the client should wait. The helper outputs a delay and a cookie.
  • 6: the CA does not understand the request
  • 16: the helper needs SCEP data
  • 17: the client needs to try again using a different key pair

The log also displays when the pre-save and post-save commands are called, and finally the success of the operation.

When the operation fails, you can increase the debug level to get more information: edit (or create) /etc/ipa/server.conf and add the following:

[global]
debug = True

and modify the helper to put it in debug mode with the -vv option:

$ getcert modify-ca -c dogtag-ipa-ca-renew-agent -e '/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -vv'

At this point, the helper will log information in /var/log/ipa/renew.log that may help diagnose the issue:

[...] ipa DEBUG Starting external process
[...] ipa DEBUG args=/usr/libexec/certmonger/dogtag-ipa-renew-agent-submit -vv --submit-option requestor_name=IPA
[...] ipa DEBUG Process finished, return code=3
[...] ipa DEBUG stdout=Error 7 connecting to http://ipaserver.domain.com:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.

[...] ipa DEBUG stderr=* Trying 2620:52:0:224e:21a:4aff:fe23:14c7...
* connect to 2620:52:0:224e:21a:4aff:fe23:14c7 port 8080 failed: Connection refused
* Trying 10.34.58.51...
* connect to 10.34.58.51 port 8080 failed: Connection refused
* Failed to connect to ipaserver.domain.com port 8080: Connection refused
* Closing connection 0
GET "http://ipaserver.domain.com:8080/ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=14&renewal=true&xml=true&requestor_name=IPA"
code = 7
code_text = "Couldn't connect to server"
results = "(null)"

In this case the issue was simply that pki server was stopped. The solution was to restart and resubmit the certificate request.

 

On non-renewal master

The helper has a different behavior when it does not run on the renewal master. It connects to the local LDAP server and reads the renewed certificate below cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN, then copies the certificate into the right location. It is easy to check the access log of the directory server and make sure that the LDAP search succeeded (/var/log/dirsrv/slapd-DOMAIN-COM/access):

[...] conn=1274 op=1 SRCH base="cn=subsystemCert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=domain,dc=com" scope=0 filter="(objectClass=*)" attrs="userCertificate"
[...] conn=1274 op=1 RESULT err=0 tag=101 nentries=1 etime=0

In this log we can see that the renewed certificate ‘subsystemCert cert-pki-ca’ was found (nentries=1).

The certificate is not found if:

  • there are replication issues and the replica did not get cn=nickname,cn=ca_renewal,cn=ipa,cn=etc,$BASEDN updated
  • the renewal on the master did not succeed to upload the new certificate in LDAP

 

Potential issues with IPA helper

This helper is used to renew the server certificate for HTTPd, LDAP and PKI. The helper is communicating with FreeIPA server using XML-RPC (with the URI defined in /etc/ipa/default.conf as xmlrpc_uri), trusting the CA certificate in /etc/ipa/ca.crt. Note that the helper may connect to FreeIPA server running on a different host.

When FreeIPA server receives the request, it handles the operation by submitting the renewal to the PKI server through CA REST API, using the certificate ipaCert located in /etc/httpd/alias for authentication.

If the certificates are not up-to-date, the communication will fail with the following error:

Request ID '20161214165647':
 status: CA_UNREACHABLE
 ca-error: Server at https://vm-058-051.abc.idm.lab.eng.brq.redhat.com/ipa/xml failed request, will retry: 4016 (RPC failed at server. Failed to authenticate to CA REST API)

Sometimes just running $ ipa-certupdate and resubmitting the request will fix the issue. On a non-renewal master you may also need to update ipaCert using $ getcert resubmit -i requestID_for_ipaCert.

The log files that may help troubleshoot are the following:

  • /var/log/httpd/access_log and error_log: they should display the communication between the CA helper and FreeIPA server
  • /var/log/pki/pki-tomcat/ca/debug: this log will show PKI server handling the renewal requests, in multiple steps (profileSubmit, profileReview, profileProcess).
  • the journal

 

Potential issues with dogtag-ipa-renew-agent

This helper is used to renew PKI Server certificate. It is also using ipaCert to communicate with the PKI server and may fail on a replica when ipaCert is not up-to-date:

Request ID '20161219133932':
 status: MONITORING
 ca-error: Server at "https://ipareplica.domain.com:8443/ca/agent/ca/profileProcess" replied: 1: Invalid Credential.

In this case, start by renewing ipaCert. This command will download the new ipaCert from LDAP:

$ getcert list -n ipaCert | grep Request
Request ID '20161214210956':
$ getcert resubmit -i 20161214210956

When ipaCert has been updated, you can try to re-submit the request for PKI Server-Cert.

 

Advertisement

Using Certmonger to track certificates

When FreeIPA is installed with an integrated IdM CA, it is using certmonger to track and renew its certificates. But what does this exactly mean?

When the certificates are reaching their expiration date, certmonger detects that it needs to renew them and takes care of the renewal (request a renewed certificate, then install the new certificate at the right location and finally restart the service so that it picks up the new certificate). It means that the system administrator does not need to bother anymore with renewals!

Well… When everything works well it is really a great functionality. But sometimes a small problem can prevent the renewal and FreeIPA ends up with expired certificates and HTTP or LDAP services refusing to start. In this case, it is really difficult to understand what has gone wrong, and how to fix the issue.

In this post, I will explain what is happening behind the scene with certmonger, so that you understand where to look for if you need to troubleshoot.

Certmonger concepts

Certmonger daemon and CLI

Certmonger provides 2 main components:

  • the certmonger daemon that is the “engine” tracking the list of certificates and launching renewal commands
  • the command-line interface: getcert, that allows to send commands to the certmonger daemon (for instance request a new certificate, list the tracked certificates, start or stop tracking a certificate, renew a certificate…)

Certificate Authority

Certmonger provides a generic interface allowing to communicate with various certificate systems, such as Dogtag, FreeIPA… A simple definition for Certificate System would be a software solution able to deliver certificates. This allows to use the same certmonger command independently of the Certificate System that will actually handle the request. The getcert command just reads the additional argument -c to know with which Certificate authority to interface.

Then certmonger needs to know how to interface with each type of Certificate System. This is done by defining Certificate Authorities that can be listed with:

$ getcert list-cas
CA 'SelfSign':
 is-default: no
 ca-type: INTERNAL:SELF
 next-serial-number: 01
CA 'IPA':
 is-default: no
 ca-type: EXTERNAL
 helper-location: /usr/libexec/certmonger/ipa-submit
[...]

Each section starting with ‘CA’ defines a type of Certificate Authority that certmonger knows to handle. The output of the command also shows a helper-location, which is the command that certmonger will call to discuss with the Certificate Authority. For instance:

$ getcert list-cas -c IPA
CA 'IPA':
 is-default: no
 ca-type: EXTERNAL
 helper-location: /usr/libexec/certmonger/ipa-submit

shows that certmonger will run the command “/usr/libexec/certmonger/ipa-submit” when interfacing with IPA certificate authority.

Each helper command is following an interface imposed by certmonger. For instance, environment variables are set by certmonger to provide the operation to execute, the CSR etc…

Certificate tracking

List of tracked certificates

In order to know the list of certificates currently tracked by certmonger, the command getcert list can be used. It shows a lot of information:

  • the certificate location (for instance HTTP server cert is stored in the NSS database /etc/httpd/alias)
  • the certificate nickname
  • the file storing the pin
  • the Certificate Authority that will be used to renew the certificate
  • the expiration date
  • the status of the certificate (MONITORING when it is tracked and not expired)

For instance, to list all the tracking requests for certificates with a nickname “Server-Cert” stored in the NSS db /etc/httpd/alias:

$ getcert list -n Server-Cert -d /etc/httpd/alias/
Number of certificates and requests being tracked: 8.
Request ID '20161122101308':
 status: MONITORING
 stuck: no
 key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
 certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
 CA: IPA
 issuer: CN=Certificate Authority,O=DOMAIN.COM
 subject: CN=ipaserver.domain.com,O=DOMAIN.COM
 expires: 2018-11-23 10:09:34 UTC
 key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
 eku: id-kp-serverAuth,id-kp-clientAuth
 pre-save command: 
 post-save command: /usr/lib64/ipa/certmonger/restart_httpd
 track: yes
 auto-renew: yes

Certificate renewal

When a certification is near its expiration date, certmonger daemon will automatically issue a renewal command using the CA helper, obtain a renewed certificate and replace the previous cert with the new one.

It is also possible to manually renew a certificate in advance by using the command getcert resubmit -i <id>, where <id> is the Request ID displayed by getcert list for the targetted certificate. This command will renew the certificate using the right helper command.

Start/Stop tracking a certificate

The commands getcert start-tracking and getcert stop-tracking enable or disable the monitoring of a certificate. It is important to understand that they do not manipulate the certificate (stop-tracking does not delete it or remove it from the NSS database) but simply add/remove the certificate to/from the list of monitored certificates.

Pre and post-save commands

When a certificate is tracked by certmonger, it can be useful to define pre-save and post-save commands that certmonger will call during the renewal process. For instance:

$ getcert list -n Server-Cert -d /etc/httpd/alias/
Number of certificates and requests being tracked: 8.
Request ID '20161122101308':
 status: MONITORING
 stuck: no
 key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
 certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
[...]
 pre-save command: 
 post-save command: /usr/lib64/ipa/certmonger/restart_httpd
 track: yes
 auto-renew: yes

shows that the renewal of HTTPd Server Cert:

  • will be handled by IPA Certificate Authority. Remember, we can find the associated helper using getcert list-cas -c IPA
  • will also launch the command restart_httpd

This is useful when a service needs to be restarted in order to pick up the new certificate.

Troubleshooting

 Certmonger logs

Certmonger uses the journal log. For instance, when a certificate is near its expiration date, the journal will show:

$ sudo journalctl -xe -t certmonger | more
Nov 05 11:35:47 ipaserver.domain.com certmonger[59223]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" will not be valid after 20161115150822.

And when the certificate has been automatically renewed, the journal will show:

$ journalctl -t certmonger | more
Nov 24 12:23:15 ipaserver.domain.com certmonger[36674]: Certificate named "ipaCert" in token "NSS Certificate DB" in database "/etc/httpd/alias" issued by CA and saved.

Output of getcert list

It is possible to check the status for each certificate using getcert list:

  • when the certificate is still valid, the status should be MONITORING.
  • when the certificate is near its expiration date, certmonger will request its renewal and the status will change from MONITORING to SUBMITTING and finally back to MONITORING (you may also see intermediate status PRE_SAVE_CERT and POST_SAVE_CERT).

When the renewal fails, getcert list will also show an error message. It will help determine which phase failed, and from there you will need to check the logs specific to the CA helper or to the pre-save or post-save commands.

In the next post, I will detail the errors that can arise with the helpers used with FreeIPA.

Using a Dogtag instance as external CA for FreeIPA installation

A FreeIPA user recently had issues installing FreeIPA with an external CA. He was using Dogtag certificate system as external CA and FreeIPA installation was failing, complaining about the certificate provided by Dogtag.

So I decided to try the same deployment and share my findings in this post.

A little background…

FreeIPA server can be configured to act as a Certificate Authority inside FreeIPA IDM domain. It will then be able to create the certificates used by the LDAP server, the Apache server used for the Web GUI or the users and hosts.

This CA can be set-up in different ways:

  • The CA is a root CA, meaning that its certificate is self-signed
  • or the CA is subordinate to an external, 3rd-party CA, meaning that its certificate is signed by the 3rd party CA.

There are a wide range of products that can be used as 3rd-party CAs, among which Dogtag certificate system. In this blog post, I will explain how Dogtag can provide the certificate for IPA CA.

Instructions

The following instructions apply to Fedora 24. They will:

  1. run the 1st step of ipa-server-install to generate a CSR
  2. submit the CSR to Dogtag and have Dogtag issue a certificate for FreeIPA server
  3. run the 2nd step of ipa-server-install with the certificate obtained in step 2.

For instructions to setup the Dogtag server, you can refer to this post: Dogtag installation.

 

FreeIPA server installation – step 1

In order to install FreeIPA with an externally-signed CA, we must use the –external-ca option of ipa-server-install. The installation is then a multi-step install, where:

  • ipa-server-install produces a CSR
  • we need to submit this CSR to the external CA, that will in return provide a certificate and certificate chain
  • we need to run ipa-server-install a 2nd time, with different options and providing the certificates obtained in the previous step.

So let’s run the first step of ipa-server-install:

root@ipaserver$ ipa-server-install --setup-dns \
 --auto-forwarders \
 --auto-reverse \
 -n ipadomain.com \
 -r IPADOMAIN.COM \
 -p Secret123 -a Secret123 \
 --external-ca \
 -U
[...]
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds
 [1/8]: creating certificate server user
 [2/8]: configuring certificate server instance
The next step is to get /root/ipa.csr signed by your CA and re-run /sbin/ipa-server-install as:
/sbin/ipa-server-install --external-cert-file=/path/to/signed_certificate --external-cert-file=/path/to/external_ca_certificate

 

Generation of the certificate using Dogtag

We then need to copy this CSR on the Dogtag instance and submit the CSR, approve it and export the certificate.

The submission is an important step as it allows to specify a profile. Basically, if we pick caCACert profile, we signal our intent to use the produced certificate as a Certificate Authority in our FreeIPA deployment, and the resulting certificate will contain the required extensions:

root@dogtag$ pki ca-cert-request-submit --profile caCACert --request-type pkcs10 --csr-file ipa.csr
-----------------------------
Submitted certificate request
-----------------------------
 Request ID: 7
 Type: enrollment
 Request Status: pending
 Operation Result: success

Note the Request ID as we will need it in order to approve the submission:

root@dogtag$ pki -c Secret123 -d /root/.dogtag/nssdb/ -n "PKI Administrator for example.com" cert-request-review 7 --action approve
------------------------------
Approved certificate request 7
------------------------------
 Request ID: 7
 Type: enrollment
 Request Status: complete
 Operation Result: success
 Certificate ID: 0x7

Note the Certificate ID as we will need it to export the certificate into a file ipa.cert:

root@dogtag$ pki -c Secret123 -d /root/.dogtag/nssdb/ -n "PKI Administrator for example.com" cert-show 7 --encoded --output ipa.cert

We will also need the dogtagca certificate chain:

root@dogtag$ pki ca-cert-show 1 --encoded --output dogtagca.cert

At this point, we have a new certificate and chain (ipa.cert and dogtagca.cert), that we need to copy on FreeIPA server. We can resume FreeIPA installation.

FreeIPA server installation – step 2

In order to resume FreeIPA installation, we will follow the instructions provided in step 1:

root@ipaserver$ /sbin/ipa-server-install --external-cert-file=ipa.cert --external-cert-file=dogtagca.cert

 

The installation will resume and use the ipa.cert for IPA Certificate Authority. That’s it!

Dogtag installation

Dogtag Certificate System is an open-source Certificate Authority. It allows to issue certificates,  generate Certificate Revocation Lists and much more. In this post, I am mainly interested in the installation of the Certificate Authority (to see why, you can refer to this other post, Using a Dogtag instance as external CA for FreeIPA installation).

 

Installation of the Dogtag server

First you need to get the packages for Dogtag and 389-ds (the LDAP server used by Dogtag):

root@dogtag$ dnf install -y 389-ds-base dogtag-pki

 

Dogtag relies on the LDAP server to store its data. So the installation begins with the setup of the LDAP server. It will create an instance named pki-tomcat with the suffix dc=example,dc=com:

root@dogtag$ setup-ds.pl --silent\
 General.FullMachineName=`hostname`\
 General.SuiteSpotUserID=nobody\
 General.SuiteSpotGroup=nobody\
 slapd.ServerPort=389\
 slapd.ServerIdentifier=pki-tomcat\
 slapd.Suffix=dc=example,dc=com\
 slapd.RootDN="cn=Directory Manager"\
 slapd.RootDNPwd=Secret123
Your new DS instance 'pki-tomcat' was successfully created.
Exiting . . .
Log file is '/tmp/setupjVm7VR.log

Once the LDAP server is ready, we can proceed with the Dogtag server. The installation is an interactive process, where we will pick to install the CA subsystem and provide a password for caadmin user:

root@dogtag$ pkispawn

IMPORTANT:

Interactive installation currently only exists for very basic deployments!

For example, deployments intent upon using advanced features such as:

* Cloning,
 * Elliptic Curve Cryptography (ECC),
 * External CA,
 * Hardware Security Module (HSM),
 * Subordinate CA,
* etc.,

must provide the necessary override parameters in a separate
 configuration file.

Run 'man pkispawn' for details.

Subsystem (CA/KRA/OCSP/TKS/TPS) [CA]:

Tomcat:
 Instance [pki-tomcat]:
 HTTP port [8080]:
 Secure HTTP port [8443]:
 AJP port [8009]:
 Management port [8005]:

Administrator:
 Username [caadmin]:
 Password: Secret123
 Verify password: Secret123
 Import certificate (Yes/No) [N]?
 Export certificate to [/root/.dogtag/pki-tomcat/ca_admin.cert]:

Directory Server:
 Hostname [dogtag.example.com]:
 Use a secure LDAPS connection (Yes/No/Quit) [N]?
 LDAP Port [389]:
 Bind DN [cn=Directory Manager]:
 Password: Secret123
 Base DN [o=pki-tomcat-CA]:

Security Domain:
 Name [example.com Security Domain]:

Begin installation (Yes/No/Quit)? Yes

Log file: /var/log/pki/pki-ca-spawn.20160802152151.log
Installing CA into /var/lib/pki/pki-tomcat.
Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg.
Notice: Trust flag u is set automatically if the private key is present.
Created symlink from /etc/systemd/system/multi-user.target.wants/pki-tomcatd.target to /usr/lib/systemd/system/pki-tomcatd.target.

==========================================================================
 INSTALLATION SUMMARY
 ==========================================================================

Administrator's username: caadmin
 Administrator's PKCS #12 file:
 /root/.dogtag/pki-tomcat/ca_admin_cert.p12
 Administrator's certificate database:
 /root/.dogtag/pki-tomcat/ca/alias

To check the status of the subsystem:
 systemctl status pki-tomcatd@pki-tomcat.service

To restart the subsystem:
 systemctl restart pki-tomcatd@pki-tomcat.service

The URL for the subsystem is:
 https://dogtag.example.com:8443/ca



 PKI instances will be enabled upon system boot

==========================================================================

Your Dogtag server is now up and running, ready to handle certificate requests.

 

Dogtag client configuration

In order to submit certificate requests, approve csr or export certificates, you can use Dogtag client but need first to create a NSS DB for the client. This NSSDB (by default located in ~/.dogtag/nssdb) will store the certificate that the client is using to communicate with Dogtag server:

root@dogtag$ pki -c Secret123 client-init
root@dogtag$ pk12util -i /root/.dogtag/pki-tomcat/ca_admin_cert.p12 -d /root/.dogtag/nssdb/
Enter Password or Pin for "NSS Certificate DB":
Enter password for PKCS12 file:
pk12util: PKCS12 IMPORT SUCCESSFUL

At this point, your client is able to interact with the server using the pki CLI.

New project!

I wanted to thank all the people that followed this blog and showed interest in EUS and OUD. This project was a big milestone in my professional life, I really learned a lot and enjoyed sharing my knowledge.

Since May this year, I moved to a new project as Software Development and Integration Engineer at Red Hat, in the Free IPA team. My new blog posts will still be about Identity Management, but with different products this time. I hope that you will continue to be interested in my articles!

EUS and OUD proxy: configure the proxy to use a non-directory manager user

When OUD is used for EUS as a proxy server, it needs specific credentials to connect to the LDAP server that is actually storing the users and groups.

Those credentials are set in the configuration of the proxy-ldap-workflow-element, through the parameters remote-ldap-server-bind-dn and remote-root-dn. Usually, the credentials for the LDAP server administrator are used: cn=directory manager for ODSEE or OUD, cn=administrator,cn=users,<baseDN> for Active Directory.

Some customers do not want to use the LDAP administrator credentials. In this case, it is possible to use an alternate user identity, but this user must comply with specific requirements depending on the LDAP server flavour.

It is also possible to use 2 different users, one that will be used as remote-root-dn and another one for remote-ldap-server-bind-dn.

Reminder: the remote-ldap-server-bind-dn is the identity used to connect to the LDAP server for all the operations directly performed by the Database. The remote-root-dn is the identity used to perform internal operations triggered by the Database.

For instance, if the database connects to OUD proxy and performs a search for (uid=joe) with a control requesting the user account status, the search may have to be handled in multiple steps by OUD proxy, depending on the LDAP server flavour. A first step would be the actual search on the LDAP server, and a second step would translate the control into an internal extended operation requesting the user account status.

Follow the steps corresponding to your LDAP server.

Active Directory deployments

  • The remote-ldap-server-bind-dn must be able to read all the attributes on dc=example,dc=com.
  • The remote-root-dn must be able to read all the attributes on dc=example,dc=com.

ODSEE deployments

  • The remote-ldap-server-bind-dn must be able to read dc=example,dc=com. You can use the following command to define the required ACI on ODSEE (replace cn=eusproxy,dc=example,dc=com with the appropriate value):
$ ldapmodify -h odseehost -p odseeport -D odseeadmin -w odseepassword
dn: dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr="*")(version 3.0; acl "Read access to eus proxy user"; allow (read, search, compare) userdn="ldap:///cn=eusproxy,dc=example,dc=com";)
  • The remote-root-dn must be able to read dc=example,dc=com (replace cn=eusroot,dc=example,dc=com with the appropriate value):
$ ldapmodify -h odseehost -p odseeport -D odseeadmin -w odseepassword
dn: dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr="*")(version 3.0; acl "example"; allow (read,search,compare) userdn="ldap:///cn=eusroot,dc=example,dc=com";)
  • The remote-root-dn must be able to use the Password Policy Account Management extended operation
$ ldapmodify -h odseehost -p odseeport -D odseeadmin -w odseepassword
dn: oid=1.3.6.1.4.1.42.2.27.9.6.25,cn=features,cn=config
changetype: modify
add: act
aci: (targetattr != "aci")(version 3.0; acl "Pwd Policy Acct Mgt for eus proxy  user"; allow (read, search, compare) userdn="ldap:///cn=eusroot,dc=example,dc=com";)
  • The remote-root-dn must be able to use the Account Usable Control (already allowed by default).

OUD deployments

  • The remote-ldap-server-bind-dn must be able to use the control 2.16.840.1.113894.1.8.16. Define a global-aci using:
$ dsconfig -h oudhost -p oudadminport -D "cn=directory manager" -j pwd.txt -X -n set-access-control-handler-prop --add global-aci:\(targetcontrol=\"2.16.840.1.113894.1.8.16\"\)\(version\ 3.0\; acl\ \"Allow\ eusproxy\ user\ to\ use\ EUS\ control\"\; allow\(read\)\ userdn=\"ldap:///cn=eusproxy,dc=example,dc=com\"\;\)
  • The remote-ldap-server-bind-dn must be able to read dc=example,dc=com and to write orclaccountstatusevent attribute on users below dc=example,dc=com. Use ldapmodify to create the following aci:
dn: dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr="orclaccountstatusevent")(version 3.0; acl "EUS write orclaccountstatusenabled"; allow (write) userdn="ldap:///cn=eusproxy,dc=example,dc=com";)
aci: (targetattr="*")(version 3.0; acl "EUS reads users and groups"; allow (read,search,compare) userdn="ldap:///cn=eusproxy,dc=example,dc=com";)
  • The remote-root-dn must be able to use the Password Policy State extended operation. Define a global-aci using:
$ dsconfig -h oudhost -p oudadminport -D "cn=directory manager" -j pwd.txt -X -n set-access-control-handler-prop --add global-aci:\(extop=\"1.3.6.1.4.1.26027.1.6.1\"\)\(version\ 3.0\; acl\ \"Allow\ eusroot\ user\ to\ use\ extop\"\; allow\(read\)\ userdn=\"ldap:///cn=eusroot,dc=example,dc=com\"\;\)
  • The remote-root-dn must have the password reset privilege. Use ldapmodify to add the privilege:
dn: cn=eusroot,dc=example,dc=com
changetype: modify
add: ds-privilege-name
ds-privilege-name: password-reset
  • The remote-root-dn must have the rights to read the tree below the base DN. Use ldapmodify to define the following act:
dn: dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr="*")(version 3.0; acl "EUS reads users and groups"; allow (read,search,compare) userdn="ldap:///cn=eusroot,dc=example,dc=com";)

Novell eDirectory deployments

Refer to Novell documentation to define the appropriate eDirectory rights:

  • The remote-ldap-server-bind-dn must have read access to all the attributes on dc=example,dc=com.
  • The remote-root-dn must be able to retrieve the Universal Password and to write on dc=example,dc=com

Use an alternate identity for EUS configuration

During EUS setup, the administrator needs to provide a user DN and password to authenticate to the directory server (for instance during the dbca step, or while using eusm or Enterprise Manager).

 

In some companies, the database and the LDAP server are managed by different teams and the LDAP administrator credentials cannot be provided to the database administrator. In this case, it is possible to administer EUS using an alternate identity, i.e not cn=directory manager. The requirements for this alternate identity are the following:

  • the user must be a member of the group cn=OracleContextAdmins,cn=Groups,cn=OracleContext,<base DN>
  • the user must have the password-reset privilege

 

Here is an example of configuration steps: create a input.ldif file with the following content

$ cat input.ldif
dn: cn=eusadmin,cn=oraclecontext
changetype: add
objectclass: inetorgperson
cn: eusadmin
sn: eusadmin
uid: eusadmin
userpassword: password

dn: cn=eusadmin,,cn=oraclecontext
changetype: modify
add: ds-privilege-name
ds-privilege-name: password-reset

dn: cn=OracleContextAdmin,cn=groups,cn=OracleContext,dc=example,dc=com
changetype: modify
add: uniquemember
uniquemember: cn=eusadmin,cn=oraclecontext

And perform

$ $ORACLE_HOME/bin/ldapmodify -h localhost -p 1389 -D "cn=directory manager" -w password -f input.ldif

Note 1: this EUS admin user can be stored in your preferred location inside the DIT, but NOT BELOW cn=oraclecontext,<base DN>. For instance, cn=eusadmin,ou=people,dc=example,dc=com is valid, but cn=eusadmin,cn=oraclecontext,dc=example,dc=com is NOT valid.

Note 2: the EUS admin user does not have to be named eususer.

Note 3: if OUD is installed as a proxy server, then the EUS admin user must be stored locally inside OUD proxy, and for instance cn=eususer,cn=oraclecontext would be a valid location.

How to configure EUS + SSL authentication with OUD

During an EUS authentication, there are 2 communication channels: one between the sql client and the database, and another one between the database and the LDAP server. In a previous post, I explained that the database-to-OUD communication can be authenticated either through user/password or SSL.

EUS_auth

The sql client-to-database connection also supports multiple authentication methods:

  • user/password
  • SSL
  • Kerberos (described in this post)

In this post, I will explain how to configure SSL authentication. Note that SSL authentication is usually used in conjunction with DB-to-OUD SSL authentication. This post assumes that DB-to-OUD SSL authentication has already been set up as described there and that the database already has a wallet containing the DB certificate.

Using sqlplus, it is possible to connect to the DB using a client certificate instead of username/password. In order to do this, the DB must be configured with an SSL listener, and the sqlplus client must be configured to use a client certificate.

  1. Configure the database to allow SSL authentication:
    The file sqlnet.ora located in $ORACLE_HOME/network/admin must be modified to look like the following:

    NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
      
    ADR_BASE = <$ORACLE_BASE>
    
    #ADDED FOR SSL 
    SQLNET.AUTHENTICATION_SERVICES= (TCPS, BEQ)
    SSL_CLIENT_AUTHENTICATION = TRUE
    WALLET_LOCATION = 
      (SOURCE =
        (METHOD = FILE)
        (METHOD_DATA =
          (DIRECTORY = <$ORACLE_BASE>/admin/<$ORACLE_SID>/wallet)
        )
      )
  2. Configure a SSL listener on the database:
    The listeners are defined in the configuration file listener.ora, located in $ORACLE_HOME/network/admin. The content of this file defines the port and protocol used by the listener, and the location of the wallet that will store the database certificate.

    LISTENER =
      (DESCRIPTION_LIST =
        (DESCRIPTION =
          (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
          (ADDRESS = (PROTOCOL = TCP)(HOST = <hostname>)(PORT = 1521))
    #ADDED FOR SSL
          (ADDRESS = (PROTOCOL = TCPS)(HOST = <hostname>)(PORT = 1575))
        )
      )
    
    ADR_BASE_LISTENER = <$ORACLE_BASE>
    
    #ADDED FOR SSL
    WALLET_LOCATION = (SOURCE=
      (METHOD = FILE)
        (METHOD_DATA =
          (DIRECTORY=<$ORACLE_BASE>/admin/<$ORACLE_SID>/wallet)
        )
      )
    
  3. Restart the listener:
    the modification of listeners.ora requires to restart the database listener with the following commands

    $ $ORACLE_HOME/bin/lsnrctl stop
    $ $ORACLE_HOME/bin/lsnrctl start
  4. Configure the sql client to use SSL authentication:

    The sql client reads its configuration from sqlnet.ora and tnsnames.ora. By default, those files are located in the same location as the DB configuration files ($ORACLE_HOME/network/admin), which can cause issues because the client and the server use a different wallet. In order to avoid configuration issues, it is recommended to have separate configuration files. To do this, the environment variable TNS_ADMIN is used to configure the path containing the client configuration files:

    $ setenv TNS_ADMIN /path/to/files
    

    or

    $ export TNS_ADMIN=/path/to/files
    

    Then the file tnsnames.ora will define how to reach the database using SSL:

    $ cat /path/to/files/tnsnames.ora 
    ORCL11G =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCPS)(HOST = <hostname>)(PORT = 1575))
        (CONNECT_DATA =
          (SERVER = DEDICATED)
          (SERVICE_NAME = orcl11g)
        )
      )
    

    And finally the file sqlnet.ora defines the SSL configuration (wallet location…):

    $ cat sqlnet.ora 
    NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
    SQLNET.AUTHENTICATION_SERVICES = (TCPS,BEQ)
    SSL_CLIENT_AUTHENTICATION = TRUE 
    WALLET_LOCATION =
      (SOURCE =
        (METHOD = FILE)
        (METHOD_DATA =
          (DIRECTORY = /path/to/client/wallet)
        )
      )
  5. Create the client wallet and the client certificate:
    The client wallet will store the client certificate as well as the certificates trusted by the client (i.e. the server certificate).
    The client certificate must contain a DN that corresponds to a LDAP entry on OUD server. The following commands create a client wallet, then add a self-signed certificate to the wallet for the user cn=Joe, and finally export the self-signed certificate to a file.

    $ orapki wallet create -wallet /path/to/client/wallet -pwd <password> -auto_login
    
    $ orapki wallet add -wallet /path/to/client/wallet -dn cn=Joe,ou=users,dc=example,dc=com -keysize 1024 -self_signed -validity 365 -pwd <password>
    
    $ orapki wallet export -wallet /path/to/client/wallet -dn cn=Joe,ou=users,dc=example,dc=com -cert joe-cert.txt
    

  6. Add the client certificate to the DB wallet:
    the database must trust the client certificate in order to accept SSL connections. As the above steps created a self-signed user certificate, the client certificate has to be imported as a trusted cert in the DB wallet.

    $ orapki wallet add -wallet <PathToDBWallet> -cert joe-cert.txt -trusted_cert -pwd <password>
    
  7. Add the DB certificate to the client wallet:
    the client must trust the DB certificate. If the DB certificate is a self-signed certificate, this means that the DB cert must be added to the client wallet.

    $ orapki wallet add -wallet /path/to/client/wallet -cert db-cert.txt -trusted_cert -pwd <password>
  8. Allow EUS to accept SSL connections between the client and the database:

     log in to Enterprise Manager: https://<hostname>:1158/em

    In the “Server” tab, select “Enterprise User Security” in the “Security” section, then “Manage Enterprise Domain” and click on “Configure”. In the “Configuration” tab, select the authentication methods that you want to allow between the client and the DB: Password, SSL, Kerberos.

  9. Test the SSL connection between the client and the database:

    The following command reads the sql configuration files, extracts the path to the client wallet, then connects to the DB using the certificate found in the client wallet:

    $ export TNS_ADMIN=/path/to/files
    $ sqlplus /@orcl11g
    
    SQL*Plus: Release 11.2.0.2.0 Production on Wed Mar 28 11:18:57 2012
    
    Copyright (c) 1982, 2010, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    
    SQL>

Once authenticated with the client certificate, the enterprise user will be mapped to a schema exactly as a user/password authenticated user. EUS will also find the groups for the LDAP user and grant the corresponding roles to the EUS user.

How to configure EUS + Kerberos authentication with OUD

During an EUS authentication, there are 2 communication channels: one between the sql client and the database, and another one between the database and the LDAP server. In a previous post, I explained that the database-to-OUD communication can be authenticated either through user/password or SSL.

EUS_auth

The sql client-to-database connection also supports multiple authentication methods:

  • user/password
  • SSL
  • Kerberos

In this post, I will explain how to configure Kerberos authentication. I am assuming that you already have a Kerberos server up and running.

 

  1. Create a Kerberos Principal for the Database:
    The database service needs to have a corresponding Kerberos principal in the Kerberos server. If your Kerberos server is running on a Linux machine, this can be done with the kadmin utility. In my example, the kerberos administrator is kws/admin and creates a principal for the database having ORACLE_SID=orcl11g, running on host dbhost, with a Kerberos realm EXAMPLE.COM:

    $ kadmin -p kws/admin
    Authenticating as principal kws/admin with password.
    Password for kws/admin@EXAMPLE.COM: 
    kadmin:  add_principal -randkey orcl11g/dbhost
    WARNING: no policy specified for orcl11g/dbhost@EXAMPLE.COM; defaulting to no policy
    Principal "orcl11g/dbhost@EXAMPLE.COM" created.
    kadmin: ktadd -k /tmp/orcl11g.keytab orcl11g/dbhost@EXAMPLE.COM
    Entry for principal orcl11g/dbhost@EXAMPLE.COM with kvno 2, encryption type AES-256 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/tmp/orcl11g.keytab.
    Entry for principal orcl11g/dbhost@EXAMPLE.COM with kvno 2, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/tmp/orcl11g.keytab.
    Entry for principal orcl11g/dbhost@EXAMPLE.COM with kvno 2, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/tmp/orcl11g.keytab.
    Entry for principal orcl11g/dbhost@EXAMPLE.COM with kvno 2, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/tmp/orcl11g.keytab.
    kadmin:  exit

    The add_principal command creates a principal, and the ktadd command creates and stores the secret key in the file /tmp/orcl11g.keytab.

  2. Copy the keytab to the database host:
    $ scp /tmp/orcl11g.keytab oracle@dbhost:/path/to/orcl11g.keytab
  3. Create a Kerberos principal for the EUS user:
    The user that will use sqlplus also needs to be provisioned in the KDC. In the example, the user principal name is eususer@EXAMPLE.COM:

    $ kadmin -p kws/admin
    Authenticating as principal kws/admin with password.
    Password for kws/admin@EXAMPLE.COM: 
    kadmin:  add_principal eususer
    WARNING: no policy specified for eususer@EXAMPLE.COM; defaulting to no policy
    Enter password for principal "eususer@EXAMPLE.COM": 
    Re-enter password for principal "eususer@EXAMPLE.COM": 
    Principal "eususer@EXAMPLE.COM" created.
    kadmin:  exit
    $
  4. Configure the database to allow Kerberos authentication:
    By default the database supports only user/password authentication. In order to allow Kerberos authentication, the file sqlnet.ora (located in $ORACLE_HOME/network/admin) needs to be modified to look like the following:

    $ cat sqlnet.ora
    NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
    ADR_BASE = /export/home/oracle/app/oracle
    
    #ADDED FOR KERBEROS
    SQLNET.AUTHENTICATION_SERVICES= (BEQ, KERBEROS5)
    SQLNET.AUTHENTICATION_KERBEROS5_SERVICE=orcl11g
    SQLNET.KERBEROS5_CONF=/etc/krb5.conf
    SQLNET.KERBEROS5_KEYTAB=/path/to/orcl11g.keytab
    SQLNET.KERBEROS5_CONF_MIT=true
    
    
  5. Configure the database host as a Kerberos client:
    Edit /etc/krb5.conf and set the values for the Kerberos server in kdc and admin_server properties. In my example, the Kerberos server is running on kdc_host.fr.oracle.com:

    [...]
    [libdefaults]
     default_realm = EXAMPLE.COM
     dns_lookup_realm = false
     dns_lookup_kdc = false
     ticket_lifetime = 24h
     forwardable = yes
    [realms]
     EXAMPLE.COM = {
      kdc = kdc_host.fr.oracle.com
      admin_server = kdc_host.fr.oracle.com
     }
    [domain_realm]
     .example.com = EXAMPLE.COM
     example.com = EXAMPLE.COM
     .fr.oracle.com = EXAMPLE.COM
     fr.oracle.com = EXAMPLE.COM
    [...]

    You can test that the database host is properly configured by trying to get a Kerberos ticket-granting ticket for the database with the kinit utility:

    $ kinit -k -t /path/to/orcl11g.keytab orcl11g/dbhost@EXAMPLE.COM
    $ klist
    Ticket cache: FILE:/tmp/krb5cc_54321
    Default principal: orcl11g/dbhost@EXAMPLE.COM
    Valid starting Expires Service principal
    03/03/16 11:35:50 03/03/16 21:35:50 krbtgt/EXAMPLE.COM@EXAMPLE.COM
     renew until 03/04/16 11:35:50
    Kerberos 4 ticket cache: /tmp/tkt54321
    klist: You have no tickets cached
    $
  6. Configure the sql client host as a Kerberos client:
    this step is the same as the previous one (edition of /etc/krb5.conf), except that it must be done on the host where the sql client is running. You can test that the Kerberos configuration is working by getting a ticket-granting ticket for the Enterprise user:

    $ kinit eususer
    Password for eususer@EXAMPLE.COM: 
    $ klist
    Ticket cache: FILE:/tmp/krb5cc_54321
    Default principal: eususer@EXAMPLE.COM
    
    Valid starting     Expires            Service principal
    03/29/12 16:32:41  03/30/12 02:32:41  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 03/30/12 16:32:38
    
    
    Kerberos 4 ticket cache: /tmp/tkt54321
    klist: You have no tickets cached
    $
    
  7. Configure the sql client to authenticate with Kerberos:
    Create a directory to store the SQL client configuration, and define the environment variable TNS_ADMIN to point to this directory:

    $ mkdir -p /path/for/sql/config
    $ export TNS_ADMIN=/path/for/sql/config

    In this directory, create a tnsnames.ora file defining how to contact the database:

    $ cat $TNS_ADMIN/tnsnames.ora
    ORCL11G = (DESCRIPTION =
     (ADDRESS = (PROTOCOL = TCP)(HOST = dbhost)(PORT = 1521))
     (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = orcl11g))
     )

    In the same directory, create a sqlnet.ora file defining the connection method to the database:

    $ cat $TNS_ADMIN/sqlnet.ora 
    NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
    SQLNET.AUTHENTICATION_SERVICES = (BEQ,KERBEROS5)
    SQLNET.KERBEROS5_CONF=/etc/krb5.conf
    SQLNET.KERBEROS5_CONF_MIT=true
  8. Link the LDAP user entry to the Kerberos principal:
    The Kerberos principal that will be used with SQL client must correspond to a LDAP user. In order to do this, you have 2 options:

    • if your LDAP users have the objectclass orcluserV2, you can add the attribute krbprincipalname to the user entry. For instance, if Kerberos principal eususer must correspond to LDAP entry cn=user,dc=example,dc=com, you must add krbprincipalname: eususer@EXAMPLE.COM to the entry.
      EUS_kerberos_auth
    • if your LDAP users do not have the objectclass orcluserV2, you can user another attribute to store the Kerberos principal name. In this case, you must declare this attribute by setting the attribute orclcommonkrbprincipalattribute of the entry cn=Common,cn=Products,cn=OracleContext,dc=example,dc=com. For instance:
      dn: cn=Common,cn=Products,cn=OracleContext,dc=example,dc=com
      orclcommonkrbprincipalattribute: mail

      will correspond to a user entry

       dn: cn=user,dc=example,dc=com
       mail: eususer@EXAMPLE.COM
  9. Test the connection:

    The user must first get a Kerberos ticket-granting ticket, then call sqlplus with the / option that specifies to use the Kerberos TGT:

    $ kinit eususer
    Password for eususer@EXAMPLE.COM: 
    $ klist
    Ticket cache: FILE:/tmp/krb5cc_54321
    Default principal: eususer@EXAMPLE.COM
    
    Valid starting     Expires            Service principal
    03/29/12 16:47:29  03/30/12 02:47:29  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 03/30/12 16:47:27
    
    
    Kerberos 4 ticket cache: /tmp/tkt54321
    klist: You have no tickets cached
    $ sqlplus /@orcl11g
    
    SQL*Plus: Release 11.2.0.2.0 Production on Thu Mar 29 16:47:35 2012
    
    Copyright (c) 1982, 2010, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    
    SQL> exit
    Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    $ klist
    Ticket cache: FILE:/tmp/krb5cc_54321
    Default principal: eususer@EXAMPLE.COM
    
    Valid starting     Expires            Service principal
    03/29/12 16:47:29  03/30/12 02:47:29  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 03/30/12 16:47:27
    03/29/12 16:47:35  03/30/12 02:47:29  orcl11g/dbhost@EXAMPLE.COM
        renew until 03/30/12 16:47:27
    
    
    Kerberos 4 ticket cache: /tmp/tkt54321
    klist: You have no tickets cached
    $ 
    
    

    You can see using klist utility that after the sql connection, the user has a second ticket corresponding to the database service.

Troubleshooting tips:

If your setup is not working, you can enable Kerberos logs by adding the following in the /etc/krb5.conf of the Kerberos server:

[logging]
 kdc = FILE:/var/log/krb5kdc.log

then restart the KDC to apply the changes:

$ sudo service krb5-kdc restart

The problem can be linked to the host names (if the database hostname is defined as a fully-qualified-domain-name but the database service principal uses only the short name or vice-versa). The logs will show you which service principal is used by the database and if there is a mismatch.

Another common problem is when the database is not able to find any user entry matching the Kerberos principal from the ticket. In this case, you can have a look at OUD access log and check the search done by the database to find the user entry. This will be a search equivalent to:

$ ldapsearch -b <orclcommonusersearchbase> "(<orclcommonkrbprincipalattribute>=<krb_principal from the ticket>)"

You need to make sure that the database is using the proper search base and search filter. The search base is configured in the entry cn=Common,cn=Products,cn=OracleContext,dc=example,dc=com, in the attribute orclcommonusersearchbase. To find its value, you can perform the following search operation:

$ OracleUnifiedDirectory/bin/ldapsearch -h $LDAPSERVER -p $PORT -b cn=common,cn=products,cn=oraclecontext,$BASEDN  "(objectclass=*)" orclcommonusersearchbase
dn: cn=Common,cn=Products,cn=OracleContext,dc=example,dc=com
orclcommonusersearchbase: ou=people,dc=example,dc=com


EUS: ORA-28030 error with CONNECT/DISCONNECT in OUD logs

You may face the following issue with EUS and OUD. When trying to authenticate using sqlplus, the authentication fails and sqlplus displays:

ORA-28030: Server encountered problems accessing LDAP directory service

Unfortunately, OUD access logs do not help a lot as you can find only the following:

[23/Feb/2016:13:48:29 +0100] CONNECT conn=73 from=10.166.139.54:30238 to=10.166.139.64:1636 protocol=LDAPS
[23/Feb/2016:13:48:29 +0100] DISCONNECT conn=73 reason="Client Disconnect"

 

This type of error happens when the database is not able to find its credentials in its wallet. To troubleshoot, first check which wallet is picked by the database, then make sure that the wallet contains the DN and password for the database.

  1. Enable the database logs to find the wallet location
    Edit $ORACLE_HOME/network/admin/sqlnet.ora and add the following lines:

    DIAG_ADR_ENABLED=OFF
    TRACE_DIRECTORY_SERVER=/path/to/logs/server
    TRACE_LEVEL_SERVER=16
    TRACE_LEVEL_CLIENT=16
    TRACE_DIRECTORY_CLIENT=/path/to/logs/client
    
    
  2. run the sqlplus command and examine the logs in /path/to/logs/server. They will contain references to WALLET_LOCATION and display the path used to find the wallet.
  3. If the path is not consistent with your expectations (by default the wallet is in $ORACLE_BASE/admin/$ORACLE_SID/wallet), edit $ORACLE_HOME/network/admin/sqlnet.ora and add the following lines:
    WALLET_LOCATION =
      (SOURCE =
        (METHOD = FILE)
        (METHOD_DATA =
          (DIRECTORY = /path/to/db/wallet)
        )
      )
    
    
  4. Make sure that the specified wallet is an auto-login wallet (the wallet directory must contain a cwallet.sso file):
    $ ls /path/to/db/wallet
    cwallet.sso ewallet.p12
  5. Make sure that the specified wallet contains a DN and password for the database (they were generated by dbca when the database was registered in the LDAP server):
    $ mkstore -wrl /path/to/db/wallet -viewEntry ORACLE.SECURITY.DN
    Oracle Secret Store Tool : Version 11.2.0.2.0 - Production
    Copyright (c) 2004, 2010, Oracle and/or its affiliates. All rights reserved.
    Enter wallet password: ******** 
    ORACLE.SECURITY.DN = cn=orcl11gr2,cn=OracleContext,dc=eusovd,dc=com
    
    $ mkstore -wrl /path/to/db/wallet -viewEntry ORACLE.SECURITY.PASSWORD
    Oracle Secret Store Tool : Version 11.2.0.2.0 - Production
    Copyright (c) 2004, 2010, Oracle and/or its affiliates. All rights reserved.
    Enter wallet password: ******** 
    ORACLE.SECURITY.PASSWORD = <password generated by dbca>
    
    
  6. If it is not the case, you can re-run dbca and choose to generate a new password. dbca will then create the ORACLE.SECURITY.DN and ORACLE.SECURITY.PASSWORD entries in the wallet.