EUS: ORA-28030 error with CONNECT/DISCONNECT in OUD logs

You may face the following issue with EUS and OUD. When trying to authenticate using sqlplus, the authentication fails and sqlplus displays:

ORA-28030: Server encountered problems accessing LDAP directory service

Unfortunately, OUD access logs do not help a lot as you can find only the following:

[23/Feb/2016:13:48:29 +0100] CONNECT conn=73 from=10.166.139.54:30238 to=10.166.139.64:1636 protocol=LDAPS
[23/Feb/2016:13:48:29 +0100] DISCONNECT conn=73 reason="Client Disconnect"

 

This type of error happens when the database is not able to find its credentials in its wallet. To troubleshoot, first check which wallet is picked by the database, then make sure that the wallet contains the DN and password for the database.

1. Enable the database logs to find the wallet location
   Edit $ORACLE_HOME/network/admin/sqlnet.ora and add the following lines:

   DIAG_ADR_ENABLED=OFF
   TRACE_DIRECTORY_SERVER=/path/to/logs/server
   TRACE_LEVEL_SERVER=16
   TRACE_LEVEL_CLIENT=16
   TRACE_DIRECTORY_CLIENT=/path/to/logs/client
   
   

2. run the sqlplus command and examine the logs in /path/to/logs/server. They will contain references to WALLET_LOCATION and display the path used to find the wallet.
3. If the path is not consistent with your expectations (by default the wallet is in $ORACLE_BASE/admin/$ORACLE_SID/wallet), edit $ORACLE_HOME/network/admin/sqlnet.ora and add the following lines:

   WALLET_LOCATION =
     (SOURCE =
       (METHOD = FILE)
       (METHOD_DATA =
         (DIRECTORY = /path/to/db/wallet)
       )
     )
   
   

4. Make sure that the specified wallet is an auto-login wallet (the wallet directory must contain a cwallet.sso file):

   $ ls /path/to/db/wallet
   cwallet.sso ewallet.p12

5. Make sure that the specified wallet contains a DN and password for the database (they were generated by dbca when the database was registered in the LDAP server):

   $ mkstore -wrl /path/to/db/wallet -viewEntry ORACLE.SECURITY.DN
   Oracle Secret Store Tool : Version 11.2.0.2.0 - Production
   Copyright (c) 2004, 2010, Oracle and/or its affiliates. All rights reserved.
   Enter wallet password: ******** 
   ORACLE.SECURITY.DN = cn=orcl11gr2,cn=OracleContext,dc=eusovd,dc=com
   
   $ mkstore -wrl /path/to/db/wallet -viewEntry ORACLE.SECURITY.PASSWORD
   Oracle Secret Store Tool : Version 11.2.0.2.0 - Production
   Copyright (c) 2004, 2010, Oracle and/or its affiliates. All rights reserved.
   Enter wallet password: ******** 
   ORACLE.SECURITY.PASSWORD = <password generated by dbca>
   
   

6. If it is not the case, you can re-run dbca and choose to generate a new password. dbca will then create the ORACLE.SECURITY.DN and ORACLE.SECURITY.PASSWORD entries in the wallet.

Advertisements

Tips to troubleshoot EUS + OUD

I am often involved in customer issues involving EUS and OUD, and wanted to share some tips and strategies that could help in this type of situation.

As I am working mainly on OUD, the first thing that I usually check is OUD access log. It is enabled by default and located in $OUD_INSTANCE/OUD/logs/access. This file contains a trace of all the operations received by OUD and is often a great help in understanding where the problem lies.

I usually configure OUD so that it also logs internal operations and LDAP controls:

dsconfig set-log-publisher-prop \
 --publisher-name File-Based\ Access\ Logger \
 --set log-controls:true \
 --add operations-to-log:internal \
 --hostname $OUDHOST \
 --port 4444 \
 --trustAll \
 --bindDN cn=directory\ manager \
 --bindPasswordFile pwd.txt \
 --no-prompt

 

To debug an EUS issue, I perform the failing EUS command and look at the logs generated by this command in OUD access log. For instance, I run sqlplus joe/password and check that the command triggered operations on OUD server:

• if the command does not produce any log, this means that the authentication failed before the database actually contacted the OUD server. In this case, the root cause is likely to be a configuration issue on the database side (for instance the DB points to another LDAP server, or was not able to find its own DN/password in its wallet…)
  The next debugging step will be to enable the logs on the database.
• if the command produces logs in OUD, then the DB correctly points to OUD but there is either a communication issue (SSL, DB authentication…) or an EUS configuration issue (not able to find a user-schema mapping, unable to access the user’s password…)
  Usually the last LDAP operation can provide hints on the root cause. For instance, if the user could not be associated to any shared schema, the last LDAP operation will be a SEARCH with a filter (objectclass=orcldbEntrylevelMapping) or (objectclass=orcldbSubtreelevelMapping) that does not find any entry (nentries=0).

 

After you have identified if the issue happens between the sql client and the DB or between the DB and OUD, you can also have a look at OUD Admin guide, which provides a checklist with common configuration issues in the Troubleshooting section of Chapter 31: Integrating Oracle Unified Directory with Oracle Enterprise User Security.

EUS, OUD, AD and nested groups

EUS does not support nested groups. This means that an enterprise role is granted to a user only if the user is a direct member of the group. But it is possible to leverage the LDAP_MATCHING_RULE_IN_CHAIN feature of active directory.

If you perform a ldapsearch against AD with a filter like this: (member:1.2.840.113556.1.4.1941:=cn=myuser,cn=users,dc=example,dc=com)
then AD will return all the groups the user myuser belongs to, including the nested groups.

The trick here is to transform the search filter performed by EUS (member=cn=myuser,cn=users,dc=example,dc=com) into a search with the matching rule (member:1.2.840.113556.1.4.1941:=cn=myuser,cn=users,dc=example,dc=com). This is done by creating a transformation in OUD proxy that operates on the member attribute in a search filter, and inserting this transformation in the processing.

$ dsconfig create-transformation \
                  --set client-attribute:member=%member:1.2.840.113556.1.4.1941:% \
                  --type map-attribute \
                  --transformation-name adrecursivemember  
$ dsconfig create-workflow-element \
                 --set enabled:true \
                 --set next-workflow-element:proxy-we1 \
                 --set transformation:adrecursivemember \
                 --type transformations \
                 --element-name adrecursivegroupsearch \
                 --set entry-match-filter:'(objectclass=dummy)'
$ dsconfig set-workflow-element-prop \
          --element-name eus-we1 \
          --set next-workflow-element:adrecursivegroupsearch

Note that this transformation requires OUD 11gR2PS3 (a bug in older releases prevents its use).

EUS, OUD and dynamic groups

EUS allows to define enterprise roles or proxy permissions and assign them to all the members of a given group. The issue is that EUS supports only static groups (it expects the groups to have the objectclass “groupofuniquenames” or “groupofnames”), and eusm will fail to grant an Enterprise Role to a dynamic group with the following error:

$ eusm grantRole enterprise_role=my_role domain_name=OracleDefaultDomain realm_dn=dc=example,dc=com group_dn=cn=dyngroup,ou=groups,dc=example,dc=com ldap_host=$OUD_HOST ldap_port=1389 ldap_user_dn="cn=directory manager" ldap_user_password=$PWD
EUSException: There is no such group in directory

Enterprise Manager will also fail with this type of message, even though the group exists:

Configure Domain : OracleDefaultDomain - Errors
my_role - cn=dyngroup,ou=groups,dc=example,dc=com : EUSException: There is no such group in directory

 

OUD provides a nice feature to workaround this issue: Virtual Static Groups. You need to create a virtual static group that will appear as a static group to Enterprise User Security, but mirrors a dynamic group.

For instance, perform a ldapmodify with the following ldif to define a virtual static group for your group cn=dyngroup:

dn: cn=virtualstatic,ou=groups,dc=example,dc=com
cn: virtualstatic
ds-target-group-dn: cn=dyngroup,ou=groups,dc=example,dc=com
objectclass: groupofuniquenames
objectclass: ds-virtual-static-group
objectclass: top

You will then be able to use this group instead of the dynamic group in eusm or EM, and you can still benefit from the dynamicity of the original group:

$ eusm grantRole enterprise_role=my_role domain_name=OracleDefaultDomain realm_dn=dc=example,dc=com group_dn=cn=virtualstatic,ou=groups,dc=example,dc=com ldap_host=$OUD_HOST ldap_port=1389 ldap_user_dn="cn=directory manager" ldap_user_password=$PWD

How to use SSL authentication between the EUS database and OUD

During an EUS authentication, Oracle Database connects to OUD server using a simple bind over SSL. The username and the password are stored in the database wallet (default location is $ORACLE_BASE /admin/<ORACLE_SID>/wallet), and can be read using the mkstore command:

$ $ORACLE_HOME/bin/mkstore -wrl $ORACLE_BASE/admin/$ORACLE_SID/wallet -viewEntry ORACLE.SECURITY.DN
Oracle Secret Store Tool : Version 11.2.0.2.0 - Production
Copyright (c) 2004, 2010, Oracle and/or its affiliates. All rights reserved.

Enter wallet password: 
 
ORACLE.SECURITY.DN = cn=orcl11g,cn=OracleContext,dc=example,dc=com
$ $ORACLE_HOME/bin/mkstore -wrl $ORACLE_BASE/admin/$ORACLE_SID/wallet -viewEntry ORACLE.SECURITY.PASSWORD
Oracle Secret Store Tool : Version 11.2.0.2.0 - Production
Copyright (c) 2004, 2010, Oracle and/or its affiliates. All rights reserved.

Enter wallet password: 
 
ORACLE.SECURITY.PASSWORD = rIwfee96
$

This behavior can be changed, and the Database can switch to certificate authentication over SSL. In order to do this:

1. Create a certificate for the Database

   For testing purpose, it is possible to create a self-signed certificate using the orapki utility (located in $ORACLE_HOME/bin).

   $ orapki wallet add -wallet <PathToDBWallet> -dn cn=<ORACLE_SID>,cn=oraclecontext,dc=example,dc=com -keysize 1024 -self_signed -validity 365 -pwd <WalletPassword>
   

   The DB certificate must then be exported to a file:

   $ orapki wallet export -wallet <PathToDBWallet> -dn cn=<ORACLE_SID>,cn=oraclecontext,dc=example,dc=com -cert db-cert.txt
   

2. Add the DB certificate to OUD truststore

   By default, OUD installed with EUS option configures SSL and a JKS truststore. The trusted certificates must be imported into <OUD_INSTANCE>/config/truststore using /usr/bin/keytool utility:

   $ keytool -importcert -alias db-cert  -file db-cert.txt -keypass <value in keystore.pin> -keystore <OUD_INSTANCE>/config/truststore -storepass <value in keystore.pin>
   

   OUD must be stopped and restarted for the truststore to be re-read.

3. Add OUD certificate to the DB wallet

   By default, OUD uses a self-signed certificate that must be added to the DB truststore. You first need to export the certificate using keytool:

   $ keytool -exportcert  -alias server-cert -keystore <OUD_INSTANCE>/config/keystore -storepass <value in keystore.pin> -file oud-cert.txt
   

   Then the certificate must be imported in the DB wallet using orapki:

   $ orapki wallet add -wallet <PathToDBWallet> -cert oud-cert.txt -trusted_cert -pwd <WalletPassword>
   

4. Configure the DB to use certificate authentication instead of password authentication.

   $ sqlplus sys as sysdba
   
   SQL*Plus: Release 11.2.0.2.0 Production on Thu Feb 4 10:55:59 2016
   
   Copyright (c) 1982, 2010, Oracle. All rights reserved.
   
   Enter password: 
   
   Connected to:
   Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
   With the Partitioning, OLAP, Data Mining and Real Application Testing options
   
   SQL> ALTER SYSTEM SET LDAP_DIRECTORY_ACCESS = SSL;
   
   System altered.
   
   SQL>
   

   The possible values for LDAP_DIRECTORY_ACCESS are NONE, PASSWORD or SSL, and govern the authentication method between the Database and OUD server.

EUS and SSLv3 issues

Starting with JDK 7u75 release, the SSLv3 protocol (Secure Socket Layer) has been deactivated and is not available by default. If your OUD server is running with JDK 7u75 or higher, you may experience issues with EUS when trying to authenticate:

ORA-28030: Server encountered problems accessing LDAP directory service

And OUD access log will display the following error:

[08/Jan/2016:10:43:13 +0100] CONNECT conn=26 from=10.166.139.54:48916 to=10.166.139.54:1636 protocol=LDAPS
[08/Jan/2016:10:43:13 +0100] DISCONNECT conn=26 reason="I/O Error" msg="Client requested protocol SSLv3 not enabled or not supported"

The proper method to fix this issue is to apply patch 19285025 on the database, which will fix the LDAP library used to perform the connection between the database and OUD and use another algorithm.

A quick workaround is to edit the file $JRE_HOME/lib/security/java.security and remove “SSLv3” from the line defining jdk.tls.disabledAlgorithms (on the machine where OUD runs, for the java version used by OUD), then stop and restart OUD. This will allow OUD to use SSLv3. Note that this workaround should not be applied in production as SSLv3 is obsolete and should not be used anymore. The correct fix is to patch the database.

Enterprise Manager Cloud Control and eusm issues: AuthenticationException

Enterprise Manager Cloud Control is a web-based interface that allows to administer Enterprise User Security. When connecting to  OUD server, the interface may complain about an invalid password even though the credentials are correct.

The same problem happens with eusm 12c (the command-line tool delivered with Oracle Database):

$ eusm listDomains realm_dn=dc=example,dc=com ldap_host=$ldap_host ldap_port=1389 ldap_user_dn="cn=directory manager" ldap_user_password=****
javax.naming.AuthenticationException: [LDAP: error code 49 - Invalid Credentials]

If OUD access log shows the following error:

The server was not able to find any user entries for the provided username of cn=directory manager

then the fix is provided in Oracle Identity Management Suite Bundle Patch 11.1.2.2.8 (22085294), or Oracle Identity Management Suite Bundle Patch 11.1.2.3.3 (22085274)  depending on your OUD version.

The connection method between Enterprise Manager Cloud Control and OUD (or eusm 12c and OUD) is using SASL/DIGEST-MD5 authentication instead of a simple BIND. SASL/DIGEST-MD5 requires the password to be stored in a reversible password storage scheme, which means that the additional configuration steps are also needed:

1/ identify the password policy applying to the user cn=directory manager

$ OracleUnifiedDirectory/bin/ldapsearch -h $ldap_host -p 4444 -X -Z -D "cn=directory manager" -w password -b "cn=directory manager,cn=root dns,cn=config" -s base "(objectclass=*)" ds-pwp-password-policy-dn
dn: cn=Directory Manager,cn=Root DNs,cn=config
ds-pwp-password-policy-dn: cn=Root Password Policy,cn=Password Policies,cn=config

2/ modify the user’s password policy to use a reversible password storage scheme (AES for instance):

$ OracleUnifiedDirectory/bin/dsconfig set-password-policy-prop --policy-name "Root Password Policy" --set default-password-storage-scheme:AES --hostname $ldap_host --port 4444 -X -D "cn=directory manager" -j pwd.txt -n

3/ modify the user’s password to generate a new reversible password hash:

$ OracleUnifiedDirectory/bin/ldappasswordmodify -h $ldap_host -p 1389 -D "cn=directory manager" -w oldpwd  -c oldpwd -n newpwd
The LDAP password modify operation was successful

At this point, Enterprise Manager Cloud Control and eusm 12c will be able to authenticate to OUD and administer Enterprise User Security.