Using the Utilities to Prepare the Directory

So far in this chapter we have looked at the server operations, and created an LDIF file representing our initial directory information tree. In the remainder of this chapter we are going to look at two groups of tools. In this part we are going to look at the OpenLDAP utilities. In the next part we will look at the OpenLDAP clients.

Unlike the OpenLDAP clients, the utilities do not use the LDAP protocol to connect to a server and perform directory operations. Instead they work on a lower level, interacting directly with OpenLDAP directories and data files. The OpenLDAP suite includes eight utilities that perform administrative tasks. We will look at these tools as we go through the process of creating, loading, and verifying directory data.

The aim of this section is to explain the basic use of these utilities. Each utility has a handful of command-line flags that can be used to further modify the behavior of the utility. We will see some of the more useful flags, but if you want detailed information, you should consult the excellent OpenLDAP man pages.

In recent versions of OpenLDAP the utilities do not actually exist as stand-alone programs. Instead, they are all compiled into the slapd program, and symbolic links are created to point from the utility name to the slapd program. Using the ls command, we can look at the utilities to see how this is done:

  $ ls -og /usr/local/sbin

This is what we get:

total 0
lrwxrwxrwx 1 16 2006-08-17 11:37 slapacl -> ../libexec/slapd
lrwxrwxrwx 1 16 2006-08-17 11:37 slapadd -> ../libexec/slapd
lrwxrwxrwx 1 16 2006-08-17 11:37 slapauth -> ../libexec/slapd
lrwxrwxrwx 1 16 2006-08-17 11:37 slapcat -> ../libexec/slapd
lrwxrwxrwx 1 16 2006-08-17 11:37 slapdn -> ../libexec/slapd
lrwxrwxrwx 1 16 2006-08-17 11:37 slapindex -> ../libexec/slapd
lrwxrwxrwx 1 16 2006-08-17 11:37 slappasswd -> ../libexec/slapd
lrwxrwxrwx 1 16 2006-08-17 11:37 slaptest -> ../libexec/slapd

All eight of the utilities are just symbolic links to the slapd program. When slapd gets executed, it checks to see what program name was used when it was executed, and then it acts like that program. For example, when slapd is called as slapadd, it acts as a program for loading data into the directory. If it is called as slaptest, it acts as a program for verifying the format of and directives in the configuration file.

As we proceed through the description of the utilities we will cover them as if they were separate programs because that is how they are treated.

Since we created an LDIF file in the last part, we will begin this section by looking at the tool that loads the LDIF file into the directory backend.

The slapadd program is used to load directory data, formated as LDIF files, directly into OpenLDAP. It is executed from within an operating system shell (for example a command prompt or shell script).

The slapadd program does not use the LDAP protocol to connect to a running server. Instead, it works directly with the OpenLDAP backend. For that reason, when you run slapadd you must first shut down the directory server. Otherwise, you may end up with conflicts between the slapd server process and the slapadd process as they both try to exclusively manage the same databases.

In the previous part of this chapter we created an LDIF file containing a handful of records for our directory tree. Now we will load this LDIF file into our directory. This will take four steps:

We covered the process of starting and stopping the server at the end of Chapter 2. To summarize, though, we can stop a version installed from the Ubuntu package using the invoke-rc.d command:

With the version compiled from source (see Appendix A), this can be done by finding the slapd process ID and killing the process (or using the killall program):

  $ sudo kill `pgrep slapd`

Next, we need to make sure that the LDIF file we created in the last part is correctly formatted.

Running in test mode before doing the actual load can greatly reduce the amount of time it takes to load a new LDIF file because it will help you catch LDIF errors before things get written to the directory. Normally slapadd adds records one at a time as it reads them. So if there are three records in a file, the first record will be added to the directory before the second or third records are read. If there is an error in a record later in the file, then the directory will be partially loaded, and you will either have to creatively alter the LDIF file or destroy the database and start again.

Using test mode, we can make sure that the LDIF file does not have any errors before we start loading records into the directory. This should just eliminate cases where an LDIF file is only partially imported because of bad records.

We can use the slapadd program to do this before we try to load the data into the directory:

This command uses five flags:

In cases where there is an error in the LDIF file, slapadd will print out some helpful information. For example, if we try to load an obviously broken file that looks like this:

In this file the broken lines are highlighted. When we run slapadd, we will get an error:

Here, slapadd tested our first record, dc=example,dc=com, without problems, but then encountered a line that did not begin with a DN (on line 11). It skipped that record. On line 18 it encountered another error: the ferble attribute is not defined by any of the object classes in the record.

When run successfully against the LDIF file we created earlier in this chapter, the output looks like this:

No errors. We are ready to proceed to the third step: importing the records into the directory.

To do the actual import of the records into the directory, we use the slapadd command with a subset of the flags used in the previous section. We omit the -u flag (for testing) and the -c flag (so that it doesn't continue if it encounters a bad record).

Now, the command looks like this:

And, this is what we get as output:

Note that the output is just slightly different this time; at the end of each line, there is an ID number enclosed in parentheses. This ID number makes up part of the record's entryCSN attribute, which is used internally to monitor the record.

We have just populated our directory with the eight records we created earlier in the chapter. We are now ready to start the directory.

It sometimes happens that midway through a slapadd, the program encounters an error—either in the LDIF file itself, or from some external consideration—and aborts the directory import part way through. In these cases you may need to start over. But merely re-running the slapadd operation will give errors like this (the error may vary depending on the backend you are using):

$ sudo slapadd -v -f /usr/local/etc/openldap/slapd.conf -l 
    basics.ldif
=> hdb_tool_entry_put: id2entry_add failed: DB_KEYEXIST: Key/data 
    pair already exists (-30996)
=> hdb_tool_entry_put: txn_aborted! DB_KEYEXIST: Key/data pair 
    already exists (-30996)
slapadd: could not add entry dn="dc=example,dc=com" (line=9): 
    txn_aborted! DB_KEYEXIST: Key/data pair already exists (-30996)

What is going on here?

What has happened is that some of the entries from the basics.ldif file have already been imported into the directory, but perhaps not all of them. There are various ways to attempt to work around this. You can try to prune the LDIF file down to just the records that haven't been added already. You can try to run the slapadd program in continuation mode (with the -c flag) and hope that all of the remaining records are added correctly.

But you may find that the best way of dealing with these cases is to simply destroy and rebuild the directory. While this sounds like a rather extreme measure, it has one distinct advantage over other methods: it avoids the problem of inconsistent records that can be caused with failed slapadd commands. Thus, it is often the best way of recovering from failed directory imports.

In most of the OpenLDAP backends that can be loaded with slapadd, the backend stores data somewhere on the file system or in a relational database. After a failed slapadd you may find that the best way to recover is to destroy all of the data in the underlying backend, and then start over.

Currently, we are using the hdb backend (see Chapter 2). The method used here will apply equally well to other BerkeleyDB backends (bdb and ldbm in bdb mode), and can be easily adapted to cover the (deprecated) ldbm with gdbm backend.

For other sorts of backends, such as those that use relational databases like PostgreSQL, or custom backends like back-perl, you will need to examine the documentation on those backends to determine the best way of clearing the records from the directory.

For the hdb and bdb backends, the directory data files are stored on the file system. In Ubuntu, these are located at /var/lib/ldap. If you followed the directions in Appendix A, the database files are located at /usr/local/var/openldap-data/.

Here's what the contents of the /var/lib/ldap directory look like:

alock    __db.002    __db.005    dn2id.bdb    objectClass.bdb
cn.bdb    __db.003    DB_CONFIG    id2entry.bdb
__db.001    __db.004    DB_CONFIG.example    log.0000000001

Here you can see all of the directory database files (which start with __db.), the directory index files (which end with .bdb), and the BerkeleyDB transaction logs (which begin with log.). There are a few other files in this directory, such as alock and DB_CONFIG, that we don't need to delete. To delete the files, we use rm with a list of expressions that match only the files we want to delete:

  $ sudo rm __db.* *.bdb log.*

This removes just the files we don't want. Now the directory should contain only a couple of files:

alock    DB_CONFIG    DB_CONFIG.example

That's all it takes to destroy the database. Now we can re-create the directory by loading the (corrected, if necessary) LDIF file with the slapadd command:

  $ sudo slapadd -v -l basics.ldif

And this message is returned:

added: "dc=example,dc=com" (00000001)
added: "ou=Users,dc=example,dc=com" (00000002)
added: "ou=Groups,dc=example,dc=com" (00000003)
added: "ou=System,dc=example,dc=com" (00000004)
added: "uid=matt,ou=Users,dc=example,dc=com" (00000005)
added: "uid=barbara,ou=Users,dc=example,dc=com" (00000006)
added: "cn=LDAP Admins,ou=Groups,dc=example,dc=com" (00000007)
added: "uid=authenticate,ou=System,dc=example,dc=com" (00000008)

That is all there is to destroying and recreating a directory.

The next utility that we will examine is slapindex. This utility manages the index files for OpenLDAP backends that use indexes (such as hdb, bdb, and the deprecated ldbm).

OpenLDAP maintains a set of index files to expedite searching for records. These are stored outside of the main directory database, and as records are added, modified, and removed from the directory, the slapd server modifies the index files accordingly.

But in certain circumstances, the slapd server may not have sufficient information to know about changes it needs to make to the index files and, in those cases, the indexes will need to be rebuilt manually.

There are three common cases that require use of the slapindex command:

In these three cases, slapindex should be run:

This will rebuild all of the indexes for the first database defined in slapd.conf (we only have one database defined).

The -q flag instructs slapindex to perform some additional checking operations, which will greatly expedite the process of re-indexing. Skipping such checks is generally safe with the slapindex utility, though it should only be done with great care when using slapadd.

The -f flag, which takes the path to a configuration file, specifies the slapd configuration file. If this flag is omitted (as we have done), slapindex will look in the default location for the slapd.conf file.

If you want to monitor the progress of slapindex, you can use the -v flag to turn on verbose output.

The slapcat program dumps the entire contents of a directory into an LDIF file. It is a convenient tool for creating a backup of the directory, and can also be useful for examining the data is in the directory.

Of course, there is a similar client application, ldapsearch, which can also dump the entire contents of the directory. How do you know when to use each? Since ldapsearch uses the LDAP protocol to contact the server, bind, and then run LDAP search operations, it incurs more overhead. slapcat, on the other hand, works directly with the backend. ldapsearch is limited by time and size limits, set both in the client configuration file, ldap.conf, and in the server's configuration in slapd.conf (see Chapter 2). The ldapsearch command is also limited by ACLs, while no ACLs are applied to slapcat.

Clearly then, for operations such as backing up the directory, slapcat ought to be used rather than ldapsearch.

As of version 2.3 of OpenLDAP, if you are using the hdb or bdb backends, you can safely run slapcat while slapd is running; there is no need to shutdown the directory server in order to make a backup copy.

When we covered slapadd earlier in this chapter, we used that utility to load records in basics.ldif into the directory. Now we can use slapcat to view those records.

The -l flag, which takes a path for an argument, indicates what file the output should be written to. In this case it is writing to the file basics-out.ldif. If -l is omitted, then the LDIF data will be sent to standard output, which will usually be printed straight to your screen.

As with the other utilities, the -f flag can be used to specify the path to the SLAPD configuration file. The -a flag, which takes an LDAP filter, can be used to specify a pattern that records must match before they are dumped to output. You can use this flag to dump just a subtree. For example, we could dump only records in the Users OU with this command:

This would return complete records for only the following three DNs:

Let's take a closer look at the output for just the record of the base DN:

$ sudo slapcat -a "(dc=example)"
dn: dc=example,dc=com
description: Example.Com, your trusted non-existent corporation.
dc: example
o: Example.Com
objectClass: top
objectClass: dcObject
objectClass: organization
structuralObjectClass: organization
entryUUID: b1a00a7c-c587-102a-9eb2-412127118751
creatorsName: cn=Manager,dc=example,dc=com
modifiersName: cn=Manager,dc=example,dc=com
createTimestamp: 20060821173908Z
modifyTimestamp: 20060821173908Z
entryCSN: 20060821173908Z#000000#00#000000

The highlighted attributes should look unfamiliar, as they did not exist in the original LDIF file that we created. These are internal operational attributes that OpenLDAP automatically maintains.

Different operational attributes play different roles in OpenLDAP, and these attributes may be useful for directory managers and LDAP-aware applications.

For example, the creatorsName, modifiersName, createTimestamp, and modifyTimestamp fields often come in useful. OpenLDAP automatically retains the following record-level information:

The entryUUID attribute provides a Universally Unique Identifier (UUID) for a record, which serves as an identifier that is more stable than DN (which can change), and is supposed to be, according to the specification in RFC 4122 (http://rfc-editor.org/rfc/rfc4122.txt), "an identifier unique across both space and time, with respect to the space of all UUIDs." See the entryUUID RFC at http://rfc-editor.org/rfc/rfc4530.txt.

The entryCSN (Change Sequence Number) attribute is used by the SyncRepl replication provider to determine what records need to be synchronized between LDAP servers. We will see this in more detail in Chapter 7.

Finally, the attribute structuralObjectClass is added. This attribute specifies which of the object classes is to be treated as the structural object class. Recall that when we created our records for Matt and Barbara, each record had three object classes: person, organizationalPerson, and inetOrgPerson. All three are structural object classes, and all three are related (inetOrgPerson is a child of organizationalPerson, which in turn is a child of person). But each record can have only one structural object class. As I noted above, the one farthest down the tree becomes the structural object class, and the others are treated, essentially, as abstract object classes. We can see this if we use slapcat to dump Barbara's record:

$ sudo slapcat -a '(uid=barbara)'
dn: uid=barbara,ou=Users,dc=example,dc=com
ou: Users
uid: barbara
sn: Jensen
cn: Barbara Jensen
givenName: Barbara
displayName: Barbara Jensen
mail: barbara@example.com
userPassword:: e1BMQUlOfXNlY3JldA==
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
structuralObjectClass: inetOrgPerson
entryUUID: b1ae9916-c587-102a-9eb7-412127118751
creatorsName: cn=Manager,dc=example,dc=com
modifiersName: cn=Manager,dc=example,dc=com
createTimestamp: 20060821173908Z
modifyTimestamp: 20060821173908Z
entryCSN: 20060821173908Z#000005#00#000000

Note that the structuralObjectClass attribute has the value inetOrgPerson.

At this point we've examined the slapcat tool, as well as the slapindex and slapadd tools. These three are the most often used utilities. But there are a few others that can come in handy in certain circumstances. So next, we will look at slapacl.

Writing ACLs can be frustrating and difficult to test. In order to ease the process of testing the efficacy of ACLs in the slapd.conf file, the OpenLDAP suite includes a tool for testing ACLs directly. We will make greater use of this tool when we test ACLs in Chapter 4, but we will see an introduction to the utility here.

In Chapter 2, we added the following ACL to slapd.conf:

access to attrs=userPassword
       by anonymous auth
       by self write
       by * none

This ACL specifies that for any given record in the directory, if it has userPassword, the following rules should be applied to requests for access to that attribute:

  • The anonymou s user should be able to authenticate using userPassword.
  • It should allow a DN the permissions to modify (and read) its own password.
  • It should deny all other DNs all access to this record's userPassword.

That means that uid=matt,ou=Users,dc=example,dc=com should not be able to write a new userPassword value for uid=barbara,ou=Users,dc=example,dc=com. We can use the slapacl utility to test this:

  $ sudo slapacl -v -D "uid=matt,ou=Users,dc=example,dc=com" -b 
       "uid=barbara,ou=Users,dc=example,dc=com" "userPassword/write"

This command might look daunting at first, but it is actually very simple. Let's look at the arguments in sequence:

  • The -v flag tuns on verbose output.
  • The -D flag is used to tell slapacl which DN is trying to access the directory. In this case, we said: -D "uid=matt,ou=Users,dc=example,dc=com". That is, slapacl is testing to see if the DN for Matt can get access.
  • The -b flag indicates which record we want the given DN to try to access. In this case it is Barbara's DN, since we want to test if Matt can write Barbara's password: -b "uid=barbara,ou=Users,dc=example,dc=com".
  • Finally, the last argument specifies what attribute we want to access, and what sort of privilege we are requesting. In this case, we want the userPassword attribute, and we want to see if Matt has write access to it ("userPassword/write").

So, in the end, we are testing to see if Matt's DN can write a new userPassword for Barbara's record. Here is the result of the slapacl command:

authcDN: "uid=matt,ou=users,dc=example,dc=com"
write access to userPassword: DENIED

That's the result we would expect. Because of this ACL, Matt cannot write to Barbara's userPassword attribute.

The slapauth tool is used to test SASL authentication to the directory. When an application attempts to bind using SASL, instead of specifying a complete DN (like uid=matt,ou=Users,dc=example,dc=com), the application passes in a user ID (u: matt) along with a few other bits of information, such as a realm identifier and an authentication mechanism.

We will cover SASL authentication in Chapter 4. If you do not already have experience with SASL you may want to read on, and come back to this section after reading Chapter 4.

OpenLDAP can then take that information and use a regular expression to guess what DN that user belongs to. But it can be difficult to figure out what the regular expressions will look like. The slapauth tool is useful in testing what one particular SASL request will look like when OpenLDAP receives it.

For example, we could add the following SASL configuration directives to our slapd.conf file:

authz-policy from
authz-regexp 
  "^uid=([^,]+).*,cn=auth$" 
  "uid=$1,ou=Users,dc=example,dc=com"

The regular expression in authz-regexp should convert from a SASL authzID format to an LDAP DN:

$ sudo slapauth -U "matt" -X "u: matt"
ID: <matt>
authcDN: <uid=matt,ou=users,dc=example,dc=com>
authzDN: <uid=matt,ou=users,dc=example,dc=com>
authorization OK

The first parameter, -U matt, sends a test request with the SASL authcID of matt. The -X "u: matt" parameter sends a test request with the authzID u: matt. These should then output a correctly formatted DN, according the the regular expression in authz-regexp.

We will use slapauth more in Chapter 4 when we set up SASL authentication.

The slapdn tool is used to test whether a given DN is valid for this directory server. Specifically, it tests a DN against the defined schemas to make sure that the DN is valid.

Here are a few examples of slapdn in action:

$ sudo slapdn 'cn=Foo,dc=example,dc=com'
DN: <cn=Foo,dc=example,dc=com> check succeeded
normalized: <cn=foo,dc=example,dc=com>
pretty: <cn=Foo,dc=example,dc=com>

$ sudo slapdn 'ou=New Unit,dc=example,dc=com'
DN: <ou=New Unit,dc=example,dc=com> check succeeded
normalized: <ou=new unit,dc=example,dc=com>
pretty: <ou=New Unit,dc=example,dc=com>

In these two examples, the DNs checked out. slapdn tested the DNs, and then printed out the normalized version (all lowercase, extra spaces removed) and the pretty (originally formated) version.

Here's an example of a failure:

$ sudo slapdn 'fakeAttr=test,dc=example,dc=com'
DN: <fakeAttr=test,dc=example,dc=com> check failed 21 
    (Invalid syntax)

In this case no schema was found that had the attribute fakeAttr. Here's another failed case:

$ sudo slapdn 'documentSeries=Series 18,dc=example,dc=com'
DN: <documentSeries=Series 18,dc=example,dc=com> check failed 21 
    (Invalid syntax)

While documentSeries is defined in a schema it is an object class, not an attribute, and object class names cannot be used in constructing DNs.

The usefulness of the slapdn program is limited to only rare cases where you need to test a DN against a directory without being able to look at the slapd.conf file to find out what schemas are loaded (or, alternately, search the schemas using the ldapsearch program).

The slappasswd utility is a tool for encrypting passwords according to schemes supported by OpenLDAP, such as the one described in RFC 2307 (http://rfc-editor.org/rfc/rfc2307.txt).

When we created our basic LDIF file, we used the userPassword attribute for storing passwords. For example, our authentication account record looked like this:

# Special Account for Authentication:
dn: uid=authenticate,ou=System,dc=example,dc=com
uid: authenticate
ou: System
description: Special account for authenticating users
userPassword: secret
objectClass: account
objectClass: simpleSecurityObject

The userPassword field has the password in plain text. When the value is loaded into the directory userPassword is encoded with base-64, and looks like this:

userPassword:: c2VjcmV0

But this is not encrypted—just encoded in an easily reversible way. While it might prevent the directory administrator from accidentally seeing the user's password, base-64 encoding will do nothing to prevent an attacker from figuring out the password.

But OpenLDAP does not require you to store passwords in unencrypted text. In fact, it is best if you do not. OpenLDAP supports a number of one-way hashing algorithms that can be used to store the passwords in a way in which they cannot be decrypted.

The slappasswd program provides the tools to create a hashed value of a password. That hashed value can then be used in the userPassword field of an LDIF file.

OpenLDAP supports five different password hashing schemes: Crypt (CRYPT), Message Digest 5 (MD5), salted MD5 (SMD5), Secure Hashing Algorithm, the SHA-1 version (SHA), and Salted SHA (SSHA). By default, OpenLDAP uses the most secure of the available hashing algorithms: SSHA.

Passwords are stored in the userPassword field in a format according to section 5.3 of RFC 2307 (http://rfc-editor.org/rfc/rfc2307.txt). An encrypted password looks like this:

At the beginning of the password, the section in curly braces ({}) indicates which of the five password schemes was used. In this case it is the default SSHA algorithm. The remainder of the field is the digested hash of the password.

While the hashed password cannot be decrypted, when a user tries to bind to the server, OpenLDAP takes the password the user supplies and encrypts it using the same algorithm as the value (and same salt) of the value of userPassword. If the two hashed passwords match, then OpenLDAP logs the user on. If the two do not match, OpenLDAP responds with an error message indicating that authentication failed.

Armed with this basic understanding of how passwords are used and stored, we can now look at the slappasswd program. This program can be used to encrypt a password and format it for insertion into an LDIF file. The command can be called with no arguments:

$ slappasswd
New password: 
Re-enter new password: 
{SSHA}71xEB2E59cuoPEQLErY44bYMHwCCgbtR

In this case, since no parameters were specified on the command line, slappasswd prompts for a password, and then prompts for verification of the password. Then, it prints out the encrypted value of the password. We can use this value in an LDIF record:

dn: uid=nicholas,ou=Users,dc=example,dc=com
cn: Nicholas Malebranche
sn: Malebranche
uid: nicholas
ou: Users
userPassword: {SSHA}71xEB2E59cuoPEQLErY44bYMHwCCgbtR
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson

In some cases, typing and retyping passwords may be too tedious, and a faster method of encrypting a number of passwords is preferred. You can either use the -T flag to point to a file containing a list of clear-text passwords to be hashed, or you can specify the password on the command line with the -s flag:

In this shell command, each of the three clear-text passwords, foo, bar, and baz, are encrypted by slappasswd.

By using the -h flag, you can specify which hashing algorithm slappasswd should use:

In the above commands, the same password, test, is encrypted using three different hashing schemes.

Next we will turn to the last OpenLDAP utility—slaptest.

The slaptest utility is used for checking the format and directives used in the slapd.conf file (and any files included by slapd.conf).

Running slaptest is simple:

  $ slaptest -v -f /etc/ldap/slapd.conf

The -v flag turns on verbose output, and the -f flag, which takes one argument, specifies which configuration file to check. If -f is omitted, then the default slapd.conf file (usually /etc/ldap/slapd.conf) is checked.

If the configuration file is correctly formatted and the directives are all valid and operational, then slaptest will print out a basic success message:

If anything goes wrong however, slaptest will print out diagnostic information. For example, if I add an include directive to slapd.conf that points to a file that does not exist, slaptest will print an error:

This output should be helpful for tracking down the problem in the configuration files. In this case it was caused by a line that looks like this:

This is the last of the OpenLDAP utilities. Now we will turn to the client applications that are included with the OpenLDAP suite.