GC3: Grid Computing Competence Center

Since July 1st, 2014, GC3 merged into S3IT.
This web site is only kept for historical reasons and may be out-of-date.
Visit the S3IT website for more up-to-date information.

Blog index

GC3 graudates into S3IT
Posted early Tuesday morning, July 1st, 2014
How to create a module that also load a virtualenvironment
Posted mid-morning Friday, March 7th, 2014
Openstack workshop at GC3
Posted mid-morning Saturday, February 22nd, 2014
Moving LVM volumes used by a Cinder storage
Posted Friday evening, February 21st, 2014
How to configure swift glusterfs
Posted Monday night, February 10th, 2014
Fixing LDAP Authentication over TLS/SSL
Posted Monday night, January 6th, 2014
Linker command-line options for Intel MKL
Posted late Saturday evening, January 4th, 2014
A virtue of lazyness
Posted Saturday afternoon, December 21st, 2013
(Almost) readable CFEngine logs
Posted Thursday afternoon, December 19th, 2013
CFEngine error: ExpandAndMapIteratorsFromScalar called with invalid strlen
Posted at lunch time on Wednesday, December 11th, 2013
'Martian source' log messages and the default IP route
Posted Monday afternoon, November 25th, 2013
GC3 takes over maintenance of the Schroedinger cluster
Posted mid-morning Monday, November 4th, 2013
Grid Engine: how to find the set of nodes that ran a job (after it's finished)
Posted terribly early Wednesday morning, October 30th, 2013
Python2 vs Python3
Posted Friday afternoon, September 13th, 2013
GC3Pie 2.1.1 released
Posted at teatime on Friday, September 6th, 2013
Happy SysAdmin day!
Posted early Friday morning, July 26th, 2013
Object-oriented Python training
Posted at lunch time on Thursday, July 25th, 2013
Elasticluster 1.0.0 released
Posted late Thursday evening, July 18th, 2013
Short Autotools tutorial
Posted mid-morning Friday, July 5th, 2013
Patch Emacs' PostScript printing
Posted late Tuesday afternoon, June 11th, 2013
Slides of the Object-oriented Python course now available!
Posted late Tuesday afternoon, June 11th, 2013
Compile an Objective-C application on Ubuntu (Hobbes instance)
Posted late Friday afternoon, May 31st, 2013
Automated deployment of CFEngine keys
Posted Thursday night, May 30th, 2013
Posted late Tuesday afternoon, May 14th, 2013
Join us at the Compute Cloud Experience Workshop!
Posted early Monday morning, April 29th, 2013
GC3 Beamer theme released
Posted mid-morning Friday, April 5th, 2013
VM-MAD at the International Supercompting Conference 2013
Posted late Tuesday morning, March 26th, 2013
The GC3 is on GitHub
Posted late Monday morning, March 18th, 2013
How to enable search in IkiWiki
Posted Friday afternoon, March 15th, 2013
GC3Pie Training
Posted Thursday night, March 7th, 2013
Object-oriented Python training
Posted Thursday afternoon, March 7th, 2013
Advance Reservations in GridEngine
Posted mid-morning Thursday, March 7th, 2013
GridEngine accounting queries with PostgreSQL
Posted Wednesday night, March 6th, 2013
Floating IPs not available on Hobbes
Posted Tuesday afternoon, February 26th, 2013
Notes on SWIFT
Posted early Tuesday morning, February 12th, 2013
An online Python code quality analyzer
Posted late Saturday morning, February 9th, 2013
Seminar on cloud infrastructure
Posted Sunday night, February 3rd, 2013
GC3 announce its cloud infrastructure Hobbes
Posted at lunch time on Wednesday, January 30th, 2013
GC3Pie 2.0.2 released
Posted Monday afternoon, January 28th, 2013
Continuous Integration with Jenkins
Posted late Saturday morning, January 26th, 2013
On the importance of testing in a clean environment
Posted early Monday morning, January 21st, 2013
Weirdness with ImageMagick's `convert`
Posted Tuesday afternoon, January 15th, 2013
boto vs libcloud
Posted Tuesday afternoon, January 15th, 2013
Resolve timeout problem when starting many instances at once
Posted late Monday morning, January 7th, 2013
Proceedings of the EGI Community Forum 2012 published
Posted Monday afternoon, December 17th, 2012
SGE Workaround Installation
Posted at noon on Tuesday, December 4th, 2012
How to pass an argument of list type to a CFEngine3 bundle
Posted early Thursday morning, November 22nd, 2012
GC3 at the 'Clouds for Future Internet' workshop
Posted early Wednesday morning, November 21st, 2012
GC3 attends European Commission Cloud Expert Group
Posted early Monday morning, October 29th, 2012
SwiNG - SDCD2012 event
Posted mid-morning Monday, October 22nd, 2012
Large Scale Computing Infrastructures class starts tomorrow!
Posted Tuesday afternoon, September 25th, 2012
From bare metal to cloud at GC3
Posted early Monday morning, September 24th, 2012
GC3 at the EGI Technical Forum 2012
Posted late Thursday evening, September 20th, 2012
Training on GC3Pie and Python
Posted Friday evening, September 7th, 2012
GC3Pie used for research in Computational Quantum Chemistry
Posted Thursday afternoon, September 6th, 2012
``What's so great about MPI or Boost.MPI?''
Posted early Thursday morning, September 6th, 2012
blog/How to generate UML diagram with `pyreverse`
Posted early Thursday morning, August 23rd, 2012
Git's `rebase` command
Posted early Friday morning, June 15th, 2012
AppPot 0.27 released!
Posted mid-morning Thursday, June 14th, 2012
Urban computing - connecting to your server using `mosh`
Posted early Wednesday morning, June 6th, 2012
Whitespace cleanup with Emacs
Posted at lunch time on Tuesday, June 5th, 2012
Translate pages on this site
Posted late Thursday afternoon, May 31st, 2012
Scientific paper citing GC3Pie
Posted at teatime on Wednesday, May 30th, 2012
GC3 attends Nordugrid 2012 conference
Posted mid-morning Wednesday, May 30th, 2012
How the front page image was made
Posted Wednesday evening, May 16th, 2012
GC3 blog launched!
Posted Tuesday evening, May 15th, 2012
New GC3 Wiki now online!
Posted Tuesday afternoon, May 15th, 2012
AppPot paper on arXiv
Posted Tuesday afternoon, May 15th, 2012
GC3 at the EGI Technical Forum 2011
Posted Tuesday afternoon, May 15th, 2012

More on topic...

Fixing LDAP Authentication over TLS/SSL

We recently reconfigured our LDAP servers to only answer queries over a TLS/SSL secure connection on public IPs. Of course, this implies that we also had to reconfigure the clients to connect over TLS/SSL. Much to my surprise, it turned out that this was much more complex than just add a configuration line to turn on SSL and giving the path to the certificate store!


As we all know, NSS is an abstraction layer in (modern) UNIX C library that allows a sysadmin to select and aggregate account databases from different sources. The most-frequently used source is local text files (/etc/passwd, /etc/groups, etc.), but over time other modules have been developed for getting users from an LDAP server, for example.

The first such module is called NSS/LDAP, developed by PADL software (the original authors of RFC 2307 which introduced the LDAP schema for UNIX authentication). NSS/LDAP is relatively simple to configure; the code is mature and the format of the configuration file is very stable and is basically unchanged since about 10 years.

NSS/LDAP is what we've been running since we introduced LDAP authentication in our network, and is to-date the only LDAP-related NSS module included in the base CentOS/RHEL distribution.

The Problem

According to its manpage, NSS/LDAP allows binding to a server over TLS/SSL with a very simple configuration:

uri ldaps://server.name
ssl on
tls_cacertdir /etc/ssl/certs

where /etc/ssl/certs is the default X.509 root certificate store on Debian and Ubuntu hosts.

However, it turns out that the tls_cacertdir option is not honored due to an unfixed bug in Ubuntu1. A workaround is suggested in comment #13 to that bug report (use a file-based certificate store instead of a directory based one), but one should also note that the man page misspels the alternate option tls_cacertfile as tls_cacert...

Unfortunately, another bug then comes to bite you: set-UID programs may then start malfunctioning, again because of some issue in GnuTLS.

Fortunately, the comments in Debian bug #579647 suggest the way out: replace NSS/LDAP with NSS-PAM-LDAPD.


NSS-PAM-LDAPD is an alternate module for using LDAP as a database source for NSS, written by Arthur De Jong. There is a key architectural difference with NSS/LDAP: whereas NSS/LDAP implements a library module that connects to LDAP and performs a query for each invocation of a NSS function, NSS-PAM-LDAPD delegates this task to a single system-wide daemon named nslcd (Name Service LDAP Cache Daemon, I presume).

The nslcd daemon listens on a UNIX domain socket, and the library functions just connect to that socket and perform queries using a simpler protocol. The library part (which is loaded into every application using the libc, let us now forget!) is thus much lighter. The number of connections to the LDAP server is also limited as there is one single process per host which is actually running the queries; with NSS/LDAP, every process could be an LDAP client.2

Since the LDAP queries are effected by nslcd alone, only one configuration file is required for the operation of NSS-PAM-LDAPD: /etc/nslcd.conf. If you are familiar with the syntax of NSS/LDAP's ldap.conf, it looks like a simplified subset.

A notable difference in configuration and behavior is that, whereas NSS/LDAP has a rootbinddn option for configuring a DN to use for LDAP queries issued by the local privileged user "root", NSS-PAM-LDAPD has no such feature: every LDAP query is run using the same credentials. Therefore, you must configure the "root" DN as binddn in NSS-PAM-LDAPD; the daemon will expunge sensitive information (e.g., encrypted password) from data returned to non privileged users.

Finally, another advantage of NSS-PAM-LDAPD is that you can easily catch errors in the configuration file by running nslcd in "foreground/debug" mode with option -d::

# nslcd -d
nslcd: DEBUG: add_uri(ldaps://ldap.gc3.uzh.ch)
nslcd: DEBUG: ldap_set_option(LDAP_OPT_X_TLS_REQUIRE_CERT,2)
nslcd: DEBUG: ldap_set_option(LDAP_OPT_X_TLS_CACERTFILE,"/etc/ssl/certs/ca-certificates.crt")
nslcd: version 0.7.15 starting
nslcd: DEBUG: unlink() of /var/run/nslcd/socket failed (ignored): No such file or directory
nslcd: DEBUG: setgroups(0,NULL) done
nslcd: DEBUG: setgid(117) done
nslcd: DEBUG: setuid(111) done
nslcd: accepting connections
nslcd: [8b4567] DEBUG: connection from pid=5303 uid=0 gid=0
nslcd: [8b4567] DEBUG: nslcd_passwd_byname(rmurri)
nslcd: [8b4567] DEBUG: myldap_search(base="dc=gc3,dc=uzh,dc=ch", filter="(&(objectClass=posixAccount)(uid=rmurri))")
nslcd: [8b4567] DEBUG: ldap_initialize(ldaps://ldap.gc3.uzh.ch)
nslcd: [8b4567] DEBUG: ldap_set_rebind_proc()
nslcd: [8b4567] DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [8b4567] DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [8b4567] DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [8b4567] DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [8b4567] DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [8b4567] DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [8b4567] DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [8b4567] DEBUG: ldap_set_option(LDAP_OPT_X_TLS,LDAP_OPT_X_TLS_HARD)
nslcd: [8b4567] DEBUG: ldap_simple_bind_s("cn=root,dc=gc3,dc=uzh,dc=ch","***") (uri="ldaps://ldap.gc3.uzh.ch")
nslcd: [8b4567] DEBUG: ldap_result(): end of results

Further reading

The Debian Wiki has an excellent entry explaining the different options for LDAP authentication on GNU/Linux, and how to configure them: https://wiki.debian.org/LDAP/NSS

Another alternative is slapd-nssov: it uses the same architecture of NSS-PAM-LDAPD, but replaces the (rather simple) nslcd daemon with a local LDAP server slapd equipped with a plugin that allows it to talk the NSS-PAM-LDAPD local protocol over a UNIX domain socket. The local slapd acts as a proxy cache towards the "real" authenticating LDAP servers. You can read more about it on page 7 of http://symas.com/docs/Provisioning_AAA_in_Cloud_and_Enterprise_Environments


  1. Bug #242313, for the record. In short: Debian and Ubuntu's version of the NSS/LDAP module is compiled using GnuTLS as the underlying TLS/SSL library, but GnuTLS does not support directory-based certificate stores (like OpenSSL does instead). ↩

  2. Therefore, you must run nscd (Name Service Cache Daemon) on any system that performs LDAP auth: since the GNU libc tries to query nscd before running any NSS module, any LDAP lookup will be performed by nscd without overloading the LDAP server with senseless connections. ↩