GC3: Grid Computing Competence Center

Blog index

GC3 graudates into S3IT
Posted early Tuesday morning, July 1st, 2014
How to create a module that also load a virtualenvironment
Posted late Friday morning, March 7th, 2014
Openstack workshop at GC3
Posted at noon on Saturday, February 22nd, 2014
Moving LVM volumes used by a Cinder storage
Posted late Friday evening, February 21st, 2014
How to configure swift glusterfs
Posted Monday night, February 10th, 2014
Yet another timeout problem when starting many instances at once
Posted late Friday night, February 8th, 2014
Fixing LDAP Authentication over TLS/SSL
Posted Monday night, January 6th, 2014
Linker command-line options for Intel MKL
Posted Saturday night, January 4th, 2014
A virtue of lazyness
Posted Saturday afternoon, December 21st, 2013
(Almost) readable CFEngine logs
Posted Thursday afternoon, December 19th, 2013
CFEngine error: ExpandAndMapIteratorsFromScalar called with invalid strlen
Posted Wednesday afternoon, December 11th, 2013
'Martian source' log messages and the default IP route
Posted Monday afternoon, November 25th, 2013
GC3 takes over maintenance of the Schroedinger cluster
Posted at noon on Monday, November 4th, 2013
Grid Engine: how to find the set of nodes that ran a job (after it's finished)
Posted early Wednesday morning, October 30th, 2013
Python2 vs Python3
Posted at teatime on Friday, September 13th, 2013
GC3Pie 2.1.1 released
Posted Friday evening, September 6th, 2013
Happy SysAdmin day!
Posted mid-morning Friday, July 26th, 2013
Object-oriented Python training
Posted Thursday afternoon, July 25th, 2013
Elasticluster 1.0.0 released
Posted Thursday night, July 18th, 2013
Short Autotools tutorial
Posted at lunch time on Friday, July 5th, 2013
Patch Emacs' PostScript printing
Posted Tuesday evening, June 11th, 2013
Slides of the Object-oriented Python course now available!
Posted Tuesday evening, June 11th, 2013
Automated deployment of CFEngine keys
Posted at midnight, May 31st, 2013
Posted Tuesday evening, May 14th, 2013
Join us at the Compute Cloud Experience Workshop!
Posted early Monday morning, April 29th, 2013
GC3 Beamer theme released
Posted at lunch time on Friday, April 5th, 2013
VM-MAD at the International Supercompting Conference 2013
Posted at lunch time on Tuesday, March 26th, 2013
The GC3 is on GitHub
Posted at lunch time on Monday, March 18th, 2013
How to enable search in IkiWiki
Posted Friday afternoon, March 15th, 2013
GC3Pie Training
Posted Thursday night, March 7th, 2013
Object-oriented Python training
Posted Thursday afternoon, March 7th, 2013
Advance Reservations in GridEngine
Posted late Thursday morning, March 7th, 2013
GridEngine accounting queries with PostgreSQL
Posted Wednesday night, March 6th, 2013
Floating IPs not available on Hobbes
Posted at teatime on Tuesday, February 26th, 2013
Notes on SWIFT
Posted mid-morning Tuesday, February 12th, 2013
An online Python code quality analyzer
Posted at lunch time on Saturday, February 9th, 2013
Seminar on cloud infrastructure
Posted Sunday night, February 3rd, 2013
GC3 announce its cloud infrastructure Hobbes
Posted Wednesday afternoon, January 30th, 2013
GC3Pie 2.0.2 released
Posted Monday afternoon, January 28th, 2013
Continuous Integration with Jenkins
Posted at noon on Saturday, January 26th, 2013
On the importance of testing in a clean environment
Posted mid-morning Monday, January 21st, 2013
Weirdness with ImageMagick's `convert`
Posted at teatime on Tuesday, January 15th, 2013
boto vs libcloud
Posted Tuesday afternoon, January 15th, 2013
Resolve timeout problem when starting many instances at once
Posted at lunch time on Monday, January 7th, 2013
Proceedings of the EGI Community Forum 2012 published
Posted at teatime on Monday, December 17th, 2012
SGE Workaround Installation
Posted at lunch time on Tuesday, December 4th, 2012
How to pass an argument of list type to a CFEngine3 bundle
Posted mid-morning Thursday, November 22nd, 2012
GC3 at the 'Clouds for Future Internet' workshop
Posted mid-morning Wednesday, November 21st, 2012
GC3 attends European Commission Cloud Expert Group
Posted mid-morning Monday, October 29th, 2012
SwiNG - SDCD2012 event
Posted at lunch time on Monday, October 22nd, 2012
Large Scale Computing Infrastructures class starts tomorrow!
Posted late Tuesday afternoon, September 25th, 2012
From bare metal to cloud at GC3
Posted mid-morning Monday, September 24th, 2012
GC3 at the EGI Technical Forum 2012
Posted Thursday night, September 20th, 2012
Training on GC3Pie and Python
Posted late Friday evening, September 7th, 2012
GC3Pie used for research in Computational Quantum Chemistry
Posted late Thursday afternoon, September 6th, 2012
``What's so great about MPI or Boost.MPI?''
Posted mid-morning Thursday, September 6th, 2012
blog/How to generate UML diagram with `pyreverse`
Posted late Thursday morning, August 23rd, 2012
Git's `rebase` command
Posted mid-morning Friday, June 15th, 2012
AppPot 0.27 released!
Posted at noon on Thursday, June 14th, 2012
Urban computing - connecting to your server using `mosh`
Posted mid-morning Wednesday, June 6th, 2012
Whitespace cleanup with Emacs
Posted Tuesday afternoon, June 5th, 2012
Translate pages on this site
Posted Thursday evening, May 31st, 2012
Scientific paper citing GC3Pie
Posted Wednesday evening, May 30th, 2012
GC3 attends Nordugrid 2012 conference
Posted at lunch time on Wednesday, May 30th, 2012
How the front page image was made
Posted late Wednesday evening, May 16th, 2012
GC3 blog launched!
Posted late Tuesday evening, May 15th, 2012
New GC3 Wiki now online!
Posted Tuesday evening, May 15th, 2012
AppPot paper on arXiv
Posted Tuesday evening, May 15th, 2012
GC3 at the EGI Technical Forum 2011
Posted Tuesday evening, May 15th, 2012

More on topic...

Automated deployment of CFEngine keys

A problem statement

CFEngine uses public-key cryptography to authenticate communication between CFEngine agents and the CFEngine server. As in all such systems, there is the problem of making the initial public key exchange between client and server.

Suggestions from the CFEngine manual

The relevant section of the CFEngine manual does not, unfortunately suggest any good solution: manual key exchange is tagged as "somewhat laborious" and "a silly way to transfer keys between nearby hosts that you control yourself", but the only alternatives proposed are:

  • accepting new keys by running cf-runagent in interactive mode, thus requiring a human operator to trust the new keys (and it should be noted that cf-runagent does not print any kind of key fingerprint --unlike OpenSSH does--, so this method is vulnerable to IP impersonation);

  • or setting the trustkeysfrom (on the server) and trustkeys (on the agent) options, so to make CFEngine accept and save any new/unknown public key. The manual goes on to note that this setting is insecure and the options should be disabled once the initial key exchange has been performed.

It should be clear that this solution cannot be satisfactory in a dynamic environment where new hosts are being added or reinstalled on a frequent basis: manual screening of keys cannot be performed when the number of keys to be trusted is high, and it rules out the possibility of automated operation by scripts or APIs, which is a key requirement in these days of elastic infrastructures; on the other hand, setting CFEngine to trust any new key is equivalent to turning peer authentication off.

However, there's more to a computer than just CFEngine, so we might be able to bootstrap the initial key exchange by setting up a secure channel by other means.

A fully-automated solution

Our solution is to integrate the CFEngine key exchange in our PXE boot/installation system. In short:

  1. when we (re)install a host, a preparation command is run on the PXE boot server; in addition to setting up the PXE boot files for the new host, the command also creates a pair of CFEngine keys and an authentication token (a very long random string) for the new host and stores them in a directory /srv/pxeconf/priv/HOSTNAME/.

  2. the PXE boot procedure installs the base OS and runs a post-installation script on the new host; the authentication token is passed as part of the kernel command line during the PXE boot.

  3. the post-installation script connects to the PXE boot server over HTTPS to download the SSH public key of the PXE boot server. The PXE boot server uses an SSL server certificate signed by a well-known CA so that the authenticity of the certificate can be verified using only the root CA files distributed in the base OS install. A CGI script on the PXE boot server ensures that the download request is only honored if the HTTP client can supply the exact authentication token stored alongside the generated keys.

  4. The server SSH public key gotten in step 3. is trusted by inserting it in root's authorized_keys file.

  5. Now the PXE boot server can connect via SSH to the client. It uses scp to copy the CFEngine keys (public+private key of the host just installed, and the public key of the CFEngine server) into the correct location in the new hosts' file system.

  6. Now CFEngine's public-key exchange has been bootstrapped, and secure connection between the server and the agent is assured. The rest of the post-installation is then performed via CFEngine.

It should be noted that this procedure is still somewhat vulnerable in step 3.: an attacker could impersonate the IP of the new host, but it still has to use the right authentication token (which is passed to the client as part of the PXE boot process, hence travels unencrypted over the network).

It should also be noted that, when running VMs on a cloud infrastructure, steps 1.--4. above are automatically executed by the cloud stack, and there is no possibility for impersonation. In this case the procedure is fully secure.

How to make CFEngine trust keys

The only details missing in the above schema are the exact steps needed to make CFEngine trust the keys.

CFEngine keys can only be generated using the cf-key command; the -f option allows choosing an alternate path where to store the new key (default for user root is /var/cfengine/ppkeys/localhost.pub). So with cf-key -f the keys are created on the PXE boot server.

Now, the new host's agent has to trust the CFEngine server's public key localhost.pub, and conversely the server has to trust the new agent's public key.

In older versions of CFEngine (up to version 3.2.x), it was enough to place keys in /var/cfengine/ppkeys under the name root-A.B.C.D.pub where A.B.C.D is the IPv4 address of the peer (server or agent).

In more recent versions, the filename is hash-based (root-MD5=....) and one cannot easily compute the required hash with a few shell commands.

Solution: patch cf-key to print the hash of a public key: running cf-key -p file.pub prints the hash string for the public key. While we're at it, we can just have cf-key copy the key to the right location, i.e., cf-key --trust file.pub copies the file.pub public key into the correct location of the filesystem for the server to trust it.

The patch was submitted as pull request to CFEngine developers, which was accepted for inclusion in release 3.5.0. However, the patch applies cleanly yo version 3.4.x as well: you can download the patch for CFEngine 3.4.x, or pull it from the 3.4.x branch on the GC3 CFEngine repo on GitHub.