GC3: Grid Computing Competence Center

Blog index

GC3 graudates into S3IT
Posted early Tuesday morning, July 1st, 2014
How to create a module that also load a virtualenvironment
Posted late Friday morning, March 7th, 2014
Openstack workshop at GC3
Posted at noon on Saturday, February 22nd, 2014
Moving LVM volumes used by a Cinder storage
Posted late Friday evening, February 21st, 2014
How to configure swift glusterfs
Posted Monday night, February 10th, 2014
Yet another timeout problem when starting many instances at once
Posted late Friday night, February 8th, 2014
Fixing LDAP Authentication over TLS/SSL
Posted Monday night, January 6th, 2014
Linker command-line options for Intel MKL
Posted Saturday night, January 4th, 2014
A virtue of lazyness
Posted Saturday afternoon, December 21st, 2013
(Almost) readable CFEngine logs
Posted Thursday afternoon, December 19th, 2013
CFEngine error: ExpandAndMapIteratorsFromScalar called with invalid strlen
Posted Wednesday afternoon, December 11th, 2013
'Martian source' log messages and the default IP route
Posted Monday afternoon, November 25th, 2013
GC3 takes over maintenance of the Schroedinger cluster
Posted at noon on Monday, November 4th, 2013
Grid Engine: how to find the set of nodes that ran a job (after it's finished)
Posted early Wednesday morning, October 30th, 2013
Python2 vs Python3
Posted at teatime on Friday, September 13th, 2013
GC3Pie 2.1.1 released
Posted Friday evening, September 6th, 2013
Happy SysAdmin day!
Posted mid-morning Friday, July 26th, 2013
Object-oriented Python training
Posted Thursday afternoon, July 25th, 2013
Elasticluster 1.0.0 released
Posted Thursday night, July 18th, 2013
Short Autotools tutorial
Posted at lunch time on Friday, July 5th, 2013
Patch Emacs' PostScript printing
Posted Tuesday evening, June 11th, 2013
Slides of the Object-oriented Python course now available!
Posted Tuesday evening, June 11th, 2013
Automated deployment of CFEngine keys
Posted at midnight, May 31st, 2013
blog/Resize_an_image
Posted Tuesday evening, May 14th, 2013
Join us at the Compute Cloud Experience Workshop!
Posted early Monday morning, April 29th, 2013
GC3 Beamer theme released
Posted at lunch time on Friday, April 5th, 2013
VM-MAD at the International Supercompting Conference 2013
Posted at lunch time on Tuesday, March 26th, 2013
The GC3 is on GitHub
Posted at lunch time on Monday, March 18th, 2013
How to enable search in IkiWiki
Posted Friday afternoon, March 15th, 2013
GC3Pie Training
Posted Thursday night, March 7th, 2013
Object-oriented Python training
Posted Thursday afternoon, March 7th, 2013
Advance Reservations in GridEngine
Posted late Thursday morning, March 7th, 2013
GridEngine accounting queries with PostgreSQL
Posted Wednesday night, March 6th, 2013
Floating IPs not available on Hobbes
Posted at teatime on Tuesday, February 26th, 2013
Notes on SWIFT
Posted mid-morning Tuesday, February 12th, 2013
An online Python code quality analyzer
Posted at lunch time on Saturday, February 9th, 2013
Seminar on cloud infrastructure
Posted Sunday night, February 3rd, 2013
GC3 announce its cloud infrastructure Hobbes
Posted Wednesday afternoon, January 30th, 2013
GC3Pie 2.0.2 released
Posted Monday afternoon, January 28th, 2013
Continuous Integration with Jenkins
Posted at noon on Saturday, January 26th, 2013
On the importance of testing in a clean environment
Posted mid-morning Monday, January 21st, 2013
Weirdness with ImageMagick's `convert`
Posted at teatime on Tuesday, January 15th, 2013
boto vs libcloud
Posted Tuesday afternoon, January 15th, 2013
Resolve timeout problem when starting many instances at once
Posted at lunch time on Monday, January 7th, 2013
Proceedings of the EGI Community Forum 2012 published
Posted at teatime on Monday, December 17th, 2012
SGE Workaround Installation
Posted at lunch time on Tuesday, December 4th, 2012
How to pass an argument of list type to a CFEngine3 bundle
Posted mid-morning Thursday, November 22nd, 2012
GC3 at the 'Clouds for Future Internet' workshop
Posted mid-morning Wednesday, November 21st, 2012
GC3 attends European Commission Cloud Expert Group
Posted mid-morning Monday, October 29th, 2012
SwiNG - SDCD2012 event
Posted at lunch time on Monday, October 22nd, 2012
Large Scale Computing Infrastructures class starts tomorrow!
Posted late Tuesday afternoon, September 25th, 2012
From bare metal to cloud at GC3
Posted mid-morning Monday, September 24th, 2012
GC3 at the EGI Technical Forum 2012
Posted Thursday night, September 20th, 2012
Training on GC3Pie and Python
Posted late Friday evening, September 7th, 2012
GC3Pie used for research in Computational Quantum Chemistry
Posted late Thursday afternoon, September 6th, 2012
``What's so great about MPI or Boost.MPI?''
Posted mid-morning Thursday, September 6th, 2012
blog/How to generate UML diagram with `pyreverse`
Posted late Thursday morning, August 23rd, 2012
Git's `rebase` command
Posted mid-morning Friday, June 15th, 2012
AppPot 0.27 released!
Posted at noon on Thursday, June 14th, 2012
Urban computing - connecting to your server using `mosh`
Posted mid-morning Wednesday, June 6th, 2012
Whitespace cleanup with Emacs
Posted Tuesday afternoon, June 5th, 2012
Translate pages on this site
Posted Thursday evening, May 31st, 2012
Scientific paper citing GC3Pie
Posted Wednesday evening, May 30th, 2012
GC3 attends Nordugrid 2012 conference
Posted at lunch time on Wednesday, May 30th, 2012
How the front page image was made
Posted late Wednesday evening, May 16th, 2012
GC3 blog launched!
Posted late Tuesday evening, May 15th, 2012
New GC3 Wiki now online!
Posted Tuesday evening, May 15th, 2012
AppPot paper on arXiv
Posted Tuesday evening, May 15th, 2012
GC3 at the EGI Technical Forum 2011
Posted Tuesday evening, May 15th, 2012

Continuous Integration System for GC3Pie

I've installed a jenkins VM instance on Hobbes, configured to automatically pull from the svn repository and run the tox tests on GC3Pie. The URL to access it is: http://130.60.193.34:8080/ You don't currently need any login and password to access it.

Configuration

Jenkins is fully configurabile via web interface, which is pretty intuitive. However, configuration files are XML files stored in the home of the jenkins user on the machine, so if you feel the need to update the files directly (useful if you have to update multiple files at once) you can just edit the files and restart jenkins.

I've configured 8 different builds, one for each combination of branch (trunk or 2.0) and python version (2.4, 2.5, 2.6, 2.7), so we have:

  • gc3pie-2.0_py2.4
  • gc3pie-2.0_py2.5
  • gc3pie-2.0_py2.6
  • gc3pie-2.0_py2.7
  • gc3pie-trunk_py2.4
  • gc3pie-trunk_py2.5
  • gc3pie-trunk_py2.6
  • gc3pie-trunk_py2.7

When a build is executed, the subversion tree is updated, then a new clean virtual environment is created and tox -e py** is run. The build script also take care of installing the correct version of tox (1.3) and virtualenv (1.7.2) to use to correctly create the python 2.4 environment[1].

I had to disable MySQL because of some problems I still have to investigate: sometimes tests fail just because they don't find the table they created. I think it has to do with transactions and implicit execution of SQLAlchemy and maybe the particular way we do the tests, but I couldn't replicate the issue so, for now, let's have clean builds :).

The script used to build, for example, gc3pie-trunk-2.7, looks like:

# Clean up old virtual environment
rm -rf gc3pie-env
# create the virtualenv
virtualenv gc3pie-env
. gc3pie-env/bin/activate
# Install tox
pip install tox
# Enter the SVN directory and run tox
cd gc3pie.svn/gc3pie
tox -e py27 -r

Notifications

Notifications are defined on a per-build basis. All our builds are configured to send emails to me and to the committer who broke the build. So if Riccardo do a commit which will break compatibility with one of the different combinations of branch/py-version, an email will be sent to him and to me. (I am receiving notifications only because I feel the need to check if jenkins is working, and because I am considered unanimously a ball-proker when it's about tests :))

Why not buildbot

After a few tests, I felt that even though buildbot is more flexible because of its python configuration files, jenkins is working out of the box exactly as we want, and if we will need to script the configuration we can always create a small python script to re-create the config.xml files.

There are also a few "minor" issues:

  • jenkins' web interface is way more clear and easy to use
  • jenkins does an auto-deploy on the slaves, with buildbot you have to install it by yourself (not a big problem, if we can ever use cfengine for this)
  • when you restart the buildbot master, the slaves are not automatically available, but you have to wait for them to contact the server again. With jenkins this does not happen because it does an ssh on the slaves as soon as it's up&running.
  • while buildbot is fully (and well) documented, you still have to RTFM to understand how to do something, and sometimes it's not trivial. For instance, I had a few problem before getting the SVNPoller configuration right. With jenkins you don't have to do much: everything is on the web form and it "just work"

Deployment

So far we don't have a way to install and configure VM inside Hobbes, so this is more or less what I did to install jenkins on the cloud. The image used is "Ubuntu-12.04.1-X86_64" ("our" image realized by Sergio).

  • Install the compiler and subversion:

      sudo apt-get update
      sudo apt-get install -y build-essential python-dev subversion libfreetype6-dev libpng-dev
    
  • Install jenkins from their repository:

      wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
      sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
      sudo apt-get update
      sudo apt-get install -y jenkins
    
  • Install older versions of Python:

      sudo add-apt-repository -y ppa:fkrull/deadsnakes
      sudo apt-get update
      sudo apt-get install -y python2.{4,5,6}{,-dev}
    
  • Install other GC3Pie dependencies:

      sudo apt-get install -y libmysqlclient-dev libsqlite3-dev
    
  • If you want to use MySQL for the tests as well, you have to install and configure it:

      sudo apt-get install -y mysql-server
      sudo mysql -p -e "create database gc3"
      sudo mysql -p -e "grant all privileges on gc3.* to gc3user@localhost identified by 'gc3pwd'
    

Slave deployment

Starting from the same image:

  • Install compiler and subversion

      sudo apt-get update
      sudo apt-get install -y build-essential python-dev subversion libfreetype6-dev libpng-dev
    
  • Install java:

      sudo apt-get install -y openjdk-7-jdk
      sudo add-apt-repository -y ppa:fkrull/deadsnakes
      sudo apt-get update
      sudo apt-get install -y python2.{4,5,6}{,-dev}
    
  • Install other GC3Pie dependencies:

      sudo apt-get install -y libmysqlclient-dev libsqlite3-dev
    
  • If you want to use MySQL for the tests as well, you have to install and configure it:

      sudo apt-get install -y mysql-server
      sudo mysql -p -e "create database gc3"
      sudo mysql -p -e "grant all privileges on gc3.* to gc3user@localhost identified by 'gc3pwd'
    
  • On the slave node you must create a jenkins user and set a passwordless environment for the jenkins master:

      sudo adduser --system jenkins
      sudo install -d -o jenkins -g nogroup -m 700 ~jenkins/.ssh
      sudo install -o jenkins -g nogroup -m 600 /dev/null ~jenkins/.ssh/authorized_keys
      echo 'ssh key of jenkins user on jenkins master' > ~jenkins/.ssh/authorized_keys
    
  • From the Jenkins web interface:

    • manage Jenkins => manage nodes => new node
    • add a "dumb slave" or copy the configuration from another node. You can use the private ip address.

[1] tox 1.3 and virtualenv 1.7.2 are the latest to give support for python 2.4, so you need those in order to create a working py24 environment. This also causes the dependency from Mysql-python 1.2.3 which is hardcoded in tox.ini, BTW.

top