GC3: Grid Computing Competence Center

Blog index

GC3 graudates into S3IT
Posted early Tuesday morning, July 1st, 2014
How to create a module that also load a virtualenvironment
Posted late Friday morning, March 7th, 2014
Openstack workshop at GC3
Posted at noon on Saturday, February 22nd, 2014
Moving LVM volumes used by a Cinder storage
Posted late Friday evening, February 21st, 2014
How to configure swift glusterfs
Posted Monday night, February 10th, 2014
Yet another timeout problem when starting many instances at once
Posted late Friday night, February 8th, 2014
Fixing LDAP Authentication over TLS/SSL
Posted Monday night, January 6th, 2014
Linker command-line options for Intel MKL
Posted Saturday night, January 4th, 2014
A virtue of lazyness
Posted Saturday afternoon, December 21st, 2013
(Almost) readable CFEngine logs
Posted Thursday afternoon, December 19th, 2013
CFEngine error: ExpandAndMapIteratorsFromScalar called with invalid strlen
Posted Wednesday afternoon, December 11th, 2013
'Martian source' log messages and the default IP route
Posted Monday afternoon, November 25th, 2013
GC3 takes over maintenance of the Schroedinger cluster
Posted at noon on Monday, November 4th, 2013
Grid Engine: how to find the set of nodes that ran a job (after it's finished)
Posted early Wednesday morning, October 30th, 2013
Python2 vs Python3
Posted at teatime on Friday, September 13th, 2013
GC3Pie 2.1.1 released
Posted Friday evening, September 6th, 2013
Happy SysAdmin day!
Posted mid-morning Friday, July 26th, 2013
Object-oriented Python training
Posted Thursday afternoon, July 25th, 2013
Elasticluster 1.0.0 released
Posted Thursday night, July 18th, 2013
Short Autotools tutorial
Posted at lunch time on Friday, July 5th, 2013
Patch Emacs' PostScript printing
Posted Tuesday evening, June 11th, 2013
Slides of the Object-oriented Python course now available!
Posted Tuesday evening, June 11th, 2013
Automated deployment of CFEngine keys
Posted at midnight, May 31st, 2013
Posted Tuesday evening, May 14th, 2013
Join us at the Compute Cloud Experience Workshop!
Posted early Monday morning, April 29th, 2013
GC3 Beamer theme released
Posted at lunch time on Friday, April 5th, 2013
VM-MAD at the International Supercompting Conference 2013
Posted at lunch time on Tuesday, March 26th, 2013
The GC3 is on GitHub
Posted at lunch time on Monday, March 18th, 2013
How to enable search in IkiWiki
Posted Friday afternoon, March 15th, 2013
GC3Pie Training
Posted Thursday night, March 7th, 2013
Object-oriented Python training
Posted Thursday afternoon, March 7th, 2013
Advance Reservations in GridEngine
Posted late Thursday morning, March 7th, 2013
GridEngine accounting queries with PostgreSQL
Posted Wednesday night, March 6th, 2013
Floating IPs not available on Hobbes
Posted at teatime on Tuesday, February 26th, 2013
Notes on SWIFT
Posted mid-morning Tuesday, February 12th, 2013
An online Python code quality analyzer
Posted at lunch time on Saturday, February 9th, 2013
Seminar on cloud infrastructure
Posted Sunday night, February 3rd, 2013
GC3 announce its cloud infrastructure Hobbes
Posted Wednesday afternoon, January 30th, 2013
GC3Pie 2.0.2 released
Posted Monday afternoon, January 28th, 2013
Continuous Integration with Jenkins
Posted at noon on Saturday, January 26th, 2013
On the importance of testing in a clean environment
Posted mid-morning Monday, January 21st, 2013
Weirdness with ImageMagick's `convert`
Posted at teatime on Tuesday, January 15th, 2013
boto vs libcloud
Posted Tuesday afternoon, January 15th, 2013
Resolve timeout problem when starting many instances at once
Posted at lunch time on Monday, January 7th, 2013
Proceedings of the EGI Community Forum 2012 published
Posted at teatime on Monday, December 17th, 2012
SGE Workaround Installation
Posted at lunch time on Tuesday, December 4th, 2012
How to pass an argument of list type to a CFEngine3 bundle
Posted mid-morning Thursday, November 22nd, 2012
GC3 at the 'Clouds for Future Internet' workshop
Posted mid-morning Wednesday, November 21st, 2012
GC3 attends European Commission Cloud Expert Group
Posted mid-morning Monday, October 29th, 2012
SwiNG - SDCD2012 event
Posted at lunch time on Monday, October 22nd, 2012
Large Scale Computing Infrastructures class starts tomorrow!
Posted late Tuesday afternoon, September 25th, 2012
From bare metal to cloud at GC3
Posted mid-morning Monday, September 24th, 2012
GC3 at the EGI Technical Forum 2012
Posted Thursday night, September 20th, 2012
Training on GC3Pie and Python
Posted late Friday evening, September 7th, 2012
GC3Pie used for research in Computational Quantum Chemistry
Posted late Thursday afternoon, September 6th, 2012
``What's so great about MPI or Boost.MPI?''
Posted mid-morning Thursday, September 6th, 2012
blog/How to generate UML diagram with `pyreverse`
Posted late Thursday morning, August 23rd, 2012
Git's `rebase` command
Posted mid-morning Friday, June 15th, 2012
AppPot 0.27 released!
Posted at noon on Thursday, June 14th, 2012
Urban computing - connecting to your server using `mosh`
Posted mid-morning Wednesday, June 6th, 2012
Whitespace cleanup with Emacs
Posted Tuesday afternoon, June 5th, 2012
Translate pages on this site
Posted Thursday evening, May 31st, 2012
Scientific paper citing GC3Pie
Posted Wednesday evening, May 30th, 2012
GC3 attends Nordugrid 2012 conference
Posted at lunch time on Wednesday, May 30th, 2012
How the front page image was made
Posted late Wednesday evening, May 16th, 2012
GC3 blog launched!
Posted late Tuesday evening, May 15th, 2012
New GC3 Wiki now online!
Posted Tuesday evening, May 15th, 2012
AppPot paper on arXiv
Posted Tuesday evening, May 15th, 2012
GC3 at the EGI Technical Forum 2011
Posted Tuesday evening, May 15th, 2012

More on topic...

Resizing (shrinking) an qcow2 image

Not the easiest thing to do, resizing a virtual image, it can be really hard if you need to shrink the image.

My problem: I've created an image using hobbes and the m1.small flavor which creates a 20G big disk, but since the m1.tiny flavor on Hobbes always resize the image to 5GB I had to resize the image before uploading again on hobbes. In theory what you need to do is:

  1. download the snapshot using glance image-download
  2. shrink the image
  3. upload the image using glance image-create

While steps 1) and 3) are trivial, shrinking the image is a bit more complex. The qemu-img command in principle allows you to shrink the image, but it does not work with qcow images, and you still have to resize the filesystem before resizing the disk. So, step 2) becomes:

  1. resize the filesystem contained in the image
  2. convert the image from qcow2 to a format that can be shrunk (AFAIK, only raw allows you to)
  3. resize the *raw* image
  4. convert back from raw to qcow2. This step is not necessary but qcow2 images are better handled by OpenStack/KVM so it's good to do it.

Let's enter into details:

a. Resize the filesystem

The qcow2 file is basically a file containing the image of a partitioned disk, so you cannot access the filesystem directly since this is inside a partition. In principle you could use losetup to access the filesystem if you know where it starts, but an easier path is to attach it to an nbd device. To do that, you can use the qemu-nbd command. But first, ensure you have the nbd module loaded:

# modprobe nbd

This command shouldn't give any error. Now, you can attach the image file to a nbd device:

# qemu-nbd -c /dev/nbd0 bigdisk.qcow2

This will create a device called /dev/nbd0 which will reflect the content of the bigdisk.qcow2 file. Now you can see the partitions contained in the file:

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

     Device Boot      Start         End      Blocks   Id  System
/dev/nbd0p1   *       16065    41929649    20956792+  83  Linux

In this case I have a partition only which is around 20GB big. Since I already know the partition contains a ext3 filesystem I can use resize2fs to resize it, and as you can guess from the above output the path /dev/nbd0p1 will points directly to my partition. Before resizing the filesystem is a good idea to run a check of it:

# e2fsck -f /dev/nbd0p1
e2fsck 1.42.5 (29-Jul-2012)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
cloudimg-rootfs: 159592/1310720 files (0.1% non-contiguous), 1312462/5239198 blocks

Let's resize the filesystem now! (the exclamation mark is misleading, since the following operation takes a lot of time. Better have a cup of coffee in the meantime...)

resize2fs   /dev/nbd0p1 5000M
resize2fs 1.42.5 (29-Jul-2012)
Resizing the filesystem on /dev/nbd0p1 to 1280000 (4k) blocks.
The filesystem on /dev/nbd0p1 is now 1280000 blocks long.

Now that we know how big is our filesystem (1280000 * 4k) we have to manually reduce the partition using fdisk:

# fdisk /dev/nbd0

Command (m for help): p

Disk /dev/nbd0: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

     Device Boot      Start         End      Blocks   Id  System
/dev/nbd0p1   *       16065    41929649    20956792+  83  Linux

Please note the Start value of the partition: 16065. We will need it later on. Let's first destroy the current partition:

Command (m for help): d
Selected partition 1

Now we can create the new partition. The new size of the partition is computed by multiplying the value printed out by resize2fs (in my case: 1280000 times 4 kb) by a value reasonably greater than 1, in my case I choose 1.003, so the result is 5273600:

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-41943039, default 2048): 16065
Last sector, +sectors or +size{K,M,G} (16065-41943039, default 41943039): +5273600K

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Now we can disconnect the /dev/nbd0 device by running:

# qemu-nbd -d /dev/nbd0

b. Convert the filesystem to raw format

This is quite easy indeed, but takes some time:

# qemu-img convert -O raw bigdisk.qcow2 smalldisk.img

c. Resize the raw image

Also this is quite easy and also fast:

# qemu-img resize smalldisk.img 5G
Image resized.

The smalldisk.img should now be quite smaller in size:

# ls -sh smalldisk.img
5.0G smalldisk.img

d. Convert back from raw to qcow2

Converting back from raw to qcow2 is now trivial:

# qemu-img convert -O qcow2 smalldisk.img smalldisk.qcow2