GC3: Grid Computing Competence Center

Blog index

GC3 graudates into S3IT
Posted early Tuesday morning, July 1st, 2014
How to create a module that also load a virtualenvironment
Posted late Friday morning, March 7th, 2014
Openstack workshop at GC3
Posted at noon on Saturday, February 22nd, 2014
Moving LVM volumes used by a Cinder storage
Posted late Friday evening, February 21st, 2014
How to configure swift glusterfs
Posted Monday night, February 10th, 2014
Yet another timeout problem when starting many instances at once
Posted late Friday night, February 8th, 2014
Fixing LDAP Authentication over TLS/SSL
Posted Monday night, January 6th, 2014
Linker command-line options for Intel MKL
Posted Saturday night, January 4th, 2014
A virtue of lazyness
Posted Saturday afternoon, December 21st, 2013
(Almost) readable CFEngine logs
Posted Thursday afternoon, December 19th, 2013
CFEngine error: ExpandAndMapIteratorsFromScalar called with invalid strlen
Posted Wednesday afternoon, December 11th, 2013
'Martian source' log messages and the default IP route
Posted Monday afternoon, November 25th, 2013
GC3 takes over maintenance of the Schroedinger cluster
Posted at noon on Monday, November 4th, 2013
Grid Engine: how to find the set of nodes that ran a job (after it's finished)
Posted early Wednesday morning, October 30th, 2013
Python2 vs Python3
Posted at teatime on Friday, September 13th, 2013
GC3Pie 2.1.1 released
Posted Friday evening, September 6th, 2013
Happy SysAdmin day!
Posted mid-morning Friday, July 26th, 2013
Object-oriented Python training
Posted Thursday afternoon, July 25th, 2013
Elasticluster 1.0.0 released
Posted Thursday night, July 18th, 2013
Short Autotools tutorial
Posted at lunch time on Friday, July 5th, 2013
Patch Emacs' PostScript printing
Posted Tuesday evening, June 11th, 2013
Slides of the Object-oriented Python course now available!
Posted Tuesday evening, June 11th, 2013
Automated deployment of CFEngine keys
Posted at midnight, May 31st, 2013
blog/Resize_an_image
Posted Tuesday evening, May 14th, 2013
Join us at the Compute Cloud Experience Workshop!
Posted early Monday morning, April 29th, 2013
GC3 Beamer theme released
Posted at lunch time on Friday, April 5th, 2013
VM-MAD at the International Supercompting Conference 2013
Posted at lunch time on Tuesday, March 26th, 2013
The GC3 is on GitHub
Posted at lunch time on Monday, March 18th, 2013
How to enable search in IkiWiki
Posted Friday afternoon, March 15th, 2013
GC3Pie Training
Posted Thursday night, March 7th, 2013
Object-oriented Python training
Posted Thursday afternoon, March 7th, 2013
Advance Reservations in GridEngine
Posted late Thursday morning, March 7th, 2013
GridEngine accounting queries with PostgreSQL
Posted Wednesday night, March 6th, 2013
Floating IPs not available on Hobbes
Posted at teatime on Tuesday, February 26th, 2013
Notes on SWIFT
Posted mid-morning Tuesday, February 12th, 2013
An online Python code quality analyzer
Posted at lunch time on Saturday, February 9th, 2013
Seminar on cloud infrastructure
Posted Sunday night, February 3rd, 2013
GC3 announce its cloud infrastructure Hobbes
Posted Wednesday afternoon, January 30th, 2013
GC3Pie 2.0.2 released
Posted Monday afternoon, January 28th, 2013
Continuous Integration with Jenkins
Posted at noon on Saturday, January 26th, 2013
On the importance of testing in a clean environment
Posted mid-morning Monday, January 21st, 2013
Weirdness with ImageMagick's `convert`
Posted at teatime on Tuesday, January 15th, 2013
boto vs libcloud
Posted Tuesday afternoon, January 15th, 2013
Resolve timeout problem when starting many instances at once
Posted at lunch time on Monday, January 7th, 2013
Proceedings of the EGI Community Forum 2012 published
Posted at teatime on Monday, December 17th, 2012
SGE Workaround Installation
Posted at lunch time on Tuesday, December 4th, 2012
How to pass an argument of list type to a CFEngine3 bundle
Posted mid-morning Thursday, November 22nd, 2012
GC3 at the 'Clouds for Future Internet' workshop
Posted mid-morning Wednesday, November 21st, 2012
GC3 attends European Commission Cloud Expert Group
Posted mid-morning Monday, October 29th, 2012
SwiNG - SDCD2012 event
Posted at lunch time on Monday, October 22nd, 2012
Large Scale Computing Infrastructures class starts tomorrow!
Posted late Tuesday afternoon, September 25th, 2012
From bare metal to cloud at GC3
Posted mid-morning Monday, September 24th, 2012
GC3 at the EGI Technical Forum 2012
Posted Thursday night, September 20th, 2012
Training on GC3Pie and Python
Posted late Friday evening, September 7th, 2012
GC3Pie used for research in Computational Quantum Chemistry
Posted late Thursday afternoon, September 6th, 2012
``What's so great about MPI or Boost.MPI?''
Posted mid-morning Thursday, September 6th, 2012
blog/How to generate UML diagram with `pyreverse`
Posted late Thursday morning, August 23rd, 2012
Git's `rebase` command
Posted mid-morning Friday, June 15th, 2012
AppPot 0.27 released!
Posted at noon on Thursday, June 14th, 2012
Urban computing - connecting to your server using `mosh`
Posted mid-morning Wednesday, June 6th, 2012
Whitespace cleanup with Emacs
Posted Tuesday afternoon, June 5th, 2012
Translate pages on this site
Posted Thursday evening, May 31st, 2012
Scientific paper citing GC3Pie
Posted Wednesday evening, May 30th, 2012
GC3 attends Nordugrid 2012 conference
Posted at lunch time on Wednesday, May 30th, 2012
How the front page image was made
Posted late Wednesday evening, May 16th, 2012
GC3 blog launched!
Posted late Tuesday evening, May 15th, 2012
New GC3 Wiki now online!
Posted Tuesday evening, May 15th, 2012
AppPot paper on arXiv
Posted Tuesday evening, May 15th, 2012
GC3 at the EGI Technical Forum 2011
Posted Tuesday evening, May 15th, 2012

More on topic...

Moving LVM volumes used by a Cinder storage

After almost one year we have to realize that our Cloud testbed was not anymore a testbed but a production infrastructure. As such, we needed to fix some "mistakes" we made at the very first deployment.

Our cinder service is running on a machine with 48 disks and no hardware raid controller, and when we first deployed cinder we didn't bother to create a software raid but instead we just put a few disks into the cinder-volumes volume group, because, well, it was a test right?

However, now cinder was used in production, and some of the volumes contained important data we didn't want to lose. At the same time, we knew that keeping the infrastructure as it is it meant calling for troubles, therefore we decided to fix the situation.

The plan was, in theory, pretty clear:

  1. disable the cinder service
  2. backup all the volumes and snapshots in the cinder-volumes VG (just in case...)
  3. free a few disks
  4. create a raid6 array using the free disks
  5. add the raid array to the VG
  6. move all the extents used by the volumes/snapshots to the raid PV using pvmove
  7. remove all the remaining disks
  8. create a second raid array with the free disks
  9. add the second raid to the cinder-volumes VG

however we soon realized that step 5) was going to give us a bad time, since apparently it is impossible to just move extents used by a LVM snapshot to a different physical volume!

To solve the issue we just copied the snapshots, destroyed the snapshot, created a new standard volume with the same name, restored the backup.

Apparently, also volumes that are used as origin of a snapshot cannot entirely moved to a different physical volume, but as soon as you remove all its snapshots you can move it again.

Here I will show all the commands we issued and all the problems we encountered.

The situation

The starting point was the following: we had 15 devices used by the cinder-volumes volume group:

root@cloud-storage1:~# pvs
  PV         VG             Fmt  Attr PSize   PFree
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g
  /dev/sdaa  cinder-volumes lvm2 a-   465.76g 145.76g
  /dev/sdab  cinder-volumes lvm2 a-   465.76g  65.76g
  /dev/sdac  cinder-volumes lvm2 a-   465.76g      0
  /dev/sdad  cinder-volumes lvm2 a-   465.76g      0
  /dev/sdae  cinder-volumes lvm2 a-   465.76g  65.76g
  /dev/sdaf  cinder-volumes lvm2 a-   465.76g 365.76g
  /dev/sdag  cinder-volumes lvm2 a-   465.76g 465.76g
  /dev/sdah  cinder-volumes lvm2 a-   465.76g 465.76g
  /dev/sdai  cinder-volumes lvm2 a-   465.76g 465.76g
  /dev/sdaj  cinder-volumes lvm2 a-   465.76g 465.76g
  /dev/sdak  cinder-volumes lvm2 a-   465.76g 465.76g
  /dev/sdal  cinder-volumes lvm2 a-   465.76g 465.76g
  /dev/sdam  cinder-volumes lvm2 a-   465.76g 465.76g
  /dev/sdan  cinder-volumes lvm2 a-   465.76g 465.76g

root@cloud-storage1:~# lvs
  LV                                             VG             Attr   LSize   Origin                                      Snap%  Move Log Copy%  Convert
  _snapshot-00db31be-f73e-4def-978a-38141b55600e cinder-volumes swi-a- 200.00g volume-d6013c3d-d4a2-4560-8865-876286348d72  88.87
  _snapshot-48e20f4a-13b6-4b5b-9759-ee90afe643ff cinder-volumes swi-a-  40.00g volume-6c718c39-45fb-4efa-9fa7-4d212b514cb6   5.50
  _snapshot-80a7be58-7271-4f03-9799-3f7e4619131e cinder-volumes swi-a- 200.00g volume-d6013c3d-d4a2-4560-8865-876286348d72  10.68
  _snapshot-8dc87b68-9bca-4400-b5af-3715987dcd18 cinder-volumes swi-a-  40.00g volume-6c718c39-45fb-4efa-9fa7-4d212b514cb6  84.34
  _snapshot-945f3f95-1982-4a6f-83f9-380e71f0034e cinder-volumes swi-a-  40.00g volume-6c718c39-45fb-4efa-9fa7-4d212b514cb6   4.19
  _snapshot-97993e6d-402b-4751-a1a8-bbd3bbbf0cfc cinder-volumes swi-a- 200.00g volume-d6013c3d-d4a2-4560-8865-876286348d72  51.06
  _snapshot-ad35f612-c893-4af4-855a-a960e914c662 cinder-volumes swi-a-  40.00g volume-6c718c39-45fb-4efa-9fa7-4d212b514cb6  95.63
  _snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc cinder-volumes swi-a-  10.00g volume-3036bcd7-1bcb-41be-85f7-fefd34149ae7   0.00
  _snapshot-c6554542-4c46-4269-86c2-617441b132b5 cinder-volumes swi-a-  40.00g volume-6c718c39-45fb-4efa-9fa7-4d212b514cb6  27.90
  _snapshot-dc0c9322-d93b-4629-9fbd-5b67829750d0 cinder-volumes swi-a- 200.00g volume-d6013c3d-d4a2-4560-8865-876286348d72   4.80
  volume-3036bcd7-1bcb-41be-85f7-fefd34149ae7    cinder-volumes owi-ao  10.00g
  volume-33a607b0-2f97-4f2e-a212-26c334342897    cinder-volumes -wi-ao 500.00g
  volume-6c718c39-45fb-4efa-9fa7-4d212b514cb6    cinder-volumes owi-ao  40.00g
  volume-d6013c3d-d4a2-4560-8865-876286348d72    cinder-volumes owi-ao 200.00g
  volume-dadb743b-b47f-431c-a06b-bab678b28bc9    cinder-volumes -wi-ao 100.00g
  volume-dc9d9ccd-6a47-4355-8e24-6aa1ae810ae7    cinder-volumes -wi-a- 500.00g

10 snapshots, 6 volumes, 3 of them were used by some snapshot.

You can see what part of a PV is used by which LV using the command:

root@cloud-storage1:~# for dev in /dev/sda /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal /dev/sdam /dev/sdan; do pvs -v --segments $dev; done
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start  SSize LV                                             Start  Type   PE Ranges
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g      0  2560                                                     0 free
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g   2560  8766 volume-33a607b0-2f97-4f2e-a212-26c334342897    119234 linear /dev/sda:2560-11325
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g  11326  2560 _snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc      0 linear /dev/sda:11326-13885
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g  13886 62914                                                     0 free
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g  76800 10240 _snapshot-8dc87b68-9bca-4400-b5af-3715987dcd18      0 linear /dev/sda:76800-87039
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g  87040 10240 _snapshot-c6554542-4c46-4269-86c2-617441b132b5      0 linear /dev/sda:87040-97279
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g  97280 10240 _snapshot-48e20f4a-13b6-4b5b-9759-ee90afe643ff      0 linear /dev/sda:97280-107519
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g 107520  2560 volume-3036bcd7-1bcb-41be-85f7-fefd34149ae7         0 linear /dev/sda:107520-110079
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g 110080   186                                                     0 free
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g 110266  8766 volume-dc9d9ccd-6a47-4355-8e24-6aa1ae810ae7    119234 linear /dev/sda:110266-119031
  /dev/sda   cinder-volumes lvm2 a-   465.76g 257.27g 119032   202                                                     0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start  SSize LV                                             Start Type   PE Ranges
  /dev/sdaa  cinder-volumes lvm2 a-   465.76g 145.76g      0 10240 volume-6c718c39-45fb-4efa-9fa7-4d212b514cb6        0 linear /dev/sdaa:0-10239
  /dev/sdaa  cinder-volumes lvm2 a-   465.76g 145.76g  10240 10240 _snapshot-ad35f612-c893-4af4-855a-a960e914c662     0 linear /dev/sdaa:10240-20479
  /dev/sdaa  cinder-volumes lvm2 a-   465.76g 145.76g  20480 51200 volume-d6013c3d-d4a2-4560-8865-876286348d72        0 linear /dev/sdaa:20480-71679
  /dev/sdaa  cinder-volumes lvm2 a-   465.76g 145.76g  71680 25600                                                    0 free
  /dev/sdaa  cinder-volumes lvm2 a-   465.76g 145.76g  97280 10240 _snapshot-945f3f95-1982-4a6f-83f9-380e71f0034e     0 linear /dev/sdaa:97280-107519
  /dev/sdaa  cinder-volumes lvm2 a-   465.76g 145.76g 107520 11714                                                    0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree  Start  SSize LV                                             Start Type   PE Ranges
  /dev/sdab  cinder-volumes lvm2 a-   465.76g 65.76g      0 51200 _snapshot-00db31be-f73e-4def-978a-38141b55600e     0 linear /dev/sdab:0-51199
  /dev/sdab  cinder-volumes lvm2 a-   465.76g 65.76g  51200 51200 _snapshot-97993e6d-402b-4751-a1a8-bbd3bbbf0cfc     0 linear /dev/sdab:51200-102399
  /dev/sdab  cinder-volumes lvm2 a-   465.76g 65.76g 102400 16834                                                    0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree Start SSize  LV                                          Start Type   PE Ranges
  /dev/sdac  cinder-volumes lvm2 a-   465.76g    0      0 119234 volume-33a607b0-2f97-4f2e-a212-26c334342897     0 linear /dev/sdac:0-119233
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree Start SSize  LV                                          Start Type   PE Ranges
  /dev/sdad  cinder-volumes lvm2 a-   465.76g    0      0 119234 volume-dc9d9ccd-6a47-4355-8e24-6aa1ae810ae7     0 linear /dev/sdad:0-119233
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree  Start  SSize LV                                             Start Type   PE Ranges
  /dev/sdae  cinder-volumes lvm2 a-   465.76g 65.76g      0 51200 _snapshot-80a7be58-7271-4f03-9799-3f7e4619131e     0 linear /dev/sdae:0-51199
  /dev/sdae  cinder-volumes lvm2 a-   465.76g 65.76g  51200 51200 _snapshot-dc0c9322-d93b-4629-9fbd-5b67829750d0     0 linear /dev/sdae:51200-102399
  /dev/sdae  cinder-volumes lvm2 a-   465.76g 65.76g 102400 16834                                                    0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize LV                                          Start Type   PE Ranges
  /dev/sdaf  cinder-volumes lvm2 a-   465.76g 365.76g     0 25600 volume-dadb743b-b47f-431c-a06b-bab678b28bc9     0 linear /dev/sdaf:0-25599
  /dev/sdaf  cinder-volumes lvm2 a-   465.76g 365.76g 25600 93634                                                 0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize  LV   Start Type PE Ranges
  /dev/sdag  cinder-volumes lvm2 a-   465.76g 465.76g     0 119234          0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize  LV   Start Type PE Ranges
  /dev/sdah  cinder-volumes lvm2 a-   465.76g 465.76g     0 119234          0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize  LV   Start Type PE Ranges
  /dev/sdai  cinder-volumes lvm2 a-   465.76g 465.76g     0 119234          0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize  LV   Start Type PE Ranges
  /dev/sdaj  cinder-volumes lvm2 a-   465.76g 465.76g     0 119234          0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize  LV   Start Type PE Ranges
  /dev/sdak  cinder-volumes lvm2 a-   465.76g 465.76g     0 119234          0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize  LV   Start Type PE Ranges
  /dev/sdal  cinder-volumes lvm2 a-   465.76g 465.76g     0 119234          0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize  LV   Start Type PE Ranges
  /dev/sdam  cinder-volumes lvm2 a-   465.76g 465.76g     0 119234          0 free
    Using physical volume(s) on command line
  PV         VG             Fmt  Attr PSize   PFree   Start SSize  LV   Start Type PE Ranges
  /dev/sdan  cinder-volumes lvm2 a-   465.76g 465.76g     0 119234          0 free

or using pvdisplay -m, which gives you a more verbose output.

Step 0: disable the cinder service

First of all we detached all the instances currently in use using the cinder command. In principle the users were supposed to do it by themselves, but this is not always the case :-/

::
root@cloud1:~# cinder list --all-tenant +--------------------------------------+-----------+-----------------+------+-------------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Attached to | +--------------------------------------+-----------+-----------------+------+-------------+--------------------------------------+ | 3036bcd7-1bcb-41be-85f7-fefd34149ae7 | available | bodenbis_backup | 10 | None | | | 33a607b0-2f97-4f2e-a212-26c334342897 | available | storagemap | 500 | None | | | 6c718c39-45fb-4efa-9fa7-4d212b514cb6 | in-use | sys | 40 | None | 243d85ca-1acc-4845-8962-38092da00485 | | d6013c3d-d4a2-4560-8865-876286348d72 | in-use | project | 200 | None | 243d85ca-1acc-4845-8962-38092da00485 | | dadb743b-b47f-431c-a06b-bab678b28bc9 | available | bodenbis_space | 100 | None | | +--------------------------------------+-----------+-----------------+------+-------------+--------------------------------------+ root@cloud1:~# cinder snapshot-list --all-tenant +--------------------------------------+--------------------------------------+-----------+-----------------------------------+------+ | ID | Volume ID | Status | Display Name | Size | +--------------------------------------+--------------------------------------+-----------+-----------------------------------+------+ | 00db31be-f73e-4def-978a-38141b55600e | d6013c3d-d4a2-4560-8865-876286348d72 | available | snapshot for stacks1123 | 200 | | 48e20f4a-13b6-4b5b-9759-ee90afe643ff | 6c718c39-45fb-4efa-9fa7-4d212b514cb6 | available | snapshot for backup1207 | 40 | | 80a7be58-7271-4f03-9799-3f7e4619131e | d6013c3d-d4a2-4560-8865-876286348d72 | available | snapshot for backup1207 | 200 | | 8dc87b68-9bca-4400-b5af-3715987dcd18 | 6c718c39-45fb-4efa-9fa7-4d212b514cb6 | available | snapshot for stacks1123 | 40 | | 945f3f95-1982-4a6f-83f9-380e71f0034e | 6c718c39-45fb-4efa-9fa7-4d212b514cb6 | available | snapshot for 1218Backupwithfastst | 40 | | 97993e6d-402b-4751-a1a8-bbd3bbbf0cfc | d6013c3d-d4a2-4560-8865-876286348d72 | available | snapshot for 1205Jing | 200 | | ad35f612-c893-4af4-855a-a960e914c662 | 6c718c39-45fb-4efa-9fa7-4d212b514cb6 | available | snapshot for jingstacks1121 | 40 | | c6554542-4c46-4269-86c2-617441b132b5 | 6c718c39-45fb-4efa-9fa7-4d212b514cb6 | available | snapshot for 1205Jing | 40 | | dc0c9322-d93b-4629-9fbd-5b67829750d0 | d6013c3d-d4a2-4560-8865-876286348d72 | available | snapshot for 1218Backupwithfastst | 200 | +--------------------------------------+--------------------------------------+-----------+-----------------------------------+------+ root@cloud1:~# nova volume-detach 243d85ca-1acc-4845-8962-38092da00485 6c718c39-45fb-4efa-9fa7-4d212b514cb6 root@cloud1:~# nova volume-detach 243d85ca-1acc-4845-8962-38092da00485 d6013c3d-d4a2-4560-8865-876286348d72 root@cloud1:~# cinder list --all-tenant +--------------------------------------+-----------+-----------------+------+-------------+-------------+ | ID | Status | Display Name | Size | Volume Type | Attached to | +--------------------------------------+-----------+-----------------+------+-------------+-------------+ | 3036bcd7-1bcb-41be-85f7-fefd34149ae7 | available | bodenbis_backup | 10 | None | | | 33a607b0-2f97-4f2e-a212-26c334342897 | available | storagemap | 500 | None | | | 6c718c39-45fb-4efa-9fa7-4d212b514cb6 | available | sys | 40 | None | | | d6013c3d-d4a2-4560-8865-876286348d72 | available | project | 200 | None | | | dadb743b-b47f-431c-a06b-bab678b28bc9 | available | bodenbis_space | 100 | None | | +--------------------------------------+-----------+-----------------+------+-------------+-------------+

We also shut down the tgt service (responsible of serving the volumes via iSCSI to the compute nodes) and the cinder-volume service:

root@cloud-storage1:~# stop tgt
root@cloud-storage1:~# stop cinder-volume

Step 1: backup everything

On the same machines we had a partition big enough to be used for storing all the images. Since we didn't want to modify in any way the content of the disks (we don't know what's inside, actually), we decided to do a block copy of all the devices:

root@cloud-storage1:~# lvs|grep cinder-volumes|awk '{print $1}' \
  while read vol; do \
  dd if=/dev/cinder-volumes/$vol of=/backup/$vol bs=4M \
  done

This command took an huge amount of time, also because the disks are not so fast.

Step 2: remove free disks from the VG

Luckily enough, there were already a few disks currently unused by the VG, so we could just remove them with the command:

vgreduce -v -a cinder-volumes

This command will remove all unused devices from a volume group:

root@cloud-storage1:/srv/node/sdz# vgreduce -v -a cinder-volumes
    Finding volume group "cinder-volumes"
    Using all physical volume(s) in volume group
  Physical volume "/dev/sda" still in use
  Physical volume "/dev/sdaa" still in use
  Physical volume "/dev/sdab" still in use
  Physical volume "/dev/sdac" still in use
    Archiving volume group "cinder-volumes" metadata (seqno 113).
    Removing "/dev/sdad" from volume group "cinder-volumes"
    Creating volume group backup "/etc/lvm/backup/cinder-volumes" (seqno 114).
  Removed "/dev/sdad" from volume group "cinder-volumes"
  [...]
  Physical volume "/dev/sdae" still in use
  Physical volume "/dev/sdaf" still in use
  [...]
  Removed "/dev/sdag" from volume group "cinder-volumes"
  [...]
  Removed "/dev/sdah" from volume group "cinder-volumes"
  [...]
  Removed "/dev/sdai" from volume group "cinder-volumes"
  [...]
  Removed "/dev/sdaj" from volume group "cinder-volumes"
  [...]
  Removed "/dev/sdak" from volume group "cinder-volumes"
  [...]
  Removed "/dev/sdal" from volume group "cinder-volumes"
  [...]
  Removed "/dev/sdam" from volume group "cinder-volumes"
  [...]
  Removed "/dev/sdan" from volume group "cinder-volumes"

Step 3: create a raid6 array

Creating a raid6 array is quite easy in Linux:

root@cloud-storage1:~# mdadm --create /dev/md1 --level=6 \
    --raid-devices=8 /dev/sda{g,h,i,j,k,l,m,n}
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.

The array just created is already usable, although it's rebuilding:

root@cloud-storage1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid6 sdan[7] sdam[6] sdal[5] sdak[4] sdaj[3] sdai[2] sdah[1] sdag[0]
      2929529856 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
      [>....................]  resync =  0.0% (49664/488254976) finish=491.3min speed=16554K/sec

this could slow down a bit the process, and it may be critical in case you lose a disk while it's still rebuilding, but we decided to took our chances.

Step 4: add the array to the VG

First of all you need to create a PV on top of the raid array:

root@cloud-storage1:~# pvcreate /dev/md1
  Physical volume "/dev/md1" successfully created

and then you can add it to the volume group:

root@cloud-storage1:~# vgextend /dev/cinder-volumes /dev/md1
  Volume group "cinder-volumes" successfully extended

Step 5: remove and re-create snapshot volumes

At first we though that it was enough to run pvmove and move all the extents of both volumes and snapshots on the new array, but it seems that LVM is not able to do so because of the way snapshots are implemented.

Instead, we just removed all the snapshots and recreated as standard volumes with the same name and size. I will show here how we did it for just one snapshot. First of all, you have to check the number of extents used:

root@cloud-storage1:~# lvdisplay /dev/cinder-volumes/_snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc
  --- Logical volume ---
  LV Name                /dev/cinder-volumes/_snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc
  VG Name                cinder-volumes
  LV UUID                MkVlhU-YZ0t-VLuA-IEfY-4fdv-27KX-7Gv7Tk
  LV Write Access        read/write
  LV snapshot status     active destination for /dev/cinder-volumes/volume-3036bcd7-1bcb-41be-85f7-fefd34149ae7
  LV Status              available
  # open                 0
  LV Size                10.00 GiB
  Current LE             2560
  COW-table size         10.00 GiB
  COW-table LE           2560
  Allocated to snapshot  0.00%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:19

Note down the number of extents used by the snapshot (line Current LE and remove the snapshot:

root@cloud-storage1:~# lvremove /dev/cinder-volumes/_snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc
Do you really want to remove active logical volume _snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc? [y/n]: y
  Logical volume "_snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc" successfully removed

Create a logical volume with the same name and number of extents. Please note that we are explicitly creating the volume on the /dev/md1 physical device, otherwise we could end up on one of the disks we are trying to free!:

root@cloud-storage1:~# lvcreate \
    -n _snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc \
    -l 2560 cindder-volumes /dev/md1
  Logical volume "_snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc" created

and copy back the data:

root@cloud-storage1:~# dd if=/backup/_snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc \
    of=/dev/cinder-volumes/_snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc bs=4M

Repeat for all the snapshots.

Step 6: move all the extents to the new PV

The pvmove command allows you to move all the extents stored on a physical volume to a different physical volume. The syntax is straightforward, although again it will take an huge amount of time:

root@cloud-storage1:~# for dev in /dev/sda{,a,b,c,e,f}; do
    pvmove -v $dev /dev/md1; done
    Executing: /sbin/modprobe dm-mirror
    Finding volume group "cinder-volumes"
    Archiving volume group "cinder-volumes" metadata (seqno 123).
    Creating logical volume pvmove0
  Skipping snapshot-related LV volume-6c718c39-45fb-4efa-9fa7-4d212b514cb6
  Skipping snapshot-related LV _snapshot-ad35f612-c893-4af4-855a-a960e914c662
  Skipping snapshot-related LV volume-d6013c3d-d4a2-4560-8865-876286348d72
  Skipping snapshot-related LV _snapshot-8dc87b68-9bca-4400-b5af-3715987dcd18
  Skipping snapshot-related LV _snapshot-00db31be-f73e-4def-978a-38141b55600e
  Skipping snapshot-related LV _snapshot-97993e6d-402b-4751-a1a8-bbd3bbbf0cfc
  Skipping snapshot-related LV _snapshot-c6554542-4c46-4269-86c2-617441b132b5
  Skipping snapshot-related LV _snapshot-48e20f4a-13b6-4b5b-9759-ee90afe643ff
  Skipping snapshot-related LV _snapshot-80a7be58-7271-4f03-9799-3f7e4619131e
  Skipping snapshot-related LV _snapshot-945f3f95-1982-4a6f-83f9-380e71f0034e
  Skipping snapshot-related LV _snapshot-dc0c9322-d93b-4629-9fbd-5b67829750d0
  Skipping snapshot-related LV volume-3036bcd7-1bcb-41be-85f7-fefd34149ae7
    Moving 8766 extents of logical volume cinder-volumes/volume-33a607b0-2f97-4f2e-a212-26c334342897
  Skipping snapshot-related LV _snapshot-b49b7fce-3133-40cc-a5c8-8cbd75adaabc
    Found volume group "cinder-volumes"
    Updating volume group metadata
    Found volume group "cinder-volumes"
    Found volume group "cinder-volumes"
    Suspending cinder--volumes-volume--33a607b0--2f97--4f2e--a212--26c334342897 (252:18) with device flush
    Found volume group "cinder-volumes"
    Creating cinder--volumes-pvmove0
    Loading cinder--volumes-pvmove0 table (252:0)
    Resuming cinder--volumes-pvmove0 (252:0)
    Found volume group "cinder-volumes"
    Loading cinder--volumes-pvmove0 table (252:0)
    Suppressed cinder--volumes-pvmove0 identical table reload.
    Loading cinder--volumes-volume--33a607b0--2f97--4f2e--a212--26c334342897 table (252:18)
    Resuming cinder--volumes-volume--33a607b0--2f97--4f2e--a212--26c334342897 (252:18)
    Creating volume group backup "/etc/lvm/backup/cinder-volumes" (seqno 124).
    Checking progress before waiting every 15 seconds
  /dev/sda: Moved: 0.0%
  /dev/sda: Moved: 0.8%
  [...]
  /dev/sda: Moved: 100.0%
    Found volume group "cinder-volumes"
    Found volume group "cinder-volumes"
    Loading cinder--volumes-volume--33a607b0--2f97--4f2e--a212--26c334342897 table (252:18)
    Suspending cinder--volumes-volume--33a607b0--2f97--4f2e--a212--26c334342897 (252:18) with device flush
    Suspending cinder--volumes-pvmove0 (252:0) with device flush
    Found volume group "cinder-volumes"
    Found volume group "cinder-volumes"
    Found volume group "cinder-volumes"
    Resuming cinder--volumes-pvmove0 (252:0)
    Found volume group "cinder-volumes"
    Resuming cinder--volumes-volume--33a607b0--2f97--4f2e--a212--26c334342897 (252:18)
    Found volume group "cinder-volumes"
    Removing cinder--volumes-pvmove0 (252:0)
    Found volume group "cinder-volumes"
    Removing temporary pvmove LV
    Writing out final volume group after pvmove
    Creating volume group backup "/etc/lvm/backup/cinder-volumes" (seqno 126).
    [...]

Step 7: remove free disks from VG and add them back as a raid6 array

Now that all the bare disks have been emptied, we can remove them from the volume group:

root@cloud-storage1:~# vgreduce -v -a cinder-volumes

create a new raid6 array:

root@cloud-storage1:~# mdadm --create /dev/md2 --level=6 \
    --raid-devices=8 /dev/sda{,a,b,c,e,f} /dev/sd{c,e}

and add the new raid array to the cinder-volumes VG:

root@cloud-storage1:~# vgextend /dev/cinder-volumes /dev/md2
  Volume group "cinder-volumes" successfully extended

Optionally, you may want to move some of the extents from the /dev/md1 device to balance the load over the two arrays. pvmove also has the -n option to only move extents belonging to a specific logical volume.

Conclusions

LVM allow you a great level of flexibility, but it has some issues. Moreover, every time you do this kind of operations you need a lot of time during which the service is unavailable.

The lesson learned (once again) is therefore:

Always always think first, act later

And, of course, always do backups for data you don't ever want to lose :)

-- AM

top