My desktop has a LVM2 setup combining three harddrives for a good 389GB chunk of space. Unfortunately that just isn’t enough in this modern world. So I’ve been planning on getting another harddrive, and possibly a new motherboard as well. If I go the new motherboard route I’ll likely be reduced to two PATA slots, but I’ll gain four SATA slots. Either way I need to trim out one harddrive.

So I decided to get a jump on that because it’s something I haven’t done before.

Obviously the smallest of the three harddrives had to go, and in this case it was a 37GB harddrive (hdb1). So before I could remove it I had to shrink my Volume Group ‘vg’ by 37GB and move all the extends (like pages but larger) off of hdb1 and onto the harddrives remaining in the Volume Group (hda3 and hdc1). The Volume Group ‘vg’ had two Logical Volumes, a 30GB /opt and a 359GB /home. I moved some of the files off of the /opt Volume on to my root partition which was not part of the LVM setup and had extra space. I then deleted the rest. /opt was now empty so I ran

lvremove vg/opt

(after unmounting it). Voila, my Volume Group now had 30GB free, but I needed 7GB more.

This was a little trickier, but not too bad. I copied some more files from /home to my root partition until /home had a little over 10GB free space. Then I unmounted /home and ran

reiserfs_resize -10G /dev/mapper/vg-home

(I left some wiggle room on /home so it wasn’t 100% full because that’s just a bad plan). Once my /home partition was shrunk, I shrunk it’s Logical Volume with

lvreduce -L -9.5G vg/home

(I didn’t quite shrink the Logical Volume to match the new filesysystem size for safety sake. A little wasted space is ok, shrinking a partition any bit less than the filesystem size is fatal).

Now came the hard part, migrating the extends. As it turned out, none of the space we freed up was on the drive we wanted to remove. A shame. It was split across the other two drives, and neither chunk was the size of the drive we wanted to remove.

It was time to get familiar with the pvmove command that moves extends. As it turns out it’s not the brightest command. Ideally, I should be able to go

pvmove /dev/hdb1

and all extends on hdb1 will be moved off of it. And that’s what the manual says will happen too. Jolly days. Except pvmove is dumb, and by default, it only ‘sees’ the first chunk of free space, and it won’t use multiple chunks. So the nice and simple command I’d like to use failed. More research.

Running

vgcfgbackup

generated a “map” of extends, volume groups and how they map to the physical volumes, located in /etc/lvm/backup/vg. It took some study to understand it, but basically it has two sections. The first section merely maps drives like hdb1 to physical volumes, in this case pv1. The second section maps chunks of extends or “segments” to logical volumes. It can be a bit misleading because it has a “start” and “count” value, but those don’t match if you are mapping extends to the Physical Volume. The start corresponds to where in the Volume Group you are. If you are mapping extends to a Physical Volume you have to look down a bit more and use the line that has the Physical Volume name like “pv1” which will be followed by a number which is where on the Physical Volume this segment starts. To calculate the segments end simply add PV start + count. It’s a bit confusing. I finally got paper and mapped it out there, and I strongly suggest everyone else do the same. The first section also tells how many extends each Physical Volume has so once I had a map I could see how much space was free and in what sized chunks.

Now I started to move chunks of the segment from hdb1 to the other drives. I started at the beginning and moved as much as would fit on the first free available space. You can tell pvmove how much to take by “slicing” the Volume.

pvmove /dev/hdb1:1-1500

moves the first 1500 extends off of hdb1 to the first free space.

pvmove /dev/hdb1:1-1500 /dev/hdc1

moves the first 1500 extends off of hdb1 to the first free space on hdc1. Finally, you’ll probably want to use the -v flag and the -i # where # is the interval between progress reports from pvmove. It defaults to 15 seconds. Once the first 1500 extends are moved, you can move the next chunk with a command like

pvmove -v -i 5 /dev/hdb1:1500-5000

which moves the next 3500 extends, verbosely, showing a status update every 5 seconds instead of the default 15.

In my case, if I’d better understood all this at the beginning I would have only had to do 2 moves, but since I was a little confused I ended up with 3 or 4. You probably want to be careful because each move seems to create a new segment on the destination Volume Group, it doesn’t just merge segments. And I just assume fragmentation is less than good. But probably at this stage (a few fragments for ~400GB on two drives) not so bad either.

Now the hdb1 Physical Volume was empty, I could finally remove it. And it was as easy as

vgreduce vg /dev/hdb1

Now the drive was unused by anything and ready for removal.

Pretty sweet to be able to do all this. I even did the pvmove stuff with my /home mounted. Probably not a fantastic idea, but it worked, which is pretty crazy.

So what have we learned about LVM2? It’s insanely powerful and cool, but the pvmove command could really use some work, and getting the segment layout/map of everything is a bit of a pain to interpret. But with some care, cool things can be done.

Reference material

And if you plan on trying this, the pvdisplay command is pretty handy for tracking some progress. But really, get familiar with all the pv* lv* and vg* commands. At least read the tops of all their man pages, and skim the entire TLDP FAQ. It is a complex procedure and works on several levels. Be familiar with your tools before starting. And don’t be cocky like me, make some backups of important stuff.