Less clicks, saved time
Source: Let’s get rid of double-clicking! Apple, Microsoft — please do something!
Kind: Articles
traditional long form content: a post with an explicit title and body
Standard fare in Linux, like, forever.
♲ Stefan Judis (stefanjudis@twitter.com) 2019-06-07 09:50:40:
TIL: if you’re using iterm on a Mac you can enable “Natural Text Editing” which allows you to navigate by words using ‘option + left/right’. 🎉
apple.stackexchange.com/questions/1369…
https://video.twimg.com/ext_tw_video/1136933430535622656/pu/vid/1194×720/Jfkwn5jwFVr1PJJP.mp4?tag=10
Using Alt/Cmd + Right/Left Arrow in iTerm
Is it possible to move between words in iTerm using Alt + Right/Left Arrows ? Now if I press Alt+Left I will get ‘[D’ and ‘[C’ if I press Alt+Right.
How to safely shrink a mdadm Linux RAID containing LVM Logical Volumes
The title of this article could have been, “How to shrink a mdadm Linux RAID containing LVM Logical Volumes and replace physical disks with lower-capacity alternatives – AND keep your data safe!”.
That’s just clunky, of course, but I hope you get the point.
Preliminaries
I recently ran into the situation where I needed to reduce the size of my physical volumes on a server to allow an easier backup strategy. The data is stored across four drives, making up two separate RAID1 arrays, which are then utilised by Logical Volume Manager in Linux.
If all of this sounds overly complex, well … it perhaps is a litle. I could just store my data on two RAID1 arrays with Ext4 file systems, and not worry about LVM. However, I have long been a convert of the superior disk space management of LVM, so sitting that atop of a mdadm RAID just adds a few more steps to the process, but a lot more flexibility.
Without further ado, let’s get started!
First things, first: Do a backup.
Using RAID for storage is one thing, but it is no substitute for a proper backup. The golden rule is:
RAID provides availability; backups provide recovery.
If everyting goes south in this process, having a current backup will save the day. (I won’t divulge how many times LTO backups have saved my posterior…). In short, I fully trust LTO backups, and a tape backup strategy generally (here’s why).
The disk sizes of my two RAID arrays are 2x400GB and 2x2TB. However, the storage used on each is only 100GiB and 500GiB, respectively. Sitting on a shelf, I have:
- 2 x 250GB Seagate 7200.12 ST3250312AS drives (PDF)
- 2 x 1000GB Seagate Constellation ES ST1000NM0011 drives (PDF)
On this server, I don’t envisage my data growth exceeding about 20GB & 40GB per year, respective to the arrays, so I can safely downgrade my storage capacity and make best use of this hardware for some years to come.
Connected to my server are two LTO tape drives: an old Ultrium 232 (100GB/200GB) drive, and a slightly younger Ultrium 1760 (800GB/1600GB) drive. The beauty of my set-up is that nearly everything is parallelised:
- Dual core CPU with dual channel memory
- 4 x SATA drive interfaces running 2 x RAID 1 arrays
- 2 x PCIe-to-SCSI host bus adapters
- 2 x SCSI tape drives, one connected to each HBA
All of this is configured within a tiny HP Microserver (PDF), excluding one of the tape drives which is an external unit. And the beauty of this parallelisation is that I can run backups from each RAID array to each tape drive at full speed, without the CPU even breaking a sweat!
For each mounted volume, I simply used the GNU dump command to back everything up on each drive:
root@stargate:/mnt/company# dump -b 64 -0u -f /dev/st0 /mnt/company/ DUMP: Date of this level 0 dump: Thu Aug 16 15:57:06 2018 DUMP: Dumping /dev/mapper/md0--vg-lv--company (/mnt/company) to /dev/st0 DUMP: Label: company DUMP: Writing 64 Kilobyte records DUMP: mapping (Pass I) [regular files] DUMP: mapping (Pass II) [directories] DUMP: estimated 94061522 blocks. DUMP: Volume 1 started with block 1 at: Thu Aug 16 15:57:19 2018 DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] DUMP: 6.27% done at 19662 kB/s, finished in 1:14 DUMP: 13.30% done at 20855 kB/s, finished in 1:05 [ ... ] DUMP: 91.52% done at 19117 kB/s, finished in 0:06 DUMP: 96.59% done at 18915 kB/s, finished in 0:02 DUMP: Closing /dev/st0 DUMP: Volume 1 completed at: Thu Aug 16 17:21:06 2018 DUMP: Volume 1 94060544 blocks (91856.00MB) DUMP: Volume 1 took 1:23:47 DUMP: Volume 1 transfer rate: 18711 kB/s DUMP: 94060544 blocks (91856.00MB) on 1 volume(s) DUMP: finished in 4974 seconds, throughput 18910 kBytes/sec DUMP: Date of this level 0 dump: Thu Aug 16 15:57:06 2018 DUMP: Date this dump completed: Thu Aug 16 17:21:06 2018 DUMP: Average transfer rate: 18711 kB/s DUMP: DUMP IS DONE
root@stargate:~# dump -b 512 -0u -f /dev/st1 /mnt/archive/ DUMP: Date of this level 0 dump: Thu Aug 16 15:59:53 2018 DUMP: Dumping /dev/mapper/md1--vg-lv--archive (/mnt/archive) to /dev/st1 [ ... ] DUMP: 3.48% done at 59490 kB/s, finished in 2:18 DUMP: 7.98% done at 68256 kB/s, finished in 1:55 [ ... ] DUMP: 512711168 blocks (500694.50MB) on 1 volume(s) DUMP: finished in 6143 seconds, throughput 83462 kBytes/sec DUMP: Date of this level 0 dump: Thu Aug 16 15:59:53 2018 DUMP: Date this dump completed: Thu Aug 16 17:44:01 2018 DUMP: Average transfer rate: 82455 kB/s DUMP: DUMP IS DONE
The keen reader may notice that I change the block size of dump for each drive. I have found doing so reduced the disk I/O operations for each backup, which improved throughput. I had trouble using a block size of 1024KiB though, so 512KiB sufficed on the faster drive.
While 20MiB/s and 80MiB/s may be laughed at these days (and yes, it doesn’t compare with modern USB3.1 RAIDs, for example), this speed does at least allow backups to be completed within a reasonable time. And the reason I am an LTO stalwart is that media is cheaper (per GiB), and more resilient when kept in cold storage.
I should mention at this point, for anyone interested in using GNU dump/restore to manage their backups, that the software isn’t limited to addressing locally-connected tape drives. You can also backup to a local file, a file shared on a NAS server, or a remote tape drive too. The GNU dump/restore documentation is well worth investigating.
Resizing storage in Logical Volume Manager
With backups available, you could ask why I didn’t just whip out the old drives and install the new ones, create new arrays and then restore data from backup. My reason for not doing so is twofold:
- I want to keep the backup tapes pristine in case of a real, last-resort need. I am confident they store the data because of LTO’s verify-as-written approach.
- Replacing the drives one-by-one allows them to be fully exercised when the RAID arrays are rebuilt. This will test that each drive is performing reliably; any problem can be easily rectified at this stage.
The next step is to resize the logical volumes that sit on those RAID arrays. In LVM you have to work outwards, starting from the innermost point: the logical volume. Then we’ll look at the volume group, and finally the physcal volume.
Resize the logical volumes:
root@stargate:~# lvresize -L 200G -r /dev/md0-vg/lv-company fsck from util-linux 2.25.2 company: 619459/24412160 files (0.2% non-contiguous), 24895191/97644544 blocks resize2fs 1.42.12 (29-Aug-2014) Resizing the filesystem on /dev/mapper/md0--vg-lv--company to 52428800 (4k) blocks. The filesystem on /dev/mapper/md0--vg-lv--company is now 52428800 (4k) blocks long. Size of logical volume md0-vg/lv-company changed from 372.48 GiB (95356 extents) to 200.00 GiB (51200 extents). Logical volume lv-company successfully resized
root@stargate:~# lvresize -L 800G -r /dev/md1-vg/lv-archive fsck from util-linux 2.25.2 archive: 2097049/108986368 files (0.1% non-contiguous), 134664369/435915776 blocks resize2fs 1.42.12 (29-Aug-2014) Resizing the filesystem on /dev/mapper/md1--vg-lv--archive to 209715200 (4k) blocks. The filesystem on /dev/mapper/md1--vg-lv--archive is now 209715200 (4k) blocks long. Size of logical volume md1-vg/lv-archive changed from 1.62 TiB (425699 extents) to 800.00 GiB (204800 extents). Logical volume lv-archive successfully resized
Now that we’ve resized the logical volumes, let’s check the size of the volume groups:
root@stargate:~# vgs VG #PV #LV #SN Attr VSize VFree md0-vg 1 1 0 wz--n- 372.48g 172.48g md1-vg 1 1 0 wz--n- 1.82t 1.04t
This looks good, and broadly corresponds to what we’ve done so far:
- I resized the LV on md0-vg to 200GiB, which leaves around 172.5GiB free
- I resized the LV on md1-vg to 800GB, which leaves around 1TiB free
So far, so good.
Next, we want to reduce the size of those Volume Groups to fit within the new mdadm RAID sizes. In LVM, sometimes multi-level operations are combined. One example of this is resizing a Volume Group. As there isn’t a vgresize command in lvm, we have to look at the next level out – the physical volume. For this, we use pvresize.
root@stargate:~# pvresize -v --setphysicalvolumesize 210G /dev/md0 DEGRADED MODE. Incomplete RAID LVs will be processed. Using physical volume(s) on command line Archiving volume group "md0-vg" metadata (seqno 5). /dev/md0: Pretending size is 440401920 not 781156528 sectors. Resizing volume "/dev/md0" to 781156528 sectors. Resizing physical volume /dev/md0 from 0 to 53759 extents. Updating physical volume "/dev/md0" Creating volume group backup "/etc/lvm/backup/md0-vg" (seqno 6). Physical volume "/dev/md0" changed 1 physical volume(s) resized / 0 physical volume(s) not resized
root@stargate:~# pvresize -v --setphysicalvolumesize 810G /dev/md1 DEGRADED MODE. Incomplete RAID LVs will be processed. Using physical volume(s) on command line Archiving volume group "md1-vg" metadata (seqno 14). /dev/md1: Pretending size is 1698693120 not 3906762895 sectors. Resizing volume "/dev/md1" to 3906762895 sectors. Resizing physical volume /dev/md1 from 0 to 207359 extents. Updating physical volume "/dev/md1" Creating volume group backup "/etc/lvm/backup/md1-vg" (seqno 15). Physical volume "/dev/md1" changed 1 physical volume(s) resized / 0 physical volume(s) not resized
Again, the astute reader will see that my resized capacities are slightly higher (by 10GiB) than the LVs contained within. I do this (mainly out of habit) to allow a little “wiggle room” for data, i.e. to allow safe boundaries between filesystem, LV, VG and PV. Subsequently resizing every compoment to its fullest capacity is easily done.
Checking the physical storage of data
Before we go ahead with a drive removal and replacement, we should test that the PVs (Physical Volumes) in LVM are storing data at the beginning of the allocatable region – i.e. at the beginning of the “disk”. This is accomplished quite easily:
root@stargate:~# pvdisplay --maps --- Physical volume --- PV Name /dev/md1 VG Name md1-vg PV Size 810.00 GiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 207359 Free PE 2559 Allocated PE 204800 PV UUID luUdnY-3416-wh3O-8fpC-JK6B-0oM7-fbYvo4 --- Physical Segments --- Physical extent 0 to 204799: Logical volume /dev/md1-vg/lv-archive Logical extents 0 to 204799 Physical extent 204800 to 207358: FREE --- Physical volume --- PV Name /dev/md0 VG Name md0-vg PV Size 210.00 GiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 53759 Free PE 2559 Allocated PE 51200 PV UUID Xzj831-tu43-G6cA-05LB-cccU-Tcae-dy2LqP --- Physical Segments --- Physical extent 0 to 51199: Logical volume /dev/md0-vg/lv-company Logical extents 0 to 51199 Physical extent 51200 to 53758: FREE
This looks good: In LVM, each PV is allocating all of its Volume Group data at the beginning of the space.
But what about mdadm? How do we know that mdadm is storing its data at the beginning of the disks?
As it turns out, this doesn’t matter. The main thing is that need to shrink mdadm’s RAID to be the right size for the physical disks that the RAID inhabits. This is where things could get a little funky, as mdadm has no idea where LVM data is stored. Hence, the reason for a backup.
Safely shrinking the mdadm RAID
To quickly re-cap, we have:
- in md0:
- shrunk the LV and its ext4 filesystem to 200GiB
- shrunk the PV/VG to 210GiB
- in md1:
- shrunk the LV and its ext4 filesystem to 800GiB
- shrunk the PV/VG to 810GiB
According to the mdadm manual (my underlines):
For create, build, or grow:
-z, –size=
Amount (in Kibibytes) of space to use from each drive in RAID levels 1/4/5/6. This must be a multiple of the chunk size, and must leave about 128Kb of space at the end of the drive for the RAID superblock. If this is not specified (as it normally is not) the smallest drive (or partition) sets the size, though if there is a variance among the drives of greater than 1%, a warning is issued.
A suffix of ‘M’ or ‘G’ can be given to indicate Megabytes or Gigabytes respectively.
Sometimes a replacement drive can be a little smaller than the original drives though this should be minimised by IDEMA standards. Such a replacement drive will be rejected by md. To guard against this it can be useful to set the initial size slightly smaller than the smaller device with the aim that it will still be larger than any replacement.
This value can be set with —grow for RAID level 1/4/5/6 though CONTAINER based arrays such as those with IMSM metadata may not be able to support this. If the array was created with a size smaller than the currently active drives, the extra space can be accessed using —grow. The size can be given as max which means to choose the largest size that fits on all current drives.
Before reducing the size of the array (with —grow –size=) you should make sure that space isn’t needed. If the device holds a filesystem, you would need to resize the filesystem to use less space.
After reducing the array size you should check that the data stored in the device is still available. If the device holds a filesystem, then an ‘fsck’ of the filesystem is a minimum requirement. If there are problems the array can be made bigger again with no loss with another
—grow –size= command.
Before we start the mdadm resize operation, we’ll do a little filesystem check to ensure our data is ok:
root@stargate:~# fsck.ext4 -f /dev/md0-vg/lv-company e2fsck 1.42.12 (29-Aug-2014) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information company: 619459/13107200 files (0.3% non-contiguous), 24184088/52428800 blocks root@stargate:~# fsck.ext4 -f /dev/md1-vg/lv-archive e2fsck 1.42.12 (29-Aug-2014) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information archive: 2097049/52428800 files (0.1% non-contiguous), 131114698/209715200 blocks
All good. To ensure no data gets written to our VGs while the mdadm “grow” (shrink) process proceeds, we should now disable the VGs:
root@stargate:~# vgchange -an md0-vg 0 logical volume(s) in volume group "md0-vg" now active root@stargate:~# vgchange -an md1-vg 0 logical volume(s) in volume group "md1-vg" now active
As an interesting exercise in maximising data protection, I also decided to “fail” one drive on each array before attempting the resize. This means that should the resize not work, and fsck reports file system errors, I can recover the RAID and re-think my options.
For md0 the drives are /dev/sdb1 and /dev/sdc1. Let’s go ahead and “fail” one:
root@stargate:~# mdadm /dev/md0 -f /dev/sdb1 mdadm: set /dev/sdb1 faulty in /dev/md0
Nothing more to do now but “byte the bullet” (sorry about that). Let’s try the resize:
root@stargate:~# mdadm --grow /dev/md0 --size=220G mdadm: component size of /dev/md0 has been set to 230686720K unfreeze
Now let’s now check that filesystem:
root@stargate:~# vgchange -ay md0-vg 1 logical volume(s) in volume group "md0-vg" now active root@stargate:~# fsck.ext4 -f /dev/md0-vg/lv-company e2fsck 1.42.12 (29-Aug-2014) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information company: 619459/13107200 files (0.3% non-contiguous), 24184088/52428800 blocks
This looks encouraging. Let’s try this process on the second array:
root@stargate:~# mdadm /dev/md1 -f /dev/sdd1 mdadm: set /dev/sdd1 faulty in /dev/md1 root@stargate:~# mdadm --grow /dev/md1 --size=820G mdadm: component size of /dev/md1 has been set to 859832320K unfreeze root@stargate:~# vgchange -ay md1-vg 1 logical volume(s) in volume group "md1-vg" now active root@stargate:~# fsck.ext4 -f /dev/md1-vg/lv-archive e2fsck 1.42.12 (29-Aug-2014) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information archive: 2097049/52428800 files (0.1% non-contiguous), 131114698/209715200 blocks
Fantastic! fsck reports identical figures after our mdadm resizing.
The next step is to remove the physical drives and replace them with smaller versions.
Physical drive removal & replacement
Before physically removing two drives from the server, we need to tell mdadm that the drives are to be removed:
root@stargate:~# mdadm /dev/md0 -r /dev/sdb1 mdadm: hot removed /dev/sdb1 from /dev/md0 root@stargate:~# mdadm /dev/md1 -r /dev/sdd1 mdadm: hot removed /dev/sdd1 from /dev/md1
Now we shutdown the server and physically remove the drives (my server doesn’t support actual, physical hot-swapping of drives). Be careful how you handle hot-swapping on your machine if it purpotedly does support this – do read the system documentation.
And this is where my first problem occurred. On reboot, the mdadm arrays didn’t assemble and start. I think this may have been due to a rogue entry in /etc/fstab which dumped me into the maintenance terminal at start-up. Luckily, all this was easily resolved, and issuing this:
root@stargate:~# mdadm -A /dev/md0 /dev/sdc1 mdadm: /dev/md0 has been started with 1 drive (out of 2).
brought the mdadm RAID back to life.
Now that the drives are replaced, and the RAID is running, we quickly need to partition the new drives, which will then be ready for syncing into the RAID arrays. I used fdisk to create a GPT partitioning scheme, created a partition, then set the partition type to Linux RAID.
Now that’s done, I could add the new partition to the array and let the syncing commence, and check its progress with cat:
root@stargate:~# mdadm /dev/md0 -a /dev/sdb1 mdadm: added /dev/sdb1 root@stargate:~# cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb1[2] sdc1[3] 230686720 blocks super 1.2 [2/1] [_U] [>....................] recovery = 0.0% (32512/230686720) finish=236.3min speed=16256K/sec bitmap: 1/2 pages [4KB], 65536KB chunk
That’s promising, so here goes for the second array:
root@stargate:~# mdadm -A /dev/md1 /dev/sde1 mdadm: /dev/md1 has been started with 1 drive (out of 2). root@stargate:~# mdadm /dev/md1 -a /dev/sdd1 mdadm: added /dev/sdd1 root@stargate:~# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdd1[3] sde1[2] 859832320 blocks super 1.2 [2/1] [_U] [>....................] recovery = 0.0% (344384/859832320) finish=124.7min speed=114794K/sec bitmap: 2/7 pages [8KB], 65536KB chunk [ snip ]
Great stuff! Now the long wait, until the first round of syncing has completed.
Time to take the dog for a walk and enjoy a cup of tea aftwards (or, in reality, work on other stuff).
Wash, rinse, repeat: adding the second set of disks
After the syncing completed, I could add the second set of replacement disks to the array.
It’s the same process as before:
- citing the partitions of the drives to be replaced, set each RAID member to ‘faulty’
- then ‘hot-remove’ each drive
- physically shut down the server, remove and replace the hard disks
- boot up, partition the drives, then add them to the array:
root@stargate:~# mdadm /dev/md0 -a /dev/sdc1 mdadm: added /dev/sdc1 root@stargate:~# mdadm /dev/md1 -a /dev/sde1 mdadm: added /dev/sde1 root@stargate:~# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sde1[2] sdd1[3] 859832320 blocks super 1.2 [2/1] [U_] [>....................] recovery = 0.0% (424832/859832320) finish=168.5min speed=84966K/sec bitmap: 0/7 pages [0KB], 65536KB chunk md0 : active raid1 sdc1[3] sdb1[2] 230686720 blocks super 1.2 [2/1] [U_] [>....................] recovery = 0.7% (1683968/230686720) finish=38.5min speed=99056K/sec bitmap: 0/2 pages [0KB], 65536KB chunk unused devices: <none>
Finalising the storage size
Now that the sync is complete, it’s time to check the volumes’ filesystems one more time:
root@stargate:~# fsck.ext4 -f /dev/md0-vg/lv-company e2fsck 1.42.12 (29-Aug-2014) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information company: 619459/13107200 files (0.3% non-contiguous), 24184088/52428800 blocks root@stargate:~# fsck.ext4 -f /dev/md1-vg/lv-archive e2fsck 1.42.12 (29-Aug-2014) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information archive: 2097049/52428800 files (0.1% non-contiguous), 131114698/209715200 blocks
Great – those figures still look consistent with our previous checks. With that done, we can maximise the available storage. This is what we have:
root@stargate:~# lvs LV VG Attr LSize lv-company md0-vg -wi-a----- 200.00g lv-archive md1-vg -wi-a----- 800.00g
Now let’s get to work. First, increase the size of the mdadm RAIDs:
root@stargate:~# mdadm --grow /dev/md0 --size=max mdadm: component size of /dev/md0 has been set to 244066471K unfreeze root@stargate:~# mdadm --grow /dev/md1 --size=max mdadm: component size of /dev/md1 has been set to 976630471K unfreeze
Now let’s grow the Physical Volumes (PVs) of lvm:
root@stargate:~# pvresize /dev/md0 Physical volume "/dev/md0" changed 1 physical volume(s) resized / 0 physical volume(s) not resized root@stargate:~# pvresize /dev/md1 Physical volume "/dev/md1" changed 1 physical volume(s) resized / 0 physical volume(s) not resized
And finally, let’s grow those Logical Volumes, with a filesystem resize of course 😉 :
root@stargate:~# lvresize -l +100%FREE -r md0-vg/lv-company fsck from util-linux 2.25.2 company: clean, 619459/13107200 files, 24184088/52428800 blocks Size of logical volume md0-vg/lv-company changed from 200.00 GiB (51200 extents) to 232.76 GiB (59586 extents). Logical volume lv-company successfully resized resize2fs 1.42.12 (29-Aug-2014) Resizing the filesystem on /dev/mapper/md0--vg-lv--company to 61016064 (4k) blocks. The filesystem on /dev/mapper/md0--vg-lv--company is now 61016064 (4k) blocks long. root@stargate:~# lvresize -l +100%FREE -r md1-vg/lv-archive fsck from util-linux 2.25.2 archive: clean, 2097049/52428800 files, 131114698/209715200 blocks Size of logical volume md1-vg/lv-archive changed from 800.00 GiB (204800 extents) to 931.38 GiB (238434 extents). Logical volume lv-archive successfully resized resize2fs 1.42.12 (29-Aug-2014) Resizing the filesystem on /dev/mapper/md1--vg-lv--archive to 244156416 (4k) blocks. The filesystem on /dev/mapper/md1--vg-lv--archive is now 244156416 (4k) blocks long.
Finally, time to mount the file systems:
root@stargate:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/md0--vg-lv--company 229G 89G 130G 41% /mnt/company /dev/mapper/md1--vg-lv--archive 917G 488G 388G 56% /mnt/archive
If things go wrong…
The main worry is loss of data, so this short section is dedicated to restoring backups from tape. If you have run into trouble and need to restore from backup, restore is the command you need! Bear in mind that when using it, if you (like me, above) used a non-standard block size when dumping, you need to specify that again when restoring.
The -i option in this example would invoke an interactive session in restore, allowing you to select which directories and files to restore. Instead, by omitting that option, you would just restore the entire backup.
root@stargate:~# restore -i -b 64 -f /dev/st0
Closing thoughts
The GNU/Linux operating system provides some amazing storage management tools. LVM is a real joy to use; it makes hard stuff very simple to achieve. Sure, there’s a learning curve, but it’s one I’m forever glad I embraced.
I would genuinely be interested what equivalents to this power and flexibility exist in the Windows world. Not saying for a moment that none do; I’m sure they do. But the use-case for shrinking storage might be questionned, and rightly so. It’s uncommon, to say the least.
But this is where free, open source software shines – in “scratching that itch” that isn’t commonplace, or easy to achieve. And for my efforts, at least, I have gained 2 x 400GB drives, and 2 x 2TB drives, for my next project…
If you have had similar fun on your set up, or run into pitfalls with this type of approach, do leave a comment.
What kind of storage challenges have you experienced, that were (or weren’t) solved in Linux? Leave a link to your story, or a comment, below!
Experiencing The Life-Changing Magic of Decluttering
Marie Kondo’s bestseller, The Life Changing Magic of Tidying Up, is quite an inspiration. She takes what is, essentially, a slightly mundane activity – decluttering – and transforms it into a ritual; a rite of passage for one who wants to transcend from disorganised hoarder, to cleansed, enlightened minimalist.
Ok, perhaps that is slightly strong an example, but this is the idea. The KonMari method of tidying is quite simple: do this, then do that. Kondo hand-holds her reader through the treacherous caverns of uncertainty and out into a bright, new world. Her two basic tenets are, that one must organise (and purge) by category, not by room, and that one should also focus on what to keep, but only if it “sparks joy”.
It’s simple, but it works – and surprisingly well, too. When followed correctly, it’s an efficient way to declutter, reorganise, and reset.
Starting the Journey
Getting started is remarkably easy. KonMari dictates that purging one’s superfluous clothing is the best starting point. I’d agree. It’s interesting how you can start clinging on to item from your past, instead of looking forward to wearing something in the future!
I have since discovered what I like, what suits me, and what I feel comfortable in
It’s surprisingly easy to start in this way. It teaches very effectively to be selective and mindful about what you keep for your wardrobe, and why. The key lesson, letting go, is learned here.
Like many minimalists, I have since discovered what I like about my clothing, what I think suits me, and – perhaps most importantly – what I feel comfortable in. And, like many, my wardrobe now features a number of more plain black, grey and white items. But I don’t subscribe to monotone styling; blue is the colour I love, so I have a lot of blue in my wardrobe too, plus warmer hues.
Minimalism, to me, is far more about quantity. It doesn’t mean I need to sacrifice style.
Decluttering the House
After clothes comes the decluttering of books, paperwork, CDs/DVDs and miscellanea (random bits and pieces, called komono by Kondo). How much you have kept will determine how long this takes. But a little dedication can bring surprisingly quick results. Several hours are really all that’s needed for decluttering.
Five bin bags of clothes, and four boxes of books later, I feel well on my way to simplifying and minimising.
And what a great feeling, being unburdened.
As I proceeded with old paperwork, the voice in my mind became yet more balanced. I had to double-check on certain items, and felt freedom and empowerment to change my mind part way through – sometimes choosing to discard what I initially wanted to keep, and vice versa. This process is mentally decluttering, and (if I were spiritual), spiritually cleansing.
Decluttering the Mind
Computing, for me, is a big thing. As soon as I could record stuff on computer, I did. I have databases and documents dating from the early 1990s onwards. Sadly, the file formats used for those documents are not widely supported any more.
This creates a problem: Do I keep those old documents, which probably serve no purpose to me now? If not, why did I keep them for so long? Or, do I go to the trouble of getting an older computing platform to convert them from, into a more modern, or at least less-encumbered, file format? Do I still need them? Will I ever? Or do I just archive them all off and save myself the time and effort? (Do you see how all this digital clutter is causing angst and complexity? Why am I thinking about this stuff…? etc.)
These questions have haunted me for quite some time, but eventually I came to realise that while it would be nice to have access to all my data, in reality, I don’t need it. And having access to it would increase the amount of digital clutter that I don’t need to be concerned with.
Am I Swapping Physical Hoarding for Digital Hoarding?
Although previously I wanted to convert all my old files into modern formats, I am beginning to realise that there really is little benefit to doing this. Up to roughly 1999, I would have used my Amiga computer for word processing and other work (e.g. editing images). When I got a PC in 2000, and installed SuSE Linux 7.2, I started using StarOffice (the precursor to OpenOffice/LibreOffice) instead. StarOffice files are still supported in modern versions of LibreOffice, much to my delight.
But for pre-2000 data, I must accept that without significant time, energy and devotion, I cannot easily liberate my files. This means all my university essays, college assignments, personal notes, documents, databases, images and other data are now part of that fabric of digital cruft. The best I can do is securely archive them off, hoping that one day an easy conversion solution will be available.
Yet, in some ways, this is for the best. What good does reading my own essay on Chaucer do, for me, in 2017? I am too busy with other things that are contemporary and relevant, right now, to indulge in that. Digital decluttering is as important as physical decluttering.
Closing Thoughts
I have observed that, during this decluttering process so far, I have had a tendency to hang on to things because of their value to me in the past. My mindset in the present has been influenced by past events, of course, but this has also lured me into some complacency with regard to my beliefs and philosophy on life.
When we start becoming defined by the things we have, instead of the things we do, there is little value in “having” those things whatsoever.
I am looking forward to doing more, and having less.
Have you had a similar experience with de-cluttering? Please comment – I’d love to hear about it!
Optimising fluxbox for multiple monitors and power events (ACPI)
[ this article is an incomplete draft, published for posterity ]
If you want to learn more about the GNU / UNIX operating system, and how Linux interacts with it, using a minimal installation of GNU/Linux will help. It is harder work than installing and using GNOME 3 or KDE, but the benefits soon outweigh the costs.
Preface: Migrating to a leaner window manager
This article was created on fluxbox, but can probably be applied to any minimalist window manager for GNU/Linux. My current operating system is Devuan, a fork of Debian.
If you are coming from Linux and have used XFCE, GNOME or KDE, or if you use macOS or Windows, prepare to invest some time in learning a new, yet more basic way of doing things. Many people will claim that manually doing things in a terminal window is “old fashioned” or slow. Actually, the more cloud-based and cloud-focused the world becomes, the more all of our programmatic and systematic workflows will rely on Linux. Having a reasonable understanding of the GNU operating system software can only be an advantage for people these days.
But I digress. I have written to some length about my love/hate relationship with GNOME 3. Many of the design decisions of GNOME 3 are admirable but, in implementation, some of its features can become burdensome. Using fluxbox, there is enough of a window manager for general productivity, but no more. fluxbox is fast, yet it is so minimal that there is/are:
- No native GUI tools to adjust its settings
- No way of handling multiple monitors
- No built in sound management
- No native network management
- No concept of power management
- No icons on the “desktop”, and in fact, it’s not really a “desktop” at all – just a screen
- A menu! Yes, right-click on the desktop to access a menu and launch your programs! 😉
- Workspaces. Yep, that bastion of GNU/Linux productivity that us “open sorcerers” all enjoyed for years, before Windows 10 and Mac OS X (Lion) copied on…
- Settings. If you’re happy editing a text file, that is. But if there’s one thing most people know how to do, that’s edit text files.
So, if none of the above phases you, then either you already use fluxbox, or you’re planning to and have now realised that this article is not about installing it for you! Ah no… if you want some good guides to fluxbox, check out fluxbox.org, Arch’s fluxbox page or Debian’s fluxbox page.
Configuring easier multi-monitor support
Being such a minimalistic “desktop”, fluxbox is not built to handle multiple monitors. In GNU/Linux, a popular tool to handle this task is xrandr.
xrandr is handy. It provides descriptive text output that can be used fairly easily as logical input in a script.
Here’s an example of xrandr on my dual display set-up:
# xrandr Screen 0: minimum 320 x 200, current 1920 x 1980, maximum 8192 x 8192 LVDS1 connected 1600x900+0+1080 (normal left inverted right x axis y axis) 309mm x 174mm 1600x900 60.01*+ 40.00 1440x900 59.89 1360x768 59.80 59.96 1152x864 60.00 1024x768 60.00 800x600 60.32 56.25 640x480 59.94 VGA1 disconnected (normal left inverted right x axis y axis) HDMI1 disconnected (normal left inverted right x axis y axis) DP1 disconnected (normal left inverted right x axis y axis) HDMI2 disconnected (normal left inverted right x axis y axis) HDMI3 connected 1920x1080+0+0 (normal left inverted right x axis y axis) 478mm x 269mm 1920x1080 60.00*+ 1680x1050 59.88 1280x1024 75.02 1440x900 74.98 59.90 1280x960 60.00 1280x800 59.91 1152x864 75.00 1280x720 59.97 1152x720 59.97 1024x768 75.08 70.07 60.00 832x624 74.55 800x600 72.19 75.00 60.32 56.25 640x480 75.00 72.81 66.67 60.00 720x400 70.08 DP2 disconnected (normal left inverted right x axis y axis) DP3 disconnected (normal left inverted right x axis y axis)
My laptop’s display is identified as LVDS1, and my external monitor is HDMI3, despite that I connect via DVI. This output was generated with my laptop in a docking station, so without this it may report a DVI connection as HDMI1 or HDMI2. The T420 also has a DisplayPort++ interface, which would appear to be one of DP{1-3}, and a VGA output too.
The sections we’re interested in here are:
LVDS1 connected 1600x900+0+1080 (normal left inverted right x axis y axis) 309mm x 174mm 1600x900 60.01*+ 40.00 [...] HDMI3 connected 1920x1080+0+0 (normal left inverted right x axis y axis) 478mm x 269mm 1920x1080 60.00*+
Two things of note: Firstly, when a monitor is connected to a display interface, xrandr reports this as “connected”. Otherwise, it’s “disconnected”. Secondly, a monitor may be connected but may not be active. How do we tell this? Well, the resolution line displays an asterisk if the display is active, and omits an asterisk if not. Working on the basis that the xrandr output will always list resolutions from maximum to minimum, and that we would want any monitor to run at best (native/highest) resolution, we can assume that it’s ok to test for the presence of this asterisk in the line that follows the main display line.
(UPDATE: 8 Dec 2016)
Next steps
Since drafting this article in August, my computer and computing needs have changed drastically over the past few months. Despite a happy 16+ year relationship with Linux on the desktop (YMMV, BTW, but for me every year for me was the “Year of LOTD“), my working and personal computing needs came to an impasse which could only be resolved by moving over to a Mac. My feelings and initial impressions of Mac usage are still true; for a better desktop, get GNOME – even if there have been several annoying problems.
Coming back to configuring xrandr, I’m afraid I never completed this exercise and instead opted for a quick and dirty logic script that determined which monitors were connected. Because monitor positions would rarely change, I hard-coded the positional relationship into the script. The script is will be below (when I’ve found it).
As I continue on with the Mac, I will dump more of the old Linux-y stuff into my blog, to use mainly as a reference for myself should I every have the pleasure of going back there one day.
(UPDATE: 14 Feb 2017)
After 10 months of trial and error, I am finally giving on up the Mac as a means to do work. I’m faster and happier on GNU+Linux, so that’s where’s I’m headed. Again. Happy times! 😀
I’ll still post the script when I find it.
Why a viral video of a cyclist’s revenge on catcallers may not be all it seems
Why a viral video of a cyclist’s revenge on catcallers may not be all it seems. http://google.com/newsstand/s/CBIwoL3DsDQ
UK Snoopers’ Charter gagging order drafted for London Internet Exchange directors
UK Snoopers’ Charter gagging order drafted for London Internet Exchange directors.
http://www.theregister.co.uk/2017/02/17/linx_snoopers_charger_gagging_order/
Sinking to new lows.
British Muslim teacher denied entry to US on school trip
British Muslim teacher denied entry to US on school trip
https://www.theguardian.com/us-news/2017/feb/20/british-muslim-teacher-denied-entry-to-us-on-school-trip
20 mins to spare. Decluttering my blog :)
I am ahead of schedule! With a little spare time I wasn’t anticipating, I am decluttering my blog.
I have found a number of posts that have been orphaned (having no category set) plus a whole load of drafts that should have been published and – for some reason – never were.
Commence Operation: Rapid Cleanse!
7 year-old Xeon workstation to replace my 10 month old Macbook Pro
I’ve ordered a machine to replace my Macbook Pro in the office: Dell Precision T3500 Xeon W3540 2.66GHz w/12GB 🙂
Great performance at 1/10th the cost?! What the Dell?!
I have been suffering as a would-be Mac user for the best part of 10 months now, on and off. It’s been a painful experience, physically and mentally. I was only going to post a short “microblog” post and be done with this topic, but I felt the need to expand upon my decision to do this.
Perhaps it will help dissuade potential future purchasers of Apple‘s overpriced, underwhelming and non-expandable machines. I hope it does, as one of the worst problems we create for ourselves in the 21st century is planned obsolescence – something, arguably, which Apple is guilty of.
Background
In my day job as managing director (CEO) of a UK web development & cloud hosting business, I – predictably – develop websites and administer servers. I’m the kind of guy who likes to keep his hands dirty, and my skills up.
Like many other people running a small business, my daily activities can vary rapidly. A computer which is good at switching quickly is a boon. Actually, it’s a frikkin’ necessity. Yet my core activity – PHP & JavaScript development, rely on a few basic things.
Very basic things, in fact.
Very Basic Things I continue to rely upon, to get work done:
- A keyboard with sufficient key travel, tactile feedback;
- A keyboard that broadly adheres to the standard PC 105-key layout (with or without a numeric keypad). This means:
- Not putting CTRL (Control) in a stupid place.
- Not putting ALT (Option) in an equally stupid place.
- Not having a ⌘ (“Command”) key full-stop. It’s a redundant modifier.
- Having an operating system that gets out of my way.
- Having a computer fast enough to run an operating system that gets out of my way.
- Seeing the SMART status of connected drives.
- Confidence in the device’s security.
- Confidence in its ability to stay cool when working hard for long periods.
- A system-native text editor that doesn’t refuse to edit the files I tell it to!
- Expandability.
For me, the Macbook Pro fails in all of the above.
Appeasing Mac fans & celebrating the good stuff
In April 2016, I bought this “Early 2015” Macbook Pro. It has a Core i5 5257U processor, 8GB RAM and 256GB PCIe SSD. When I mentioned to fellow designers I bought this, it was met with a knowing smile and the instant acknowledgement, “ahh wow, the SSD in those machines makes them so fast!”. I also, regretfully, bought a 27″ Thunderbolt display. The total cost of these two: a few pence short of £2,100. Two-thousand, one-hundred pounds for an average-spec 2015 laptop and 27-inch QHD monitor.
Fast is something I have never, ever considered a Mac to be, and especially this MBP. It booted quick, sure, but in general use… nah. Really, no. But I’m not in the habit of upsetting people, so more often than not I’d reply with some kind of non-opinionated remark like, “yeah? Right… I look forward to seeing that”. I’d argue, though, that the apparent lack of speed is much more to do with the operating system than the hardware.
This isn’t an Apple-bashing post. It’s just an expression of my preference. Yet there are things I really do like about the MBP:
- Ambient light-sensitive backlit keyboard – very classy
- A 3:2 ratio screen. Apple has the right idea here, and the rest of the world is stupid for putting widescreen displays in productivity laptops. Stupid. Well done Apple.
- Build quality is really excellent. If you like computers because they can be built well, I guess you may already have a Mac. 😉
- Key spacing & travel. You’re probably thinking, “but you just said…”. More on this in a sec.
- The port selection, while not excellent, still rocks more than on a MBP 2016 (like, duh!)
- The 13″ retina display
- The laptop’s general weight, shape, size and physical feel. It’s solid, if a little cold to the touch sometimes.
Talking KJeyboards…
I am typiubg this post on Apple’s “Magic Keyboard 2”. This section, including heading, is intentionally left with all the typos in as I make them. Why? Because the MAgic Ketword 2 is uterly crap compared to the keyvoard on the MBP itself. It pales in comparison in terms of typing experience. I would strongly recommend against anyone buying it, unless it’s vital to you to have a mininalist desk you can take photos of and swoon over all day. I spend hours of wasted time correcting typos that occur as a direct resylt of using this keyvoard.
By comparison, I was really quite glad how usable the keyboard on the MBO really is. ITs typing experience, much to my genuine surprisem ws excellent. The key travel is good abd the spacing between keys works really well. Although chiclet in style, with slightly rteduced key sizes compared to, say, an old school LEnobo Thinkpad (like my old T420), it’s so much more intuitive to use than the Magix Keyboard 2 that I shall no longer labvout the point and just move on.
The Problem with using a Mac: Mac OS / OS X / macos
macos is stupid and has been out-developed by GNU+Linux and the GNOME free software project. Strong statement, huh? Here’s a few reasons why.
- macos requires two keys for Mission Control and Launchpad. You cannot view open windows and search for an application in the same mode. In contrast, GNOME provides an overview by pressing the Super (Windows) key to see open windows, and accepts text search for launching a new app immediately.
- macos doesn’t support writing to NTFS partitions. Or writing to any Extended File System (EXT2,3,4), or other UNIX-based file systems.
- macos’ Finder doesn’t handle SFTP connections to remote servers.
- macos Finder supports the file operation ‘Move’ across file systems only through the undocumented keystroke, Shift-Command-V. Why is this undocumented (or at least so hard to find in the documentation)?!
- macos doesn’t do workspaces / virtual desktops as well as GNOME. No other OS does. GNOME uses the extra horizontal width to manage a vertical list of workspaces. It’s totally logical and fluid in use, if unconventional. But then, one has to “Think Different” to get on with unconventional.
- macos doesn’t open an application in the workspace in which it was launched. It seems to “remember” the last-used workspace in which the application was opened, which is pretty stupid when a second display is connected.
- macos doesn’t support focus under the pointer. When you move the pointer over another window, the previous window is still active. Clicking, say on a button on the inactive window, first activates the window. You then have to click on the button again in order to perform the expected action. Again, serious inefficiencies when done multiple times per day.
- Some macos keyboard shortcuts, relying on Cmd, really suck. Here’s an example:
Copying (Cmd-C) on a Mac. Note the awkward, cramp-inducing hand-crunch. 
Copying (Ctrl-C) on a PC. Possible without contortions. - Like other proprietary operating systems, macos includes features that are not wanted (Siri?! Siri-ously..?) or installed as standard (i.e. bloatware) that have no place on a business machine, Garageband being one example.
- The list could go on, and on, and on… [ EDIT 15 Feb 2017 ] and it will!
- Open a Finder window and the icons are not automatically sorted. There is no general sorting setting, so each folder must have a “Arrange by” setting applied.
- Copy a file from one Finder window and Paste into another. The new file doesn’t appear in the destination Finder window. That’s ok, just refresh the window’s contents…. except you can’t refresh a Finder window’s contents (amazing design decision there)! And why does the file not even appear in the folder you’ve just pasted it into?!
But the most important thing is that GNU+Linux and GNOME (or really any other free software desktop environment) is so much better. At least for someone like me, working with remote servers, or SSH sessions in a terminal, or doing lots of text editing.
What’s in a saying?
Here is a phrase you may have heard somewhen:
- Choose an occupation you love, and you will never work a day in your life
I believe this is true. I love my occupation and I am so privileged that people pay me to do it. When I get into the office, I cherish that feeling of biting off more than I can possibly chew, and working the problem towards a solution.
In the business, we make every effort to deliver the highest quality at the lowest possible cost. However, in web design, development and hosting, there are quite a number of significant costs to meet while trying to keep the end price reasonable. One such cost is test equipment.
Another cost is time; a hidden cost if, as a developer, you are always fighting your equipment in order to achieve a comfortable, efficient workflow. Using a Mac, while semi-enjoyable, also taught me just how efficient I had become using GNU+Linux to deliver results to clients. I can’t imagine a more fluid workflow than Emacs, Chrome and GNOME.
Looks are nothing
So, to the new (old) machine, which will be with me tomorrow. For the enormous sum of £179.99 + VAT and delivery (£9.99), I am getting:
- T3500 Workstation
- Windows 10 Pro 64-bit (this will be kept on the HDD for testing purposes)
- Intel Xeon W3520 2.66GHz (4 Cores / 8 Threads)
- 12GB RAM
- 500GB SATA drive
- DVD-RW
- 512MB NVIDIA Quadro FX 580
There are a few discussions online about the merits of this workstation, and I’m glad I opted for one instead of a new laptop to supplant the MBP. The Xeon 3520 processor is not new by any stretch of the imagination. It’s 8 years old. But it’s still capable enough by far and comparable to a core i7 920; a processor we still have in use in a server at Warp.
But let’s focus instead on someone else’s video, which is a nice way to tail off…