Tag: hard-drive

Bounty: 100

So far, I have a semi-automated backup approach for the whole system installed in different LVs, using LVM (CoW) snapshots (which can be created upon boot to avoid data corruption). That is, if I feel like I need to create a backup, I will run a script to mount those snapshot LVs (in the same way the live system is mounted) to some location in read-only mode and then run another script to perform a compressed backup of that location, what finally produces a compressed backup of the whole system as a single archive file. Neat…

Now, I have a new use case for which I’m trying to find a good solution. Say, I got an external HDD, which purpose is to store stuff. Well, you know, it is encrypted, also has its own VG and LVs on it, each of which is dedicated for storing different types of data, e.g. pictures, videos, documents, and even those system backup archives. This gives flexibility of choosing file systems depending on a type of data and all the cool features of LVM. That is from time to time, I plug in this beast and copy more stuff to either of these LVs depending on what I want to store. Next, I have yet another external HDD, which I would like (in the ideal world) to act as a mirror of the first external HDD. Furthermore, ideally that mirroring should happen automagically when both devices are plugged in and I explicitly copy stuff only onto the first one (i.e. something like CoW, maybe we should call it incremental mirroring, though I’m not sure it’s possible). The situations when, for example, the first device is connected and being written, while the second one is not, should also be handled. That is when the second device is connected next time, it should not only mirror the data that is being written to the first one this time, but also the one that was missed last time. I guess this is why I refer to mirroring, which is probably not possible to perform incrementally in such situations and a complete clone overwrite is required.

Up until now, I considered the following solutions:

1. LVM snapshots seem useless here:
   • they are CoW, i.e. they don’t actually maintain a complete mirror on a snapshot LV (unless all of that data changes after writing to the first device);
   • they would require to span one VG not only over separate and independent PVs but also over separate and, technically speaking, independent physical devices (I don’t like such coupling and would like to avoid it).
2. sector cloning via dd:
   • looks solid in terms of producing the exact clone on the second device;
   • according to LVM and cloning HDs, is dangerous because LVM configurations are obviously duplicated in this case and, as a result, may result in data corruption if some renaming black magic is not done (I don’t want to script that renaming unless we will not find another solution in this thread as it looks fragile to me);
   • smells like an overkill to overwrite the whole second device every time I copy a couple of files into one of the LVs on the first device, just to maintain a mirror;
   • not really automated as it does not detect what was changed where and copy it transparently just based on the fact that both are plugged in and the first one was written (this is what I call “on demand”).
3. rsync of files between corresponding LVs of the two devices:
   • the good thing is that it will synchronize only the changes and will satisfy even the case when we missed to plug the second device once or twice (and that’s a big plus);
   • looks convoluted;
   • requires scripting for simultaneous mounting of corresponding LVs;
   • not automated (has to be triggered manually), thus error-prone;
   • requires manual recreation of the same partitioning scheme and LVM on the second device based on the first device (yes, this could be automated, but still).
4. LVM RAID1? I have no idea how reliable and/or possible it is to set up with two plug-and-play devices which may or may not be plugged in and/or mounted simultaneously, and, thus, how it will compare to the above options. Any experiences shared are much appreciated.
5. Some other solution, maybe utilizing vgimportclone?

I appreciate any inputs on this and especially already tested/used solutions which either self-designed and customized by you or existing adopted tools to manage this scenario. The goal is to make it as simple/least error-prone/automated as possible. Thank you!

Get this bounty!!!