#StackBounty: #backup #disk-image #ext4 #rescue-disk #sparseimage How to create sparse, bootable Linux backup images?

Bounty: 50

I’ve set up a new Linux workstation and would like to create a backup image in order to save time in case something breaks. This is the situation:

  • Workstation: 512 GB SSD, 12 GB used, ext4 file system, may contain bad blocks
  • Backup drive: 64 GB USB flash drive

The backup image should be bootable for the added flexibility of being able to carry around and start the workstation system from any compatible host computer. All programs, configurations, network settings (proxies, VPNs) etc. should be preserved.

As I understand, a sparse, file-based image will be necessary, since the backup drive (and possibly also the replacement drive) is smaller than the source drive.

What is a good way to do this? AFAIK, dd just copies bit by bit and ddrescue can only sparsify blocks of zeros into metadata, but not unused space filled with random data. Some disk cleanup tools offer to fill unused space with zeros, but this usually takes many hours and I’m not sure if this is good practice for collapsing data.

I need a reliable, free, and easy solution as I don’t have any expertise on file systems, partition structures, boot sectors etc.


Get this bounty!!!

#StackBounty: #files #symlink #ext4 #delete #xattr How to make a symlink read only (`chattr +i /location/symlink`)?

Bounty: 50

How can we lock a symlink so it cannot be deleted?

With a normal file/directory chattr +i /file/location can achieve this but doing so with a symlink we get chattr: Operation not supported while reading flags on my-file.

There is a similar question, How to set chattr +i for my /etc/resolv.conf?, but without a solution that could be applied here.


Get this bounty!!!

#StackBounty: #linux #rsync #hardware-raid #ext4 Hard drive with many hard links 'fills' almost overnight

Bounty: 50

I recently installed a cheap 2tb hard drive on a server for backing up files which are also backed up elsewhere. It’s basically an overflow drive. The other drives on the server are configured as 1TB in a Raid 6 array. This single drive I configured as Raid 0 just for convenience.

In essence I was moving about 700GB of data from the Raid 6 drive to the Raid 0 drive because the Raid 6 drive was almost full. So … 2 TB should be way more than enough, right?

The data is in the form of data rsynced from a remote server, with 6 days of incremental backups handled in a standard ‘hard link’ manner to ensure I am only storing/transferring changes, and not backing up the entire data every day.

However the behaviour I am seeing is that data that was stored at around 700GB on the Raid 6 drives quickly balloons to almost fill the 2TB drive, as if I were not using hard links.

Yesterday I deleted about 300GB of data which is no longer needed, and overnight the storage was back to 97% full.

Does anybody know what’s going on? Is the drive really ‘full’, or is it just bad calculation of hard linking?

All drives are formatted as Ext4.

** Edit **

Details of backup process:

Each day a cronjob copies backup0 to backup1 using cp -al backup0 backup1. Previous backups are moved by mv backup1 backup2, etc, prior to an rsync taking place.

backup5 is deleted each day. After that happens, a remote server rsyncs to backup0 (thus updating only changed files). Thus, 5 days of incremental backups. This is basically how software like ‘backintime’ works too.

** Second Edit **

I just deleted backup3 to backup 5 and it freed up about 2 thirds of the drive. So, the problem seems to be how storage is being calculated. (I use df -h to monitor storage).

The question remains … will the drive be considered ‘full’ even though there should be ample space, when it reaches “100%”.


Get this bounty!!!

#StackBounty: #linux #rsync #hardware-raid #ext4 Hard drive with many hard links 'fills' almost overnight

Bounty: 50

I recently installed a cheap 2tb hard drive on a server for backing up files which are also backed up elsewhere. It’s basically an overflow drive. The other drives on the server are configured as 1TB in a Raid 6 array. This single drive I configured as Raid 0 just for convenience.

In essence I was moving about 700GB of data from the Raid 6 drive to the Raid 0 drive because the Raid 6 drive was almost full. So … 2 TB should be way more than enough, right?

The data is in the form of data rsynced from a remote server, with 6 days of incremental backups handled in a standard ‘hard link’ manner to ensure I am only storing/transferring changes, and not backing up the entire data every day.

However the behaviour I am seeing is that data that was stored at around 700GB on the Raid 6 drives quickly balloons to almost fill the 2TB drive, as if I were not using hard links.

Yesterday I deleted about 300GB of data which is no longer needed, and overnight the storage was back to 97% full.

Does anybody know what’s going on? Is the drive really ‘full’, or is it just bad calculation of hard linking?

All drives are formatted as Ext4.

** Edit **

Details of backup process:

Each day a cronjob copies backup0 to backup1 using cp -al backup0 backup1. Previous backups are moved by mv backup1 backup2, etc, prior to an rsync taking place.

backup5 is deleted each day. After that happens, a remote server rsyncs to backup0 (thus updating only changed files). Thus, 5 days of incremental backups. This is basically how software like ‘backintime’ works too.

** Second Edit **

I just deleted backup3 to backup 5 and it freed up about 2 thirds of the drive. So, the problem seems to be how storage is being calculated. (I use df -h to monitor storage).

The question remains … will the drive be considered ‘full’ even though there should be ample space, when it reaches “100%”.


Get this bounty!!!

#StackBounty: #linux #rsync #hardware-raid #ext4 Hard drive with many hard links 'fills' almost overnight

Bounty: 50

I recently installed a cheap 2tb hard drive on a server for backing up files which are also backed up elsewhere. It’s basically an overflow drive. The other drives on the server are configured as 1TB in a Raid 6 array. This single drive I configured as Raid 0 just for convenience.

In essence I was moving about 700GB of data from the Raid 6 drive to the Raid 0 drive because the Raid 6 drive was almost full. So … 2 TB should be way more than enough, right?

The data is in the form of data rsynced from a remote server, with 6 days of incremental backups handled in a standard ‘hard link’ manner to ensure I am only storing/transferring changes, and not backing up the entire data every day.

However the behaviour I am seeing is that data that was stored at around 700GB on the Raid 6 drives quickly balloons to almost fill the 2TB drive, as if I were not using hard links.

Yesterday I deleted about 300GB of data which is no longer needed, and overnight the storage was back to 97% full.

Does anybody know what’s going on? Is the drive really ‘full’, or is it just bad calculation of hard linking?

All drives are formatted as Ext4.

** Edit **

Details of backup process:

Each day a cronjob copies backup0 to backup1 using cp -al backup0 backup1. Previous backups are moved by mv backup1 backup2, etc, prior to an rsync taking place.

backup5 is deleted each day. After that happens, a remote server rsyncs to backup0 (thus updating only changed files). Thus, 5 days of incremental backups. This is basically how software like ‘backintime’ works too.

** Second Edit **

I just deleted backup3 to backup 5 and it freed up about 2 thirds of the drive. So, the problem seems to be how storage is being calculated. (I use df -h to monitor storage).

The question remains … will the drive be considered ‘full’ even though there should be ample space, when it reaches “100%”.


Get this bounty!!!

#StackBounty: #linux #rsync #hardware-raid #ext4 Hard drive with many hard links 'fills' almost overnight

Bounty: 50

I recently installed a cheap 2tb hard drive on a server for backing up files which are also backed up elsewhere. It’s basically an overflow drive. The other drives on the server are configured as 1TB in a Raid 6 array. This single drive I configured as Raid 0 just for convenience.

In essence I was moving about 700GB of data from the Raid 6 drive to the Raid 0 drive because the Raid 6 drive was almost full. So … 2 TB should be way more than enough, right?

The data is in the form of data rsynced from a remote server, with 6 days of incremental backups handled in a standard ‘hard link’ manner to ensure I am only storing/transferring changes, and not backing up the entire data every day.

However the behaviour I am seeing is that data that was stored at around 700GB on the Raid 6 drives quickly balloons to almost fill the 2TB drive, as if I were not using hard links.

Yesterday I deleted about 300GB of data which is no longer needed, and overnight the storage was back to 97% full.

Does anybody know what’s going on? Is the drive really ‘full’, or is it just bad calculation of hard linking?

All drives are formatted as Ext4.

** Edit **

Details of backup process:

Each day a cronjob copies backup0 to backup1 using cp -al backup0 backup1. Previous backups are moved by mv backup1 backup2, etc, prior to an rsync taking place.

backup5 is deleted each day. After that happens, a remote server rsyncs to backup0 (thus updating only changed files). Thus, 5 days of incremental backups. This is basically how software like ‘backintime’ works too.

** Second Edit **

I just deleted backup3 to backup 5 and it freed up about 2 thirds of the drive. So, the problem seems to be how storage is being calculated. (I use df -h to monitor storage).

The question remains … will the drive be considered ‘full’ even though there should be ample space, when it reaches “100%”.


Get this bounty!!!

#StackBounty: #linux #rsync #hardware-raid #ext4 Hard drive with many hard links 'fills' almost overnight

Bounty: 50

I recently installed a cheap 2tb hard drive on a server for backing up files which are also backed up elsewhere. It’s basically an overflow drive. The other drives on the server are configured as 1TB in a Raid 6 array. This single drive I configured as Raid 0 just for convenience.

In essence I was moving about 700GB of data from the Raid 6 drive to the Raid 0 drive because the Raid 6 drive was almost full. So … 2 TB should be way more than enough, right?

The data is in the form of data rsynced from a remote server, with 6 days of incremental backups handled in a standard ‘hard link’ manner to ensure I am only storing/transferring changes, and not backing up the entire data every day.

However the behaviour I am seeing is that data that was stored at around 700GB on the Raid 6 drives quickly balloons to almost fill the 2TB drive, as if I were not using hard links.

Yesterday I deleted about 300GB of data which is no longer needed, and overnight the storage was back to 97% full.

Does anybody know what’s going on? Is the drive really ‘full’, or is it just bad calculation of hard linking?

All drives are formatted as Ext4.

** Edit **

Details of backup process:

Each day a cronjob copies backup0 to backup1 using cp -al backup0 backup1. Previous backups are moved by mv backup1 backup2, etc, prior to an rsync taking place.

backup5 is deleted each day. After that happens, a remote server rsyncs to backup0 (thus updating only changed files). Thus, 5 days of incremental backups. This is basically how software like ‘backintime’ works too.

** Second Edit **

I just deleted backup3 to backup 5 and it freed up about 2 thirds of the drive. So, the problem seems to be how storage is being calculated. (I use df -h to monitor storage).

The question remains … will the drive be considered ‘full’ even though there should be ample space, when it reaches “100%”.


Get this bounty!!!

#StackBounty: #linux #rsync #hardware-raid #ext4 Hard drive with many hard links 'fills' almost overnight

Bounty: 50

I recently installed a cheap 2tb hard drive on a server for backing up files which are also backed up elsewhere. It’s basically an overflow drive. The other drives on the server are configured as 1TB in a Raid 6 array. This single drive I configured as Raid 0 just for convenience.

In essence I was moving about 700GB of data from the Raid 6 drive to the Raid 0 drive because the Raid 6 drive was almost full. So … 2 TB should be way more than enough, right?

The data is in the form of data rsynced from a remote server, with 6 days of incremental backups handled in a standard ‘hard link’ manner to ensure I am only storing/transferring changes, and not backing up the entire data every day.

However the behaviour I am seeing is that data that was stored at around 700GB on the Raid 6 drives quickly balloons to almost fill the 2TB drive, as if I were not using hard links.

Yesterday I deleted about 300GB of data which is no longer needed, and overnight the storage was back to 97% full.

Does anybody know what’s going on? Is the drive really ‘full’, or is it just bad calculation of hard linking?

All drives are formatted as Ext4.

** Edit **

Details of backup process:

Each day a cronjob copies backup0 to backup1 using cp -al backup0 backup1. Previous backups are moved by mv backup1 backup2, etc, prior to an rsync taking place.

backup5 is deleted each day. After that happens, a remote server rsyncs to backup0 (thus updating only changed files). Thus, 5 days of incremental backups. This is basically how software like ‘backintime’ works too.

** Second Edit **

I just deleted backup3 to backup 5 and it freed up about 2 thirds of the drive. So, the problem seems to be how storage is being calculated. (I use df -h to monitor storage).

The question remains … will the drive be considered ‘full’ even though there should be ample space, when it reaches “100%”.


Get this bounty!!!

#StackBounty: #linux #rsync #hardware-raid #ext4 Hard drive with many hard links 'fills' almost overnight

Bounty: 50

I recently installed a cheap 2tb hard drive on a server for backing up files which are also backed up elsewhere. It’s basically an overflow drive. The other drives on the server are configured as 1TB in a Raid 6 array. This single drive I configured as Raid 0 just for convenience.

In essence I was moving about 700GB of data from the Raid 6 drive to the Raid 0 drive because the Raid 6 drive was almost full. So … 2 TB should be way more than enough, right?

The data is in the form of data rsynced from a remote server, with 6 days of incremental backups handled in a standard ‘hard link’ manner to ensure I am only storing/transferring changes, and not backing up the entire data every day.

However the behaviour I am seeing is that data that was stored at around 700GB on the Raid 6 drives quickly balloons to almost fill the 2TB drive, as if I were not using hard links.

Yesterday I deleted about 300GB of data which is no longer needed, and overnight the storage was back to 97% full.

Does anybody know what’s going on? Is the drive really ‘full’, or is it just bad calculation of hard linking?

All drives are formatted as Ext4.

** Edit **

Details of backup process:

Each day a cronjob copies backup0 to backup1 using cp -al backup0 backup1. Previous backups are moved by mv backup1 backup2, etc, prior to an rsync taking place.

backup5 is deleted each day. After that happens, a remote server rsyncs to backup0 (thus updating only changed files). Thus, 5 days of incremental backups. This is basically how software like ‘backintime’ works too.

** Second Edit **

I just deleted backup3 to backup 5 and it freed up about 2 thirds of the drive. So, the problem seems to be how storage is being calculated. (I use df -h to monitor storage).

The question remains … will the drive be considered ‘full’ even though there should be ample space, when it reaches “100%”.


Get this bounty!!!

#StackBounty: #ext4 #ssd #cryptsetup #trim #fstrim independently verify that TRIM indeed works on SSD

Bounty: 200

I have a LUKS partition /dev/sda1 which I luksOpen with --allow-discards:

cryptsetup --allow-discards luksOpen /dev/sda1 root

I then mount the ext4 filesystem with discard option:

grep /dev/mapper/root /proc/mounts
/dev/mapper/root / ext4 ro,relatime,block_validity,discard,delalloc,barrier,user_xattr,acl 0 0

I then trim free space on the mounted partition:

fstrim -v /

with df, I see / has 80% free space.
That means that on /dev/sda1, 80% of the disk are binary zeros.

If I clone the image with cat

cat /dev/sda1 > sda1.img

and compress the image with xz, I would expect all the zeros on the disk to be compressed. Since the 20% of the data on the disk is encrypted, it should look like random and be uncompressible. Therefore, the xz-compressed image should be aprox. 20% of the raw size.

However, the resulting xz-compressed image is approximately same size as the raw original.

Is my reasoning correct?

Why does not my theory translate into practice ?


Get this bounty!!!