#StackBounty: #backup #cloud Desktop software for cloud backup in buckets?

Bounty: 50

I already do backup on external HDD and I’m planing to do cloud backup as well.

As I could see, for long-term storage, not readly available (i.e., you have to wait hours to begin the recovery), some storage services are cheaper (AWS, BackBlaze, and so on).

But the storage is done in buckets, and also data transfer (upload and download) is billed.

So I need some backup tool that:

  • can store files on the cloud

  • Does incremental backups (to minimize costs of data transfer)

  • keeps control of files stored in buckets for long term storage

  • perhaps can deal with the local, external HDD backup.

Bonus for GPL or not proprietary data formats, as I need to be sure that I’ll be able to recover the files if necessary without depending on a proprietary software/format to untar/decrypt/unzip my files.


Get this bounty!!!

#StackBounty: #adb #backup #restore #android-emulator ADB backup and restore app data between device and emulator?

Bounty: 100

* = Edited on 03/25/2018

I have an application for which I’d like to transfer the app data from my physical device to an emulator on my PC. *The data consists of a single .sol file which is saved at:

datadatapackage.namefilesfile.sol  

I’m not proficient with ADB / Linux / Bash, so I’d appreciate any efforts spared to elaborate with more verbose guidance (I’m not afraid to read through essays, I’m afraid of the one-line answers). The device has to be kept stock (rootless). Here are the specs:

Device (Source):
Samsung SM-N950U Stock (T-MO)
Android 8.0
*Bootloader Version: N950USQS5CRL1
Emulator (Destination):
Runtime Environment: 6.0.6 Good
Device Model: Samsung Galaxy Note 8 Android 5.1.1 (US)

After quite a bit of digging around, I finally got ADB backup to work do something (after an OTA update, it kept producing a 1KB file with no error echoed back). I’m not 100% sure the backup is now fully functional, but it now produces a 17KB .ab file (I am using the -noapk argument, and the contents of the backup should decompress into a .sol file so it wouldn’t be a huge file or anything). Here’s the syntax I used:

adb -d backup com.example.example "-noapk -obb -shared" -f examplebackup.ab

When using ADB restore to the emulator, everything goes smoothly and no error message is produced (unlock dialogue prompt comes up so I know it’s going to the correct device). However, opening the application it acts like I’ve just opened it for the first time, so no save data was transferred over. For reference, this is the syntax I used:

adb -s 127.0.0.1:21513 restore examplebackup.ab

I know there are several similar threads, but none addressed this type of cross platform transfer that I saw, and none of the potential workarounds or solutions seemed to work (although one of them did help me figure out the 1KB backup issue). Hoping to get some help with this, and my apologies if this was addressed somewhere but I just couldn’t understand it.

Thanks in advance for any assistance!


*UPDATE

In the past weeks, I’ve pieced together more information from here and there, and I’ve decided to come back and update this post for any others who might be struggling with the same issue. At this time, I believe it will not be possible to accomplish my objective of retrieving my data, but it’s not necessarily true for others. Below, I’ll explain my attempts and the setbacks I encountered from a layman’s perspective, but if anyone more knowledgeable on the subject feels inclined to clarify anything I’m unaware of or misunderstood, it’d be greatly appreciated.

  • I have spoken with technicians at Google, Samsung, the mobile carrier, the community members from all of the above, and obviously here on Android Stack Exchange (with zero answers). All of the “official” sources where I was able to get a response from have told me it isn’t possible, and unexpectedly proceeded to push the blame onto the developer and/or each other.
  • From my understanding, in order for the ADB backup method to fully function as I want it to, I would either need root access, or the application needs to be debuggable. I have checked the app manifest and have seen that it isn’t (but backup is enabled, by the way).
    • Another method through ADB I had read was to invoke run-as, essentially “borrowing” the application’s identity and therefore inheriting its own permissions, and then pulling the data thereafter. Unfortunately, it seems this command has either been deprecated or broken since Marshmallow 6.0+ (I’m not sure if that’s universal to the OS or just Samsung, either way it doesn’t help my situation).
    • I also wasn’t sure whether a rootless BusyBox installation would make it possible to at least grant me access permissions to read the .sol file, if not copy the contents. I wasn’t able to completely comprehend the XDA thread contents, but from what I could discern from the comments it still wouldn’t have the required permissions to access datadata.. (there seems to be some mention of a potential workaround by installing BusyBox into the same directory(?) but there wasn’t any further explicit guidance or example that I could find).
  • I believe the device-to-device backup methods (such as Google Backup or Samsung Smart Switch) might possibly work, since they should have the proper permissions(?); however, as I can only restore onto a device with same or higher OS version among other factors, I have no way of verifying whether or not it works other than by deleting my local data and trying to restore it (obviously, I’d be shooting myself in the foot if it didn’t actually work).
    • As the Google Backup method states it needs to be restored onto a device of equivalent OS version or higher, one option I had thought of was restoring onto an emulator running Oreo 8.0, and from there enabling root to extract the data I needed. However, the only one I could find was the default provided through AVD from the Android SDK; an 8.0 emulator was available with Play services enabled and preloaded, but for whatever reason it shows the application is incompatible, so that was a dead-end.
    • Smart Switch does offer a backup “to external storage” option, but even though I have encryption disabled on both my device as well as the Smart Switch backup settings the backup still results in an .enc container, which I’ve not been able to find a way to decompress or extract from (see below for further on this).
  • Unfortunately, I was stupid enough to install the latest OTA update before checking, but there doesn’t seem to be a method for root at this time on the newer bootloader/firmware versions. So for the time being, I’m just stuck with no way to retrieve my own data.

Things I’d still like to find out:

  • I am inferring that invoking adb backup with the correct syntax and -noapk argument outputs a non-zero / -single kb file because it is properly backing up all the application-related data except what’s stored in datadata.. ? Or is the inclusion of the enabled backup flag in manifest supposed to make ADB backup work regardless of root? If the latter is true, what am I doing wrong?
  • I’m wondering if I can somehow create a virtual “external storage” in an emulator and use Smart Switch to restore the .enc file; the emulator I use does offer shared storage between VM and desktop, but they are not recognized in Smart Switch as mounted “external storage”, therefore leaving no way to point to the file). Otherwise, it’d be great to be able to directly access the .enc container somehow, I just don’t understand the point of letting the user choose unencrypted if it’s going to be encrypted anyway.
  • Although I have a (likely biased) feeling it shouldn’t have to come to this, I do have the means to contact the app’s developer. If I do go that route, what would be the more appropriate “means” to ask for in order to achieve the desired result? E.g., “can you please make your app debuggable?” vs “can you change the directory for this particular file to save in a non-system directory?”, etc.
  • I’m curious if there’s anything I can do via SetEdit (after elevating permissions as instructed) that might help me with this problem?


Get this bounty!!!

#StackBounty: #hard-drive #backup #raid #lvm #cloning Backup: How to mirror/clone LVM LVs (or VGs) on demand?

Bounty: 100

So far, I have a semi-automated backup approach for the whole system installed in different LVs, using LVM (CoW) snapshots (which can be created upon boot to avoid data corruption). That is, if I feel like I need to create a backup, I will run a script to mount those snapshot LVs (in the same way the live system is mounted) to some location in read-only mode and then run another script to perform a compressed backup of that location, what finally produces a compressed backup of the whole system as a single archive file. Neat…

Now, I have a new use case for which I’m trying to find a good solution. Say, I got an external HDD, which purpose is to store stuff. Well, you know, it is encrypted, also has its own VG and LVs on it, each of which is dedicated for storing different types of data, e.g. pictures, videos, documents, and even those system backup archives. This gives flexibility of choosing file systems depending on a type of data and all the cool features of LVM. That is from time to time, I plug in this beast and copy more stuff to either of these LVs depending on what I want to store. Next, I have yet another external HDD, which I would like (in the ideal world) to act as a mirror of the first external HDD. Furthermore, ideally that mirroring should happen automagically when both devices are plugged in and I explicitly copy stuff only onto the first one (i.e. something like CoW, maybe we should call it incremental mirroring, though I’m not sure it’s possible). The situations when, for example, the first device is connected and being written, while the second one is not, should also be handled. That is when the second device is connected next time, it should not only mirror the data that is being written to the first one this time, but also the one that was missed last time. I guess this is why I refer to mirroring, which is probably not possible to perform incrementally in such situations and a complete clone overwrite is required.

Up until now, I considered the following solutions:

  1. LVM snapshots seem useless here:
    • they are CoW, i.e. they don’t actually maintain a complete mirror on a snapshot LV (unless all of that data changes after writing to the first device);
    • they would require to span one VG not only over separate and independent PVs but also over separate and, technically speaking, independent physical devices (I don’t like such coupling and would like to avoid it).
  2. sector cloning via dd:
    • looks solid in terms of producing the exact clone on the second device;
    • according to LVM and cloning HDs, is dangerous because LVM configurations are obviously duplicated in this case and, as a result, may result in data corruption if some renaming black magic is not done (I don’t want to script that renaming unless we will not find another solution in this thread as it looks fragile to me);
    • smells like an overkill to overwrite the whole second device every time I copy a couple of files into one of the LVs on the first device, just to maintain a mirror;
    • not really automated as it does not detect what was changed where and copy it transparently just based on the fact that both are plugged in and the first one was written (this is what I call “on demand”).
  3. rsync of files between corresponding LVs of the two devices:
    • the good thing is that it will synchronize only the changes and will satisfy even the case when we missed to plug the second device once or twice (and that’s a big plus);
    • looks convoluted;
    • requires scripting for simultaneous mounting of corresponding LVs;
    • not automated (has to be triggered manually), thus error-prone;
    • requires manual recreation of the same partitioning scheme and LVM on the second device based on the first device (yes, this could be automated, but still).
  4. LVM RAID1? I have no idea how reliable and/or possible it is to set up with two plug-and-play devices which may or may not be plugged in and/or mounted simultaneously, and, thus, how it will compare to the above options. Any experiences shared are much appreciated.
  5. Some other solution, maybe utilizing vgimportclone?

I appreciate any inputs on this and especially already tested/used solutions which either self-designed and customized by you or existing adopted tools to manage this scenario. The goal is to make it as simple/least error-prone/automated as possible. Thank you!


Get this bounty!!!

#StackBounty: #postgresql #backup #centos #pgpool Postgres 9.6 – Taking differenctial backup

Bounty: 100

I have a few CentOS boxes running Postgres 9.6. I am planning a central server which can make local copies of all DBs on the other servers. These servers are accessible via SSH on Internet.

What is the best approach to take differential backup? I am considering below points :

  1. Central Backup Server may not be up 24/7.
  2. Internet Link may fail.
  3. Bandwidth Consumption on the DB server.

And I am considering the below approaches :

  1. Directly Rsync the Postgres data directory.
  2. Some kind of log archive, to be read by Postgres.
  3. Best of open source solution such as the pitrtool or pgpool2.

Which is the most efficient or even possible from above? Any other recommendation?


Get this bounty!!!