diff options
| -rw-r--r-- | content/posts/2023-09-02-setup-external-backup.md | 120 | ||||
| -rw-r--r-- | content/posts/2023-10-29-setup-external-backup.md | 295 |
2 files changed, 295 insertions, 120 deletions
diff --git a/content/posts/2023-09-02-setup-external-backup.md b/content/posts/2023-09-02-setup-external-backup.md deleted file mode 100644 index 6a94158..0000000 --- a/content/posts/2023-09-02-setup-external-backup.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: "Setup backup on external disk" -date: 2023-09-02 -draft: true -lang: en -categories: [ blog ] -tags: [] -translationKey: "2023-09-02-setup-external-backup" ---- - -## Backstory - -A few weeks ago (not today), I fucked up: -During irregular disk cleanup, I accidentally removed the config folder! - -Fortunately, I "backed up" the config with git, didn't I? 😌 - -Well yes, but actually no! 🥲 I did commit, but I didn't push. -The latest pushed config had been 9 months before this incident. - -On top of that, many programs generate unreadable configs, which I prefer not -to put into version control, or put sensitive information or large data into -`.config`, which shouldn't be pushed into a remote repository on a server I -don't own. For example, nheko puts authentication info there, and after this -incident I lost the session with several messages---well, matrix's irregularly -regular decryption failure could be expanded into a post on its own, so let's -not digress. - -And then there are more things to back up than just configs, say, my photos and -music. I mirror them between my devices with [syncthing], providing -redundancy, but [syncthing is not backup][syncthing-failure], -and neither is [RAID][raid-not-backup] - -[syncthing]: https://syncthing.net/ -[syncthing-failure]: https://forum.syncthing.net/t/syncthing-deleted-all-files-on-every-device/20518 -[raid-not-backup]: https://www.raidisnotabackup.com/ - -I should back them up, or I might one day be doomed to repeat the mistake. - -## Preparation - -Despite the bitterness from the partial data loss, I kinda didn't have the -spoon to do what I must, so I've been delaying it hitherto. - -### Hardware - -One of the reason I was cleaning up the disk in the first place was that it's -start to be filled up, so I bought a large hard drive to store more data and -backup. Doesn't seem too safe, I know: if the hard drive gets lost or broken, -I'll lose both the main data and the backup, but with a limited budget and -space, that's good enough for now. - -I got a Seagate One Touch 2 TB[^0] HDD. It seems to come with builtin backup -software, which doesn't support Linux of course, and I likely wouldn't use it -even if it did. A USB hub is also needed, as my laptop has a limited number of -USB ports. - -### Formatting - -Coming with the hard drive are several files, such as an `.exe` program to -initialize the backup, I suppose. I don't need these, and the disk can be -formatted ~~right away~~ after I do full-disk encryption on that. - -No partitioning is needed: while I do seek [vegan alternative][margarine], I -intended to use [btrfs][btrfs-kernel] as the file system, which can create -subvolumes, which can act as separate mounted partitions. I will just reformat -the existing partition to btrfs. - -```sh -mkfs.btrfs -L margarine /dev/sdb1 -``` - -[margarine]: https://outerheaven.club/notice/AZ8wYpKOyp2p329V5M -[btrfs-kernel]: https://www.kernel.org/doc/html/latest/filesystems/btrfs.html - -I intend to have three subvolumes, one for big data (not Big Data™) such as -movies or music, one for more personal data and another for backup: - -```sh -mount /dev/sdb1 /mnt -btrfs subvolume /mnt/data/hoard -btrfs subvolume /mnt/data/perso -btrfs subvolume /mnt/backup -``` - -Finally, I need to declare those in the nix hardware configuration file. - - -```nix -{ - (...) - fileSystems = { - "/" = { - (...) - }; - "/data/hoard" = { - device = "/dev/disk/by-uuid/[subvolume uuid]"; - fsType = "btrfs"; - options = [ "subvol=data/hoard" "compress-force=zstd" "noatime" ]; - }; - "/data/perso" = { - device = "/dev/disk/by-uuid/[subvolume uuid]"; - fsType = "btrfs"; - options = [ "subvol=data/perso" "compress-force=zstd" "noatime" ]; - }; - "/backup" = { - device = "/dev/disk/by-uuid/[subvolume uuid]"; - fsType = "btrfs"; - options = [ "subvol=backup" "compress-force=zstd" ]; - }; - }; - (...) -} -``` - -## Setting up - -[^0]: That means 1.81 [TiB] and I feel cheated sometimes. - -[TiB]: https://en.wikipedia.org/wiki/Byte#Multiple-byte_units. diff --git a/content/posts/2023-10-29-setup-external-backup.md b/content/posts/2023-10-29-setup-external-backup.md new file mode 100644 index 0000000..017fee9 --- /dev/null +++ b/content/posts/2023-10-29-setup-external-backup.md @@ -0,0 +1,295 @@ +--- +title: "Setup backup on external disk" +date: 2023-10-29 +lang: en +categories: [ blog ] +tags: [] +translationKey: "2023-10-29-setup-external-backup" +--- + +## Backstory + +A few weeks ago (not today, and by the time I published this post it's a few +months ago), I fucked up: +During irregular disk cleanup, I accidentally removed the config folder! + +Fortunately, I "backed up" the config with git, didn't I? 😌 + +Well yes, but actually no! 🥲 I did commit, but I didn't push. +The latest pushed config had been 9 months before this incident. + +On top of that, many programs generate unreadable configs, which I prefer not +to put into version control, or put sensitive information or large data into +`.config`, which shouldn't be pushed into a remote repository on a server I +don't own. For example, nheko puts authentication info there, and after this +incident I lost the session with several messages---well, matrix's irregularly +regular decryption failure could be expanded into a post on its own, so let's +not digress. + +And then there are more things to back up than just configs, say, my photos and +music. I mirror them between my devices with [syncthing], providing +redundancy, but [syncthing is not backup][syncthing-failure], +and neither is [RAID][raid-not-backup] + +[syncthing]: https://syncthing.net/ +[syncthing-failure]: https://forum.syncthing.net/t/syncthing-deleted-all-files-on-every-device/20518 +[raid-not-backup]: https://www.raidisnotabackup.com/ + +I should back them up, or I might one day be doomed to repeat the mistake. + +## Preparation + +Despite the bitterness from the partial data loss, I kinda didn't have the +spoon to do what I must, so I've been delaying it hitherto. + +### Hardware + +One of the reason I was cleaning up the disk in the first place was that it's +start to be filled up, so I bought a large hard drive to store more data and +backup. Doesn't seem too safe, I know: if the hard drive gets lost or broken, +I'll lose both the main data and the backup, but with a limited budget and +space, that's good enough for now. I am planning for some off-site backup. + +I got a Seagate One Touch 2 TB[^0] HDD. It seems to come with builtin backup +software, which doesn't support Linux of course, and I likely wouldn't use it +even if it did. A USB hub is also needed, as my laptop has a limited number of +USB ports. + +### Formatting + +Coming with the hard drive are several files, such as an `.exe` program to +initialize the backup, I suppose. I don't need these, and the disk can be +formatted ~~right away~~ after I do full-disk encryption on that. + +```sh +cryptsetup luksFormat /dev/sdb +cryptsetup open --type luks /dev/sdb extern +``` + +No partitioning is needed: while I do seek [vegan alternative][margarine], I +intended to use [btrfs][btrfs-kernel] as the file system, which can create +subvolumes, which can act as separate mounted partitions. I would just +reformat the existing partition to btrfs. + +```sh +mkfs.btrfs -L margarine /dev/mapper/extern +``` + +[margarine]: https://outerheaven.club/notice/AZ8wYpKOyp2p329V5M +[btrfs-kernel]: https://www.kernel.org/doc/html/latest/filesystems/btrfs.html + +I intended to have three subvolumes, one for big data (not Big Data™) such as +movies or music, one for more personal data and another for backup: + +```sh +mkdir /data +mount /dev/mapper/extern /data +btrfs subvolume create /data/hoard +btrfs subvolume create /data/perso +btrfs subvolume create /data/backup +``` +### ACCIDENT! + +OK, I'll admit I don't know what the heck went on, but the main user, xarvos, +and root were apparently deleted, but not actually. I cannot log in to either +account, but they were still listed in the `/etc/passwd` file. Huh? I +cancelled a rebuild, which I have done a dozen times before and nothing went +wrong [so it should be fine this time][inductive], right? Right? (<i>Looking at +the laptop Padmely</i>) + +[inductive]: https://en.wikipedia.org/wiki/Problem_of_induction + +Per later investigation, it looks like for some unknown reason, the NixOS +configurations were removed mistakenly, and the failure was triggered by a +rebuild. + +### Rescue + +Well, it'd be a lie to say I didn't panic. But this is not something not +having happened before (translation for people confused by double negatives: +similar thing happened before). + +The easiest solution is to reinstall the thing (though it will leave me +wondering why this happenned). At the time of my NixOS installation, I didn't +make use of the service configurations in nix, only declaring those packages +and invoke them manually. Additionally, like cnx, I also didn't think of +full-disk encryption during installation. So, this is an opportunity to +rectify those wrongs + +#### Backing up + +Thanks to the available external disk as well as a ready bootable disk (you +should always have those, for occasions like these), I can simply back up my +whole home directory. I don't do stuff elsewhere, so no needs to back those +up. + +``` +mkdir /mnt/{int,ext} +mount /dev/sda1 /mnt/int +cryptsetup open --type luks /dev/sdb extern +mount /dev/mapper/extern /mnt/ext +rsync -a /mnt/int/home/xarvos /mnt/ext/backup/manual-$(date)-home-xarvos +``` + +This also mean I can do the rescue while documenting this process at the same +time 😃. + +The bootable USB drive is a NixOS installer, but it's not the latest version, +so I burn a newer installer to a second one (so handy), after backing up +the second one's content to the disk using similar process. + +#### Reinstallation + +OK, it's time to remount. I *can* install right now with `nixos-install +--root=/mnt/int`, but I want to include the external disk in the same root, so +that their hardware configuration can be generated together. Though, before +remounting, I should encrypt the main partition. + +Previously, I only have two partition---the main one and the swap, but now I +would need three, because the `/boot` can't be encrypted + +``` +parted /dev/sda +(parted) mklabel msdos +(parted) mkpart primary 128MB -8GB +(parted) mkpart primary linux-swap -8GB 100% +(parted) mkpart primary 1MB 128MB +(parted) set 3 boot on +``` + +```sh +mkswap /dev/sda2 +swapon /dev/sda2 +mkfs.ext4 -L boot /dev/sda3 + +cryptsetup luksFormat /dev/sda1 +cryptsetup open --type luks /dev/sda1 intern +mkfs.btrfs -L pain /dev/mapper/intern +``` + +(<i lang="fr">pain</i> as in French for bread, because of course we spread margarine on bread :wink:) + +```sh +mount /dev/mapper/intern /mnt +mkdir /mnt/data +mount /dev/mapper/extern /mnt/data +``` + +Before generating the config, I'd like to make some more subvolumes. I didn't +intend to run with my root on tmpfs like cnx, but I believe it'd be better to +have different partitions to be backupped (with snapshots) separately. + +```sh +btrfs subvolume create /mnt/etc +btrfs subvolume create /mnt/home +btrfs subvolume create /mnt/root +btrfs subvolume create /mnt/var +``` + +After that, I copied the configuration to /etc and generated the config. + +```sh +mkdir /mnt/etc/nixos +cp -r /mnt/data/backup/.../{services,configuration.nix} /mnt/etc/nixos +nixos-generate-config --root /mnt +``` + +(Because `nixos-generate-config` does not overwrite `configuration.nix` file if +it already exists, I can copy the configs there first.) + +While in an urgence (there was one day and a half before the holiday period +ended and I needed the machine back for work), I still spent the time to rewrite +some services, namely syncthing, transmission, and mpd. For example, I set mpd +home to `/data/hoard` + +```nix +services.mpd = { + enable = true; + user = "xarvos"; + dataDir = "/data/hoard/music"; + startWhenNeeded = true; +}; +``` + +Finally, I ran `nixos-install` to finish the setup, and sync'ed the backup back home. + +## Setting up + +(About two months later) + +After rebuilding the system, I kinda ran out of energy for doing the backup +(again), so I postponed the setup to the week after, but then the next week +something happened, which delayed it further and further, and since the +momentum died, it took quite a while for me to pick up this again. + +My initial plan was to use snapper, as I had good experience with it back when +I was using openSUSE. + +[Snapper][snapper] (HTTP without TLS link, no idea why) is a tool by openSUSE +project that helps managing file system snapshots, which was intended for +btrfs, but also supports several other file systems. See also its +[source][snapper-src], [the tutorial on opensuse.org][snapper-tutor], and +[ArchLinux wiki article][snapper-archwiki] for more information. My main use +of snapper was pre-post snapshots, which was integrated into zypper, but I have +no use of that on NixOS, since I already manage system changes in git, and +NixOS also create different boot options on rebuild as well. +[Timeline snapshot][snapper-timeline] is also great, but it seems a bit +overkill for me. Additionally, snapper doesn't seem to provide a way to send +snapshots to the external drive, so I'd have to set that up, and it'd probably +be harder to get the names correctly. + +[snapper]: http://snapper.io +[snapper-src]: https://github.com/openSUSE/snapper +[snapper-tutor]: https://en.opensuse.org/openSUSE:Snapper_Tutorial +[snapper-archwiki]: https://wiki.archlinux.org/title/Snapper +[snapper-timeline]: https://doc.opensuse.org/documentation/leap/reference/html/book-reference/cha-snapper.html#sec-snapper-clean-up-timeline + +So, I'd just copy cnx here, which is less thinking + +```sh +btrfs subvolume create /data/backup/home +today=$(date --iso-8601) +btrfs subvolume snapshot -r /home /home/$today +sync +btrfs send /home/$today | btrfs receive /data/backup/home +sync +``` + +Now, I have to setup a cron job to automate backup, as well as deletion. I +modified the script a little so running it twice does not accidentally delete +last snapshot. + +```sh +previous=$(find /home/ -maxdepth 1 -regex '.*20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]') +today=$(date --iso-8601) +if [[ "$previous" == "/home/$today" ]]; then + echo "Today is already backed up" +else + echo "Running backup" + btrfs subvolume snapshot -r /home /home/$today + btrfs send -p $previous /home/$today | btrfs receive /backup/home + btrfs subvolume delete $previous + sync +fi +``` + +Then, I set up the fcron service. Following the example in +[fcrontab manual][fcron], I can use `%hours` keyword so that this runs once in +the day from 9h to 16h: + +```nix +services.fcron = { + enable = true; + systab = "%hours 0 9-16 * * * /path/to/backup.sh" +}; +``` + +[fcrontab]: https://www.systutorials.com/docs/linux/man/5-fcrontab/ + +I think this is pretty much it. I will wait till tomorrow to see if the cron +will run properly. + + +[^0]: That means 1.81 [TiB] and I feel cheated sometimes. + +[TiB]: https://en.wikipedia.org/wiki/Byte#Multiple-byte_units. |