Add blog post on disk backup

2 files changed, 295 insertions, 120 deletions

diff --git a/content/posts/2023-09-02-setup-external-backup.md b/content/posts/2023-09-02-setup-external-backup.md
deleted file mode 100644
index 6a94158..0000000
--- a/content/posts/2023-09-02-setup-external-backup.md
+++ /dev/null
@@ -1,120 +0,0 @@
----
-title: "Setup backup on external disk"
-date: 2023-09-02
-draft: true
-lang: en
-categories: [ blog ]
-tags: []
-translationKey: "2023-09-02-setup-external-backup"
----
-
-## Backstory
-
-A few weeks ago (not today), I fucked up:
-During irregular disk cleanup, I accidentally removed the config folder!
-
-Fortunately, I "backed up" the config with git, didn't I? 😌
-
-Well yes, but actually no! 🥲 I did commit, but I didn't push.
-The latest pushed config had been 9 months before this incident.
-
-On top of that, many programs generate unreadable configs, which I prefer not
-to put into version control, or put sensitive information or large data into
-`.config`, which shouldn't be pushed into a remote repository on a server I
-don't own.  For example, nheko puts authentication info there, and after this
-incident I lost the session with several messages---well, matrix's irregularly
-regular decryption failure could be expanded into a post on its own, so let's
-not digress.
-
-And then there are more things to back up than just configs, say, my photos and
-music.  I mirror them between my devices with [syncthing], providing
-redundancy, but [syncthing is not backup][syncthing-failure],
-and neither is [RAID][raid-not-backup]
-
-[syncthing]: https://syncthing.net/
-[syncthing-failure]: https://forum.syncthing.net/t/syncthing-deleted-all-files-on-every-device/20518
-[raid-not-backup]: https://www.raidisnotabackup.com/
-
-I should back them up, or I might one day be doomed to repeat the mistake.
-
-## Preparation
-
-Despite the bitterness from the partial data loss, I kinda didn't have the
-spoon to do what I must, so I've been delaying it hitherto.
-
-### Hardware
-
-One of the reason I was cleaning up the disk in the first place was that it's
-start to be filled up, so I bought a large hard drive to store more data and
-backup.  Doesn't seem too safe, I know: if the hard drive gets lost or broken,
-I'll lose both the main data and the backup, but with a limited budget and
-space, that's good enough for now.
-
-I got a Seagate One Touch 2 TB[^0] HDD.  It seems to come with builtin backup
-software, which doesn't support Linux of course, and I likely wouldn't use it
-even if it did.  A USB hub is also needed, as my laptop has a limited number of
-USB ports.
-
-### Formatting
-
-Coming with the hard drive are several files, such as an `.exe` program to
-initialize the backup, I suppose.  I don't need these, and the disk can be
-formatted ~~right away~~ after I do full-disk encryption on that.
-
-No partitioning is needed: while I do seek [vegan alternative][margarine], I
-intended to use [btrfs][btrfs-kernel] as the file system, which can create
-subvolumes, which can act as separate mounted partitions.  I will just reformat
-the existing partition to btrfs.
-
-```sh
-mkfs.btrfs -L margarine /dev/sdb1
-```
-
-[margarine]: https://outerheaven.club/notice/AZ8wYpKOyp2p329V5M
-[btrfs-kernel]: https://www.kernel.org/doc/html/latest/filesystems/btrfs.html
-
-I intend to have three subvolumes, one for big data (not Big Data™) such as
-movies or music, one for more personal data and another for backup:
-
-```sh
-mount /dev/sdb1 /mnt
-btrfs subvolume /mnt/data/hoard
-btrfs subvolume /mnt/data/perso
-btrfs subvolume /mnt/backup
-```
-
-Finally, I need to declare those in the nix hardware configuration file.
-
-
-```nix
-{
-  (...)
-  fileSystems = {
-    "/" = {
-      (...)
-    };
-    "/data/hoard" = {
-      device = "/dev/disk/by-uuid/[subvolume uuid]";
-      fsType = "btrfs";
-      options = [ "subvol=data/hoard" "compress-force=zstd" "noatime" ];
-    };
-    "/data/perso" = {
-      device = "/dev/disk/by-uuid/[subvolume uuid]";
-      fsType = "btrfs";
-      options = [ "subvol=data/perso" "compress-force=zstd" "noatime" ];
-    };
-    "/backup" = {
-      device = "/dev/disk/by-uuid/[subvolume uuid]";
-      fsType = "btrfs";
-      options = [ "subvol=backup" "compress-force=zstd" ];
-    };
-  };
-  (...)
-}
-```
-
-## Setting up
-
-[^0]: That means 1.81 [TiB] and I feel cheated sometimes.
-
-[TiB]: https://en.wikipedia.org/wiki/Byte#Multiple-byte_units.
diff --git a/content/posts/2023-10-29-setup-external-backup.md b/content/posts/2023-10-29-setup-external-backup.md
new file mode 100644
index 0000000..017fee9
--- /dev/null
+++ b/content/posts/2023-10-29-setup-external-backup.md
@@ -0,0 +1,295 @@
+---
+title: "Setup backup on external disk"
+date: 2023-10-29
+lang: en
+categories: [ blog ]
+tags: []
+translationKey: "2023-10-29-setup-external-backup"
+---
+
+## Backstory
+
+A few weeks ago (not today, and by the time I published this post it's a few
+months ago), I fucked up:
+During irregular disk cleanup, I accidentally removed the config folder!
+
+Fortunately, I "backed up" the config with git, didn't I? 😌
+
+Well yes, but actually no! 🥲 I did commit, but I didn't push.
+The latest pushed config had been 9 months before this incident.
+
+On top of that, many programs generate unreadable configs, which I prefer not
+to put into version control, or put sensitive information or large data into
+`.config`, which shouldn't be pushed into a remote repository on a server I
+don't own.  For example, nheko puts authentication info there, and after this
+incident I lost the session with several messages---well, matrix's irregularly
+regular decryption failure could be expanded into a post on its own, so let's
+not digress.
+
+And then there are more things to back up than just configs, say, my photos and
+music.  I mirror them between my devices with [syncthing], providing
+redundancy, but [syncthing is not backup][syncthing-failure],
+and neither is [RAID][raid-not-backup]
+
+[syncthing]: https://syncthing.net/
+[syncthing-failure]: https://forum.syncthing.net/t/syncthing-deleted-all-files-on-every-device/20518
+[raid-not-backup]: https://www.raidisnotabackup.com/
+
+I should back them up, or I might one day be doomed to repeat the mistake.
+
+## Preparation
+
+Despite the bitterness from the partial data loss, I kinda didn't have the
+spoon to do what I must, so I've been delaying it hitherto.
+
+### Hardware
+
+One of the reason I was cleaning up the disk in the first place was that it's
+start to be filled up, so I bought a large hard drive to store more data and
+backup.  Doesn't seem too safe, I know: if the hard drive gets lost or broken,
+I'll lose both the main data and the backup, but with a limited budget and
+space, that's good enough for now.  I am planning for some off-site backup.
+
+I got a Seagate One Touch 2 TB[^0] HDD.  It seems to come with builtin backup
+software, which doesn't support Linux of course, and I likely wouldn't use it
+even if it did.  A USB hub is also needed, as my laptop has a limited number of
+USB ports.
+
+### Formatting
+
+Coming with the hard drive are several files, such as an `.exe` program to
+initialize the backup, I suppose.  I don't need these, and the disk can be
+formatted ~~right away~~ after I do full-disk encryption on that.
+
+```sh
+cryptsetup luksFormat /dev/sdb
+cryptsetup open --type luks /dev/sdb extern
+```
+
+No partitioning is needed: while I do seek [vegan alternative][margarine], I
+intended to use [btrfs][btrfs-kernel] as the file system, which can create
+subvolumes, which can act as separate mounted partitions.  I would just
+reformat the existing partition to btrfs.
+
+```sh
+mkfs.btrfs -L margarine /dev/mapper/extern
+```
+
+[margarine]: https://outerheaven.club/notice/AZ8wYpKOyp2p329V5M
+[btrfs-kernel]: https://www.kernel.org/doc/html/latest/filesystems/btrfs.html
+
+I intended to have three subvolumes, one for big data (not Big Data™) such as
+movies or music, one for more personal data and another for backup:
+
+```sh
+mkdir /data
+mount /dev/mapper/extern /data
+btrfs subvolume create /data/hoard
+btrfs subvolume create /data/perso
+btrfs subvolume create /data/backup
+```
+### ACCIDENT!
+
+OK, I'll admit I don't know what the heck went on, but the main user, xarvos,
+and root were apparently deleted, but not actually.  I cannot log in to either
+account, but they were still listed in the `/etc/passwd` file. Huh?  I
+cancelled a rebuild, which I have done a dozen times before and nothing went
+wrong [so it should be fine this time][inductive], right? Right? (<i>Looking at
+the laptop Padmely</i>)
+
+[inductive]: https://en.wikipedia.org/wiki/Problem_of_induction
+
+Per later investigation, it looks like for some unknown reason, the NixOS
+configurations were removed mistakenly, and the failure was triggered by a
+rebuild.
+
+### Rescue
+
+Well, it'd be a lie to say I didn't panic.  But this is not something not
+having happened before (translation for people confused by double negatives:
+similar thing happened before).
+
+The easiest solution is to reinstall the thing (though it will leave me
+wondering why this happenned).  At the time of my NixOS installation, I didn't
+make use of the service configurations in nix, only declaring those packages
+and invoke them manually.  Additionally, like cnx, I also didn't think of
+full-disk encryption during installation.  So, this is an opportunity to
+rectify those wrongs
+
+#### Backing up
+
+Thanks to the available external disk as well as a ready bootable disk (you
+should always have those, for occasions like these), I can simply back up my
+whole home directory.  I don't do stuff elsewhere, so no needs to back those
+up.
+
+```
+mkdir /mnt/{int,ext}
+mount /dev/sda1 /mnt/int
+cryptsetup open --type luks /dev/sdb extern
+mount /dev/mapper/extern /mnt/ext
+rsync -a /mnt/int/home/xarvos /mnt/ext/backup/manual-$(date)-home-xarvos
+```
+
+This also mean I can do the rescue while documenting this process at the same
+time 😃.
+
+The bootable USB drive is a NixOS installer, but it's not the latest version,
+so I burn a newer installer to a second one (so handy), after backing up
+the second one's content to the disk using similar process.
+
+#### Reinstallation
+
+OK, it's time to remount.  I *can* install right now with `nixos-install
+--root=/mnt/int`, but I want to include the external disk in the same root, so
+that their hardware configuration can be generated together.  Though, before
+remounting, I should encrypt the main partition.
+
+Previously, I only have two partition---the main one and the swap, but now I
+would need three, because the `/boot` can't be encrypted
+
+```
+parted /dev/sda
+(parted) mklabel msdos
+(parted) mkpart primary 128MB -8GB
+(parted) mkpart primary linux-swap -8GB 100%
+(parted) mkpart primary 1MB 128MB
+(parted) set 3 boot on
+```
+
+```sh
+mkswap /dev/sda2
+swapon /dev/sda2
+mkfs.ext4 -L boot /dev/sda3
+
+cryptsetup luksFormat /dev/sda1
+cryptsetup open --type luks /dev/sda1 intern
+mkfs.btrfs -L pain /dev/mapper/intern
+```
+
+(<i lang="fr">pain</i> as in French for bread, because of course we spread margarine on bread :wink:)
+
+```sh
+mount /dev/mapper/intern /mnt
+mkdir /mnt/data
+mount /dev/mapper/extern /mnt/data
+```
+
+Before generating the config, I'd like to make some more subvolumes.  I didn't
+intend to run with my root on tmpfs like cnx, but I believe it'd be better to
+have different partitions to be backupped (with snapshots) separately.
+
+```sh
+btrfs subvolume create /mnt/etc
+btrfs subvolume create /mnt/home
+btrfs subvolume create /mnt/root
+btrfs subvolume create /mnt/var
+```
+
+After that, I copied the configuration to /etc and generated the config.
+
+```sh
+mkdir /mnt/etc/nixos
+cp -r /mnt/data/backup/.../{services,configuration.nix} /mnt/etc/nixos
+nixos-generate-config --root /mnt
+```
+
+(Because `nixos-generate-config` does not overwrite `configuration.nix` file if
+it already exists, I can copy the configs there first.)
+
+While in an urgence (there was one day and a half before the holiday period
+ended and I needed the machine back for work), I still spent the time to rewrite
+some services, namely syncthing, transmission, and mpd.  For example, I set mpd
+home to `/data/hoard`
+
+```nix
+services.mpd = {
+  enable = true;
+  user = "xarvos";
+  dataDir = "/data/hoard/music";
+  startWhenNeeded = true;
+};
+```
+
+Finally, I ran `nixos-install` to finish the setup, and sync'ed the backup back home.
+
+## Setting up
+
+(About two months later)
+
+After rebuilding the system, I kinda ran out of energy for doing the backup
+(again), so I postponed the setup to the week after, but then the next week
+something happened, which delayed it further and further, and since the
+momentum died, it took quite a while for me to pick up this again.
+
+My initial plan was to use snapper, as I had good experience with it back when
+I was using openSUSE.
+
+[Snapper][snapper] (HTTP without TLS link, no idea why) is a tool by openSUSE
+project that helps managing file system snapshots, which was intended for
+btrfs, but also supports several other file systems.  See also its
+[source][snapper-src], [the tutorial on opensuse.org][snapper-tutor], and
+[ArchLinux wiki article][snapper-archwiki] for more information.  My main use
+of snapper was pre-post snapshots, which was integrated into zypper, but I have
+no use of that on NixOS, since I already manage system changes in git, and
+NixOS also create different boot options on rebuild as well.
+[Timeline snapshot][snapper-timeline] is also great, but it seems a bit
+overkill for me.  Additionally, snapper doesn't seem to provide a way to send
+snapshots to the external drive, so I'd have to set that up, and it'd probably
+be harder to get the names correctly.
+
+[snapper]: http://snapper.io
+[snapper-src]: https://github.com/openSUSE/snapper
+[snapper-tutor]: https://en.opensuse.org/openSUSE:Snapper_Tutorial
+[snapper-archwiki]: https://wiki.archlinux.org/title/Snapper
+[snapper-timeline]: https://doc.opensuse.org/documentation/leap/reference/html/book-reference/cha-snapper.html#sec-snapper-clean-up-timeline
+
+So, I'd just copy cnx here, which is less thinking
+
+```sh
+btrfs subvolume create /data/backup/home
+today=$(date --iso-8601)
+btrfs subvolume snapshot -r /home /home/$today
+sync
+btrfs send /home/$today | btrfs receive /data/backup/home
+sync
+```
+
+Now, I have to setup a cron job to automate backup, as well as deletion.  I
+modified the script a little so running it twice does not accidentally delete
+last snapshot.
+
+```sh
+previous=$(find /home/ -maxdepth 1 -regex '.*20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]')
+today=$(date --iso-8601)
+if [[ "$previous" == "/home/$today" ]]; then
+	echo "Today is already backed up"
+else
+	echo "Running backup"
+	btrfs subvolume snapshot -r /home /home/$today
+	btrfs send -p $previous /home/$today | btrfs receive /backup/home
+	btrfs subvolume delete $previous
+	sync
+fi
+```
+
+Then, I set up the fcron service.  Following the example in
+[fcrontab manual][fcron], I can use `%hours` keyword so that this runs once in
+the day from 9h to 16h:
+
+```nix
+services.fcron = {
+  enable = true;
+  systab = "%hours 0 9-16 * * * /path/to/backup.sh"
+};
+```
+
+[fcrontab]: https://www.systutorials.com/docs/linux/man/5-fcrontab/
+
+I think this is pretty much it. I will wait till tomorrow to see if the cron
+will run properly.
+
+
+[^0]: That means 1.81 [TiB] and I feel cheated sometimes.
+
+[TiB]: https://en.wikipedia.org/wiki/Byte#Multiple-byte_units.