---
title: "Setup backup on external disk"
date: 2023-10-29
lang: en
categories: [ blog, guide, sysadmin ]
tags: 
  - sysadmin
  - backup
  - mess up
translationKey: "2023-10-29-setup-external-backup"
---

## Backstory

A few weeks ago (not today, and by the time I published this post it's a few
months ago), I fucked up:
During irregular disk cleanup, I accidentally removed the config folder!

Fortunately, I "backed up" the config with git, didn't I? 😌

Well yes, but actually no! 🥲 I did commit, but I didn't push.
The latest pushed config had been 9 months before this incident.

On top of that, many programs generate unreadable configs, which I prefer not
to put into version control, or put sensitive information or large data into
`.config`, which shouldn't be pushed into a remote repository on a server I
don't own.  For example, nheko puts authentication info there, and after this
incident I lost the session with several messages---well, matrix's irregularly
regular decryption failure could be expanded into a post on its own, so let's
not digress.

And then there are more things to back up than just configs, say, my photos and
music.  I mirror them between my devices with [syncthing], providing
redundancy, but [syncthing is not backup][syncthing-failure],
and neither is [RAID][raid-not-backup]

[syncthing]: https://syncthing.net/
[syncthing-failure]: https://forum.syncthing.net/t/syncthing-deleted-all-files-on-every-device/20518
[raid-not-backup]: https://www.raidisnotabackup.com/

I should back them up, or I might one day be doomed to repeat the mistake.

## Preparation

Despite the bitterness from the partial data loss, I kinda didn't have the
spoon to do what I must, so I've been delaying it hitherto.

### Hardware

One of the reason I was cleaning up the disk in the first place was that it's
start to be filled up, so I bought a large hard drive to store more data and
backup.  Doesn't seem too safe, I know: if the hard drive gets lost or broken,
I'll lose both the main data and the backup, but with a limited budget and
space, that's good enough for now.  I am planning for some off-site backup.

I got a Seagate One Touch 2 TB[^0] HDD.  It seems to come with builtin backup
software, which doesn't support Linux of course, and I likely wouldn't use it
even if it did.  A USB hub is also needed, as my laptop has a limited number of
USB ports.

### Formatting

Coming with the hard drive are several files, such as an `.exe` program to
initialize the backup, I suppose.  I don't need these, and the disk can be
formatted ~~right away~~ after I do full-disk encryption on that.

```sh
cryptsetup luksFormat /dev/sdb
cryptsetup open --type luks /dev/sdb extern
```

No partitioning is needed: while I do seek [vegan alternative][margarine], I
intended to use [btrfs][btrfs-kernel] as the file system, which can create
subvolumes, which can act as separate mounted partitions.  I would just
reformat the existing partition to btrfs.

```sh
mkfs.btrfs -L margarine /dev/mapper/extern
```

[margarine]: https://outerheaven.club/notice/AZ8wYpKOyp2p329V5M
[btrfs-kernel]: https://www.kernel.org/doc/html/latest/filesystems/btrfs.html

I intended to have three subvolumes, one for big data (not Big Data™) such as
movies or music, one for more personal data and another for backup:

```sh
mkdir /data
mount /dev/mapper/extern /data
btrfs subvolume create /data/hoard
btrfs subvolume create /data/perso
btrfs subvolume create /data/backup
```
### ACCIDENT!

OK, I'll admit I don't know what the heck went on, but the main user, xarvos,
and root were apparently deleted, but not actually.  I cannot log in to either
account, but they were still listed in the `/etc/passwd` file. Huh?  I
cancelled a rebuild, which I have done a dozen times before and nothing went
wrong [so it should be fine this time][inductive], right? Right? (<i>Looking at
the laptop Padmely</i>)

[inductive]: https://en.wikipedia.org/wiki/Problem_of_induction

Per later investigation, it looks like for some unknown reason, the NixOS
configurations were removed mistakenly, and the failure was triggered by a
rebuild.

### Rescue

Well, it'd be a lie to say I didn't panic.  But this is not something not
having happened before (translation for people confused by double negatives:
similar thing happened before).

The easiest solution is to reinstall the thing (though it will leave me
wondering why this happenned).  At the time of my NixOS installation, I didn't
make use of the service configurations in nix, only declaring those packages
and invoke them manually.  Additionally, like cnx, I also didn't think of
full-disk encryption during installation.  So, this is an opportunity to
rectify those wrongs

#### Backing up

Thanks to the available external disk as well as a ready bootable disk (you
should always have those, for occasions like these), I can simply back up my
whole home directory.  I don't do stuff elsewhere, so no needs to back those
up.

```
mkdir /mnt/{int,ext}
mount /dev/sda1 /mnt/int
cryptsetup open --type luks /dev/sdb extern
mount /dev/mapper/extern /mnt/ext
rsync -a /mnt/int/home/xarvos /mnt/ext/backup/manual-$(date)-home-xarvos
```

This also mean I can do the rescue while documenting this process at the same
time 😃.

The bootable USB drive is a NixOS installer, but it's not the latest version,
so I burn a newer installer to a second one (so handy), after backing up
the second one's content to the disk using similar process.

#### Reinstallation

OK, it's time to remount.  I *can* install right now with `nixos-install
--root=/mnt/int`, but I want to include the external disk in the same root, so
that their hardware configuration can be generated together.  Though, before
remounting, I should encrypt the main partition.

Previously, I only have two partition---the main one and the swap, but now I
would need three, because the `/boot` can't be encrypted

```
parted /dev/sda
(parted) mklabel msdos
(parted) mkpart primary 128MB -8GB
(parted) mkpart primary linux-swap -8GB 100%
(parted) mkpart primary 1MB 128MB
(parted) set 3 boot on
```

```sh
mkswap /dev/sda2
swapon /dev/sda2
mkfs.ext4 -L boot /dev/sda3

cryptsetup luksFormat /dev/sda1
cryptsetup open --type luks /dev/sda1 intern
mkfs.btrfs -L pain /dev/mapper/intern
```

(<i lang="fr">pain</i> as in French for bread, because of course we spread margarine on bread :wink:)

```sh
mount /dev/mapper/intern /mnt
mkdir /mnt/data
mount /dev/mapper/extern /mnt/data
```

Before generating the config, I'd like to make some more subvolumes.  I didn't
intend to run with my root on tmpfs like cnx, but I believe it'd be better to
have different partitions to be backupped (with snapshots) separately.

```sh
btrfs subvolume create /mnt/etc
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/var
```

After that, I copied the configuration to /etc and generated the config.

```sh
mkdir /mnt/etc/nixos
cp -r /mnt/data/backup/.../{services,configuration.nix} /mnt/etc/nixos
nixos-generate-config --root /mnt
```

(Because `nixos-generate-config` does not overwrite `configuration.nix` file if
it already exists, I can copy the configs there first.)

While in an urgence (there was one day and a half before the holiday period
ended and I needed the machine back for work), I still spent the time to rewrite
some services, namely syncthing, transmission, and mpd.  For example, I set mpd
home to `/data/hoard`

```nix
services.mpd = {
  enable = true;
  user = "xarvos";
  dataDir = "/data/hoard/music";
  startWhenNeeded = true;
};
```

Finally, I ran `nixos-install` to finish the setup, and sync'ed the backup back home.

## Setting up

(About two months later)

After rebuilding the system, I kinda ran out of energy for doing the backup
(again), so I postponed the setup to the week after, but then the next week
something happened, which delayed it further and further, and since the
momentum died, it took quite a while for me to pick up this again.

My initial plan was to use snapper, as I had good experience with it back when
I was using openSUSE.

[Snapper][snapper] (HTTP without TLS link, no idea why) is a tool by openSUSE
project that helps managing file system snapshots, which was intended for
btrfs, but also supports several other file systems.  See also its
[source][snapper-src], [the tutorial on opensuse.org][snapper-tutor], and
[ArchLinux wiki article][snapper-archwiki] for more information.  My main use
of snapper was pre-post snapshots, which was integrated into zypper, but I have
no use of that on NixOS, since I already manage system changes in git, and
NixOS also create different boot options on rebuild as well.
[Timeline snapshot][snapper-timeline] is also great, but it seems a bit
overkill for me.  Additionally, snapper doesn't seem to provide a way to send
snapshots to the external drive, so I'd have to set that up, and it'd probably
be harder to get the names correctly.

[snapper]: http://snapper.io
[snapper-src]: https://github.com/openSUSE/snapper
[snapper-tutor]: https://en.opensuse.org/openSUSE:Snapper_Tutorial
[snapper-archwiki]: https://wiki.archlinux.org/title/Snapper
[snapper-timeline]: https://doc.opensuse.org/documentation/leap/reference/html/book-reference/cha-snapper.html#sec-snapper-clean-up-timeline

So, I'd just copy cnx here, which is less thinking

```sh
btrfs subvolume create /data/backup/home
today=$(date --iso-8601)
btrfs subvolume snapshot -r /home /home/$today
sync
btrfs send /home/$today | btrfs receive /data/backup/home
sync
```

Now, I have to setup a cron job to automate backup, as well as deletion.  I
modified the script a little so running it twice does not accidentally delete
last snapshot.

```sh
previous=$(find /home/ -maxdepth 1 -regex '.*20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]')
today=$(date --iso-8601)
if [[ "$previous" == "/home/$today" ]]; then
	echo "Today is already backed up"
else
	echo "Running backup"
	btrfs subvolume snapshot -r /home /home/$today
	btrfs send -p $previous /home/$today | btrfs receive /data/backup/home
	btrfs subvolume delete $previous
	sync
fi
```

Then, I set up the fcron service.  Following the example in
[fcrontab manual][fcrontab], I can use `%hours` keyword so that this runs once
in the day from 9h to 16h:

```nix
services.fcron = {
  enable = true;
  systab = "%hours 0 9-16 * * * /path/to/backup.sh"
};
```

[fcrontab]: https://www.systutorials.com/docs/linux/man/5-fcrontab/

I think this is pretty much it. I will wait till tomorrow to see if the cron
will run properly.


[^0]: That means 1.81 [TiB] and I feel cheated sometimes.

[TiB]: https://en.wikipedia.org/wiki/Byte#Multiple-byte_units.