Network Storage Considerations (2023) - Musical Chairs Update
2023-04-13Background
Since my previous storage considerations, I’ve played a game of musical chairs with rearranging the storage workflow. A fuller picture of what I started out with, is:
Current State
I’m now mid-transition where I’ve promoted the 6-disk TrueNAS and demoted the 8-disk. The 8-disk demotion is to prepare for its eventual replacement with a Synology. That comes with a significantly lower power utilization, but means I can’t rely entirely on TrueNAS / zfs utilities as I leave the ecosystem.
That transition now looks like:
The additional goals here were to:
- Promote the 6-disk to Primary
- Start offloading app hosting and backup scheduling, away from any NAS
Enter Arq
For (2), this was a good opportunity to revisit Arq backups. It’s something I’ve had my eye on for several years, but never needed in the past because I was able to rely on the underlying zfs utilities.
Initial impressions were mixed: I liked the UI/UX and built-in explorer, but the proprietary blob file structure that it drops onto the target filesystem wasn’t what I expected. I went into this assuming it’d be similar to rclone which mirrors the file and directory structure from the source. File-level encryption with rclone simply results in hashed names and data, but in a mirrored directory structure.
That wasn’t the case with arq.
However, there is a published data format as well as a (perhaps older) cloud data format, that can be found in their docs. And their application, even unlicensed, can be used to restore files–if need be. While still a far cry from the open nature of rclone, I believe this should still prove acceptable. There are features that Arq comes with which have tipped the scales in its favor:
- Point in time backup restoration
- Deduplication
- Incremental block updates
- Using S3 (and S3-compatible) immutable object-lock
What’s Next
As the last remaining major hurdle, my Windows system still needs to point to the 6-disk TrueNAS.
After that, I can make the optimizations that I’m looking for:
- Replacing the 8-disk
- Continued offloading of apps onto the M1 Air (or my idle Raspberry Pi 4… or an Intel N5095-based PC?)
- Transferring the Lightroom Catalog from my Windows system to the M1 Air
- Incorporating a DAS into the editing workflow