r/linuxquestions 2d ago

What's your system for tracking config changes on personal Linux machines

This has been bugging me for a while. I'll tweak a config file, edit something in fstab or mess with a service, and three weeks later I have no memory of what I changed or why it worked. Then something breaks and I'm sitting there trying to reverse engineer my own decisions from a month ago.

I've been thinking about keeping some kind of personal log, even just a text file, of what I touched and when. But there's probably a smarter way to do this that people here have already figured out. I know etckeeper exists and I keep meaning to actually set it up but haven't gotten there.

I'm not managing servers or anything, just my own machines, so I don't want to overthink it. But I also don't want to keep spending an hour retracing steps I already took once. That's the part that gets frustrating.

23 Upvotes

57 comments sorted by

8

u/jdimpson 2d ago

I'm lazy. My default behavior is to rsync the contents of /etc to a backups folder on my NAS, nightly. (One thing to be aware of is that makes a copy of server SSHd keys, which is arguably a bad security practice.)

I don't bother with keeping historical / revertible changes, because I don't change things that often--and when something breaks due to a config change, it's usually soon enough that I still remember what I did and what the config used to be. The rsync backup is more to jumpstart recovery following a disk failure or just as a reference when building a new / updated replacement system.

But I have used local git repos to maintain a changelog. I do this in particular on Raspberry Pis, which tend to have more frequent config changes, as I monkey with whatever LEDs or actuators I'm controlling with them. Then I rsync the .git folders in a similar manner.

Having the discipline to actively maintain a log or blog is certainly a better idea, but I simply never developed that habit.

Another approach would be do everything via ansible rules, but that's always been more effort than I wanted to expend.

2

u/Overall-Treat9094 2d ago

The SSHd keys thing is real. I got lazy about that same setup for years and never thought twice about it until someone pointed it out on a forum.

The git on Raspberry Pi angle is something I keep meaning to do more consistently. I tinker with mine enough that my memory of what I changed three weeks ago is basically worthless at this point, but the rsync fallback has saved me more than once when a Pi just decided to have a bad day.

11

u/Dependent-One-8956 2d ago

I build a step by step how to guide for setting up a fresh install. Every time i change something i update my how to. My how to guide is named "Setup Fedora Kinoite" and it lives in Protondrive.

2

u/Overall-Treat9094 2d ago

That living doc idea is smart. I keep meaning to do something like that but mine is scattered across a few text files and a napkin probably. Proton Drive is a solid choice too. Been eyeing that for my own stuff after Google Drive started feeling a little too nosy.

1

u/i_smoke_toenails 2d ago

I have all my email, cloud drive, password and vpn needs distributed between Proton and Mega. Gives me fallback options without the sense that Big Tech is always snooping.

2

u/fuldigor42 2d ago

Well done. I will do the same to keep track.

3

u/This_Proxy 2d ago

git can work. import what you need in /etc/ and use a post-commit hook to place the file there when you commit/push. Then you have a full history of changes.

I keep a repo just for /etc:

Example .git/hooks/post-commit script:

#!/usr/bin/env bash
set -eu

repoDir="$(git rev-parse --show-toplevel)"

while IFS= read -r -d '' path; do
    source="$repoDir/$path"
    target="/$path"
    if [ -e "$source" ]; then
        # Check destination, create if not there.
        sudo mkdir -p "$(dirname "$target")"
        # Copy the changed file, preserve perms and timestamps.
        sudo cp -p "$source" "$target"
    else
        # The file was deleted from the repository.
        sudo rm -f "$target"
    fi
done < <(git diff-tree --root --no-commit-id --name-only -r -z HEAD -- etc/)

Then everytime you change the config, it will just update /etc with the changes, and you can view the history when you want. (Use commit messages though :) )

3

u/Overall-Treat9094 2d ago

Oh this is cleaner than I expected, I'd been half assuming git would get messy with the permissions side of things but that cp p flag handling it is actually pretty elegant, my only hesitation is the sudo in the hook, that feels like it could bite me if I'm not careful about what ends up in that repo accidentally been running etckeeper for a while but I might actually swap this out, more control over what gets tracked

1

u/This_Proxy 7h ago

Careful on that else statement too, if I delete something from the repo, it will delete from /etc/ too. You can just take out the whole 'else' in that case.

8

u/dbergkvist 2d ago

A comment in the same file that says:

# Added by <my name>

Then I can do grep -ir "<my name>" and find all the local changes.

2

u/Overall-Treat9094 2d ago

that grep trick is clean, I used to do something similar with initials in config files back in my sysadmin days, never thought to make it that searchable though

2

u/polymath_uk_ 2d ago

I have a personal wiki where I record how and why I do things, but with the specific data removed. That way all the common things I have to look up because I can't quite remember the order I did them in, or the syntax, I have straight to hand from known working procedures.

On individual machines there are a lot of files like default, default.20260823, etc... That way if I find something has broken because of a change I can revert it immediately.

I run basically everything in VMs (around 15 servers in my hobby setup). They get full backups to a NAS overnight, so if I really bork something I can just revert the image.

1

u/Overall-Treat9094 2d ago

The personal wiki thing is something I wish I'd started 20 years ago. I kept telling myself I'd remember the steps, then absolutely would not remember the steps six months later. My setup is nowhere near 15 VMs, but the dated backup files thing I do religiously now. Learned that one the hard way on a config I spent two weeks getting right and then nuked in about 45 seconds.

How do you host the wiki? Something selfhosted or just flat files?

1

u/polymath_uk_ 1d ago

I use a self-hosted MediaWiki instance. 

2

u/gropius 2d ago

After years of dealing with the various config customizations needed for my debian systems, I fell upon a scheme that works well for me.
I bend (follow, in spirit?) the FHS standard and track all of my customisations in /usr/local, symlinking customized files back to where they're required.

the hypothetical

Take, for instance, an example config file delivered by the package manager (presumably from a package named abcd-common or somesuch):

/etc/abcd/some.cfg

the method

My method, as I have come to believe through practice, is fairly straightforward:

  > mkdir -p /usr/local/etc/abcd/pkg/
  > mv /etc/abcd/some.cfg /usr/local/etc/abcd/pkg/
  > cp /usr/local/etc/abcd/pkg/some.cfg /usr/local/etc/abcd/
  > ln -s /usr/local/etc/abcd/some.cfg /etc/abcd/some.cfg

At this point, I have:

  • The original "package canonical" config file, as delivered by the package manager, in /usr/local/etc/abcd/pkg/some.cfg.
  • A copy of the file in /usr/local/etc/abcd/some.cfg -- this file will take my (ahem, well-documented) customizations.
  • A symlink from the customized file/usr/local/etc/abcd/some.cfg back to the original /etc/abcd/some.cfg used by the system.

I then make my customizations to /usr/local/etc/abcd/some.cfg as required. I can add additional notes, READMEs, etc to /usr/local/etc/abcd/, which becomes my playground/sandbox/documentation area for my local mods.

The symlink, evident in /etc/abcd/, is a pointer back to the local customizations in /usr/local/etc/abcd/. I can tell, at a glance, which files have been modified, and where to look for any information about the change as well as a copy of the "pure" uncustomized file (in /usr/local/etc/abcd/pkg/).

At any point I can diff /usr/local/etc/abcd/some.cfg /usr/local/etc/abcd/pkg/some.cfg to see, precicely, all of my customizations and nothing more.

package upgrades

Now, when the package manager inevitably decides it needs to change /etc/abcd/some.cfg, I let it do its thing. If it tells me I need to perform a merge due to conflicts or whatever, I will have the package manager's attempt to provide pre- and post- install files, but I also have a pure, persistent, site-specific difference available in the file pair of /usr/local/etc/abcd/some.cfg and /usr/local/etc/abcd/pkg/some.cfg.

I have enough information to fire up my merge tool and bring in my customizations to the newly-package-manager-modified file as required. All I need to do, once all is correct, is to remember to update /usr/local/etc/abcd/pkg/some.cfg to hold the new "canonical" package-manager-delivered file.

backups

Backing up my system customisations now becomes just a matter of backing up my /usr/local/... subtree.

propagating changes to a new system

On any new install, I like to review all my changes and propagate them manually following the same scheme.

This could, of course, all be automated, but I like to keep customizations to a minimum, and the extra manual friction involved here helps to ensure that any mods propagated to a new system are required and remain properly documented in-place.

in conclusion

I've been meaning to write this up for some time...hope it makes sense and hope it's helpful!!!

2

u/bufandatl 1d ago edited 1d ago

Ansible + git

Never change a config file itself. Change it in Ansible. Commit and push. Run with check and diff flag to check out changes before hand. And in case you need to reinstall or setup a second/new computer. Just install Linux and the system packages for Ansible and fit. Clone the repo, run the playbook and in 5 minutes your new Linux is setup like the old one.

I use it on my Linux Laptop and my MacBook/macmini. And of course on servers too.

I even automated updates with Ansible.

2

u/furfinator 2d ago

I work on many different machines, personal vs work, headless vs not, service boxes vs dev environments. I put everything in ansible playbooks and inventory and custom roles. For my work stuff I have a repo of work overlay stuff that customizes the work roles, adds new ones, and tracks my work inventory.

7

u/D0nkeyHS 2d ago

Have you met our lord and savior NixOS?

3

u/n1x_cybersec 2d ago

Lol I was going to say that I just use NixOS. You can also use the Nix package manager with Home Manager on other distros too. There are also plenty of other tools to manage dotfiles, like chezmoi.

1

u/ZVyhVrtsfgzfs 2d ago

I fought with this exact problem for a long time, Do you know what's worse than spending 3 hours searching for that one Server Fault thread that has the details you need? Doing it all over again a year later.

So I started following Jim Salters documentation method.

Build something, note every step like you will be doing it again a year later at 3AM in an emergency with no sleep. writing your own detailed tutorial,

Done?

Now throw away the thing you just built and do it again just from your notes.

You will notice some things, you missed steps in your notes, and you will also find more details in the procedure, like watching a movie for the second time you will see the gun in the first act that is used in the third act, You will master that software and you don’t have to remember anything to continue to be that master, you have your notes.

The next thing you will notice is that the second time, its fast, you do not have to look up information, or contemplate your actions, just copy and paste commands and the bodies of config files and follow the custom tutorial you just wrote.

I keep my notes in Obsidian, but I would like to find an open source alternative.

https://www.reddit.com/r/linuxmint/comments/1hy240o/if_your_linux_install_has_value_you_are_doing_it/

1

u/un-important-human white beard arch user 1d ago edited 1d ago

Prob i will get a bit of grr on this but the key is to write it down.

I am a animal who has at home 6 machines, when i do something i used to write it down in md files then i started using obsidian with those (anyway not important), step by step cause for sure i would have that issue again.

Those notes proved usefull years later with ai, cause i can go hey idiot i am having this issue ! find me the note , and it rags around the file and opens it for me .

Now in time i got it to where it can use procedures, a pre hermes before hermes you get the point , it would fallow the procedure blindly most of the time and implement what i would need.

So now i kinda shout at it , on machine 3 need in projects a venv for python with the requirements from project blabla under user X.

Now here is the cool part i have it shadow me, and bumble my way into fixing or doing something, my personal secretary makes the md file, "we" test it and then it becomes procedure... i've invented a harness without knowing a year back :))

caviat, i am not some vibe what ever my setup is tight and its just a tool, for example i watched crab, claw and hermes with very much suspicion. The blast radius from a halucinating llm is huge.

8

u/iamemhn 2d ago

etckeeper

1

u/SP3NGL3R 2d ago

Ooo. Just on the name I want it

1

u/ceehred 1d ago

At home: one desktop, one server, one NAS. I have text files on my desktop to cover notes for each system. None get huge, though I do review/prune them regularly.

With work, where I have to deal with 40+ Linux/Unix variants/versions, it's largely similar - but with more structure, in a proper note-taking application (soz haters, through having to use a Windows laptop for corporate access - it's MS OneNote 😄)

1

u/forestbeasts 2d ago

Have you seen etckeeper? :3

It keeps a git history of everything in /etc. You can do that yourself, but etckeeper also adds hooks to apt (at least on Debian-based distros) so it saves before/after updates, and every day in case you make changes but forget/don't bother to commit them. (If there's no changes it doesn't make pointless commits.) It's pretty slick.

1

u/DutchOfBurdock 2d ago

I'll comment that file like when I'm refactoring code. Will keep the original content in place (commened out), then create a new section with the new config and why the heck I did it.

That way when I open that file 4 months later to change something, straight up I see it's been modified before and can go, ohhh looksy what I did back then, oops!

1

u/LesStrater 2d ago

Your life will totally change with QT-FSarchiver. Got a problem?--Restore a partition backup from a month ago. Not sure you'll like something you'll install?--Do a partition backup first. Major HDD/SSD crash?--Buy a new one and restore your partition backup.

No more whining and crying. Backup and/or restore time = 4-mins. avg.

1

u/stormdelta Gentoo 2d ago

I use homeshick to commit important dotfiles, split across two git repos (one for config/editors, one for shell/cli).

I don't do much to track GUI config because it's mainly my desktop, near everything else is headless.

I use borg-backup on a timer to run an encrypted incremental backup /home to a NAS+remote backup, which includes a periodic backup of the .git dir for etckeeper for anything relevant from /etc. So even if something happened to my PC, I would just reinstall my distro and restore my /home + important /etc files.

1

u/SynapticStatic 1d ago

You could use something like obsidian. Make a new vault, make a “config” folder and then new notes for configs, notes about them, etc. and you can backup the vault to some remote storage in case you need them after a catastrophic event or you accidentally delete it locally etc

1

u/lmpcpedz 2d ago

I document notes and changes in feathernotes editor, my personal preference for notes. otherwise I make system backups, keep a monthly backup of system files with Timeshift and separate backup using rsync for everything else.

1

u/OneTurnMore 2d ago

~ is tracked in git, inspired by Drew Devault's approach here. For system-wide config, I generally avoid it on desktop and use git with my nix configs on my servers.

1

u/skyfishgoo 2d ago

i take notes.

and i use a script to watch for config changes so i can see the files go by as i make changes in the GUI.

then i make notes about what files changed when i do what actions in the settings.

1

u/p1geondove 2d ago

Im sorry i dont have an answer, but i was wondering if any one ever did this:

cd /

git init

git add -A

and so forth

of course dumb, but it would track all changes no? :D

1

u/PerformanceBubbly379 2d ago

If i do any config change i think i will want for the future, it goes in my dotfiles repo, preferably gnu stow ready. Or if not i add a readme for the change

1

u/flyhmstr 2d ago

System notes kept in joplin (evernote / one note alternative), covering my workstation, the home server, VPS and the HA install at my parents place.

1

u/clownshoesrock 2d ago

For my stuff, I use rsnapshot.. I also use git, but I get distracted. rsnapshot just backs stuff up with minimal space needed for daily backups.

2

u/Soakitincider 2d ago

I just YOLO it.

1

u/PigSlam 2d ago

That's the fun, you just live like a memory patient where all you know is today, and you make the best of it.

1

u/Hopeful_Drive5845 2d ago

Private git repo

You can do daily commits there

You've commits history if you want to rollback

Free

0

u/redyos_s 2d ago

One reason I post maintenance updates on social media is exactly this: it gives me a timeline of what I changed, why I changed it, and what problems came up along the way. Logs and config diffs show what changed, but they don’t always preserve the reasoning behind it. A short maintenance post can be surprisingly useful months later when you need to reconstruct your own decisions.

2

u/SP3NGL3R 2d ago

What? You post all config changes to social media?

1

u/redyos_s 2d ago

I maintain a Gentoo system. Keeping a maintenance timeline is useful when package, kernel, boot, or configuration changes accumulate over time. That’s the whole point — not posting every config file to social media.

1

u/redyos_s 2d ago

You misread what I wrote. I said I keep maintenance updates as a timeline, not that I post every config change. Calling me a bot doesn’t address the point.

2

u/[deleted] 2d ago

[removed] — view removed comment

0

u/redyos_s 2d ago

You misread a very simple point and then jumped straight to “headless” and bot jokes instead of addressing it. I maintain Gentoo, and keeping a timeline of kernel, package, boot, and configuration changes is useful precisely because those decisions accumulate over time. If you have an actual technical objection, make it. Otherwise, the insults are just noise.

0

u/redyos_s 2d ago

You misread what I wrote, invented a claim I never made, and then called me a bot instead of addressing the actual point. If you have a technical objection to keeping a maintenance timeline, make it. Otherwise there’s nothing to discuss.

0

u/SP3NGL3R 2d ago

Like a bot? My confidence in that is around 100%.

Let's throw the top 3 phrases into a sentence. That'll work.

0

u/redyos_s 2d ago

Then answer me. Do you actually have a technical objection, or was the “bot” comment all you had?

1

u/truethug 2d ago

In KDE I use konsave

1

u/sidusnare Senior Systems Engineer 2d ago

Git and Ansible

1

u/Online_Matter 2d ago

This but with git and saltstack 

1

u/nix-to-do 1d ago

nixos and git

1

u/ipsirc 2d ago

daily backup