How "stable" are these kinds of slimming tweaks though? In a rolling release setup, aren't you going to have extraneous packages semi-regularly drag in all of Python or Perl accidentally? The setup might break and you'd not even notice?
I find the premise of a carefully re-compilable/re-creatable system very appealing, but not having a stable LTS style release rather incongruous. It takes a huge effort to get all the pieces working together - and if it's rolling and the sand is shifting/breaking underneath you it feels you never reach a meaningful stable system. Sure you can recreate your well tested working configuration, but the configurations is effectively immediately out of date and unmaintained once any packages are updated
I think this is why they effectively only target x64. I'm not a "distro guy" so maybe I'm missing something. It seems it'd be sensible to just 1-to-1 copy Ubuntu LTS package versions (+ patches) and build a NixOS "stable" version that can be patched and updated as people find issues
What is the security update policy with NixOS? I've always had the impression that I would need to upgrade every 6 months to keep getting critical security fixes, which is unappealing compared to e.g. Debian + Nix
I know I can revert easily if there are problems when upgrading, but that doesn't really apply if security fixes only land in the branch
I don't know any other OS where you could even go on such a trip so easily! Figuring out why things are where and being able to disable them like this is pretty cool.
I was really surprised you were able to replace systemdMinimal with systemd in dbus though.
I thought it was there to break the cyclic dependency between systemd and dbus
He didn't mention that nix uses a lot of ram. If your server is tiny and doesn't have swap enabled, running nix command s will make it unresponsive. Are there any nixos alternative that allow you to do system wide configuration from a single source similar to configuration.nix?
(I'm the post author)
As others said, I've moved away from doing nix builds on servers and into a less wasteful (if you're running multiple servers) approach of building once, deploying the bits into a cache, and making the servers fetch them from a cache. I've been slowly working on my own tool to make this workflow easier (nixless-agent, you can find an early version on GitHub).
You need not perform your nix evaluation on the same device you are targeting. You can nix copy a system closure to your target and activate it, and there are a number of tools in the nix ecosystem to make this easier.
Granted, if you local machine is low on RAM, or isn't Linux, then you will be in trouble.
If you have a different machine with more ram and compute, you can use 'nixos-rebuild --target-host=<server> switch'. That does all the nix building on the local machine, and then just copies binaries and activates the built configuration on the remote machine.
I don't run nixos on my PC
you don't need to have nixos to have nixos-rebuild installed. Just nix
Not similar to nix, but you can look into Yocto. You can use it to generate an OS. It is, much more involved than using nix, but suitable for low memory environments.
The problem with nix is that compiling a system uses lots of memory, but when deployed there's little overhead.
Like you would with Yocto, I just build my systems on a proper host then remotely deploy them.
Yocto is also used by Lenovo to build their software appliances like XClarity Administrator.
Also it’s very famous and loved in embedded software circles.
Well, "loved" is not the word I would use. It's an improvement over the chaos before, where everyone used some other bespoke tool to build their BSPs. Yocto at least is some kind of standard now, but not a particularly good one (granted, it's a really difficult problem to solve). I don't know anyone actually enjoying working with Yocto.
Yocto is also hated for its unnessesary complexity in embedded software circles.
I haven't tried it personnally but Guix is similar with config in guile scheme. Have a look at the documentation [1][2].
Caveat: it is a gnu project so no proprietary stuff like firmwares and drivers included out of the box (but there is a community guix nonfree project available [3]). I believe that isn't a problem for virtual machine servers anyway.
[1] guix cookbook: https://guix.gnu.org/cookbook/en/html_node/index.html#Top
[3] guix manual: https://guix.gnu.org/manual/en/html_node/index.html#Top
The widely used nonfree guided channel is https://gitlab.com/nonguix/nonguix, not the one you linked.
author does not use nix on his nixos and removed it, so no ram issue.
author does not activate new config on host machine, but deploys new host machines as needed.
he evals on build/dev/ci machine only.
It won't necessarily do that if you deploy to your server from a remote machine over ssh
guix?
Doesn't it use nixos underneath?
guix does not use nixos underneath, it's a completely independent implementation (of the same underlying concepts)
Have there been any good efforts into getting rid of the language or providing an alternative? My effort into switching and the biggest complaint I've read is that the idea behind NixOS and the ecosystem is genius but one of the biggest draw backs is writing everything in Nix.
I spend a weekend every so often defining the core of what I want next time I upgrade, but just find it so annoying I'm sure I won't use anything I've written until there's a major change in the ecosystem.
What's so bad about it?
Putting aside the poor typing (the lack of proper typing is a shame, so valid criticism), I actually really like the language - it's genuinely a great DSL for the particular problems it's supposed to handle.
It does take a bit of use for it to click, though. A lot of it has to do not with Nixlang itself but about learning nixpkgs' idioms.
It's not just poor typing. It's the lack of discoverability. "What are all my options here in this expression? What are the symbols available?"
I think a good IDE integration could solve this, but not sure how much is possible.
The nix repl can be a very valuable tool in answering these questions.
That said, I strive to structure my nix source so that portions of it can easily be pasted into a repl. ReadTree goes a long way in that regard: https://github.com/tvlfyi/kit/tree/canon/readTree
More to your point, though: I think a lot is possible. Although nix is very dynamic, it is also, for all intents and purposes, side effect free. I've had this idea that a sufficiently advanced IDE should be able to evaluate your nix code and tell you exactly what the possible values (not just types, but value!) are for any particular variable.
> I've had this idea that a sufficiently advanced IDE should be able to evaluate your nix code and tell you exactly what the possible values (not just types, but value!)
Similarly to the REPL, I'm often using `nix-instantiate --eval -E 'somethingsomething'` so it should definitely be possible.
There is this: https://github.com/nix-community/nixd
It has jump to definition and autocomplete. Which is very nice.
It's not perfect. But it's pretty good
Way too much sugar/"idioms", which makes it hard for someone new to the language to figure out what a given piece of code is actually doing. Confusing use of semicolons for what almost every other language uses commas or newlines (or nothing) for. It's the same feeling as writing bash, and needing to always look up again exactly what the syntax is and where the semicolons go.
There is Guix, which replaces the Nix language with Scheme, but which has some limitations related to a smaller user base, e.g. a smaller package collection.
Replacing the language requires duplicating all the work that went into Nix, to reach parity, so it is not easy.
> Replacing the language requires duplicating all the work that went into Nix, to reach parity, so it is not easy.
That seems like a design flaw in Nix, there's no reason the data model should be so tightly coupled to the scripting implementation that you can't reuse packages written in a different language.
There is no technical barrier against doing that. But much of the power and flexibility in nixpkgs arises from the nix language, not the data model (which is comparatively simple).
For example, see zb: https://www.zombiezen.com/blog/2024/09/zb-early-stage-build-...
Using a different language to depend on packages derived from .nix would be very much akin to depending on a docker image whose Dockerfile you can not inspect.
It seems to be an issue with testing and debugging, rather than the language itself. The same issue would also be present if you could switch to any other language for configuration.
Nickel lang is such an effort. Id say the syntax is a mix of json and lua and aims for a non-touring complete program. It is still a bit early but it looks promising
No, Nickel is Turing-complete. That's been one of the characteristics intended to distinguish it from most other configuration languages from the start.
See the 'RATIONALE' document: https://github.com/tweag/nickel/blob/378ece30b3e3c0ab488f659...
> Have there been any good efforts into getting rid of the language or providing an alternative?
Guix is conceptually similar to Nix but uses scheme.
The post is much more interesting than the title suggests. Great deep dive into slimming a NixOS system.
I agree. The title made me expect a low-quality, clickbait article. The actual article provides a lot of educational value to users of Nix and those interested in trying to build the smallest Linux distribution that can run Nix. Despite using Nix (the package manager) for so long, I haven't considered what it would take to get this. The article answers the question perfectly.
> I was trying to bring NixOS to a bare minimum, which is an exercise similar to building containers with the bare minimum required for the software in the container to run. I think this is a worthy endeavour. I think we have all the tools in regular non-docker, non-kubernetes linux to get to a similar outcome, except we won’t need docker or kubernetes or whatever in this new land, thus removing quite a bunch of complexity from the systems we build.
> But doing it on top of NixOS currently feels like a bad path to take.
The author of this blog post might be interested in playing with not-os, another, much smaller OS built with Nix: https://github.com/cleverca22/not-os
Could be a decent source of inspiration!
(I'm the post author)
Thanks! I have to admit that I've had the itch to build my own NixOS-inspired system more than once, and I haven't done that because I just don't have time to dedicate to this among all the other projects I'm working on. I wasn't aware of not-os before, but I'll definitely dig into the code!
Have you taken a look at https://github.com/nix-community/srvos ? It’s a collection of profiles for servers by the Numtide folks.
One technique employed by microvm.nix[0] is to mount the hosts /nix/store into the guest. This won't shrink the size of the system, but should allow it to be amortized across many different VMs.
I'm not sure how exploitable a read-only virtiofs share is, so this is perhaps not appropriate in some circumstances.
I just finished moving all of my servers off NixOS and I'm finally breathing a sigh of relief.
Deterministic systems are a cool idea, but we're just not there yet. The headaches and pain involved in maintaining these systems and warping the software to obey are too great.
Everything in NixOS works, until it doesn't. And when it doesn't, woe be unto you.
What did you replace it with? I've been been using NixOS for a (non-critical) server which replaces another server that was "managed" by Ansible. Now that's it running nicely on NixOS, and while it's far from perfect, I'm really struggling to see how I would manage it in any other way. The experience is so much nicer now.
I'll echo this. I can't imaging going back.
And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.
I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible).
I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).
I've basically gone full circle at this point:
- Docker
- LXD with Bash scripts
- LXD with a ton of Python
- NixOS
- Proxmox with Ansible
- Proxmox with Bash scripts (albeit much simpler and flatter than last time)
Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.
One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.
Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.
Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.
As someone who uses Proxmox without bash scripts and is scared of the day I have to re-work out all the config files I (often randomly through trial and error) changed:
Do you have any tips on how to get started? Do you simply make the change and then paste the commands needed into a script?
Also I assume you have a script to set up the (Proxmox) host machine?
I also have quite a few Proxmox specific things that I had to change (e.g. GPU pass-through) which seems to break your "Now it doesn't matter what technology (container or VM) I use" advantage.
Proxmox really needs to get their cloud-init story together. If they supported cloud-init for LXC a lot of automation and setup issues would just go away…
Couple of things spring to mind:
- didn't check if nixos uses it, but coreutil has a single binary mode (like busybox, a single binary is built and symlinks or hardlinks are used to provide all the commands); that might save some space
- instead of trying to strip down the system, maybe go the other way around: only include the command you need with its closure? closure computation is done in a few places (apparmor profile or systemd.confinement come to mind) and it should be possible to just copy whatever your server binary needs, your kernel (since microVM and not container), and run the binary directly as init (maybe with a simple wrapper that hardcodes network or whatsnot)
good luck!
> - didn't check if nixos uses it, but coreutil has a single binary mode (like busybox, a single binary is built and symlinks or hardlinks are used to provide all the commands); that might save some space
It does. If you have coreutils from Nixpkgs installed you can check this with
basename "$(realpath "$(which ls)")"
> go the other way around
I'm not sure about just the one closure since the result needs to boot, but more generally that looks reasonable if your use case fits. There's a couple efforts, IIRC, mostly centered around router firmware where "real" nixos doesn't fit, ex. https://gti.telent.net/dan/liminix
2 is very interesting to me. I'd love to see an example of a "scratch" vm build, in a similar spirit to scratch docker builds. Assume I have all my software and kernel built, what is the minimal filesystem that will boot? How much easier can it get when I am only targeting a VM?
This might be something you're interested in: https://appfs.rkeene.org/web/dir?ci=trunk&name=build/qemu
It's for running AppFS inside a VM on QEMU. It uses a statically linked Tcl (which AppFS is written in) to bring the system up.
Taking the time to delve in to things like this can teach you more about the OS you run than many other things can. It's not too unlike making a crunchgen OS, or even just a read-only OS with overlays. It's sometimes quite enlightening to see what will work with much less than we assumed it needs.
Having just installed the entirety of NetBSD on an i386 system (a 200 MHz Pentium), I see it weighs in at around 1 gig. But that's everything including X11 with WM and the toolchain (gcc 10). That's not bad, but it's really amazing how much of that isn't necessary for running the OS on a server. Particularly where you might want tiny VM images
Disk space is why one of my laptops doesn't run NixOS; 16GB soldered storage fit one generation passably, but it was clear that it wasn't gonna work (of course, that's with X and a browser; totally different use case). I vaguely recall that locales were shockingly large and I couldn't figure out how to trim them down. So that box runs Alpine now, which easily fits with no effort.
I too use Alpine. I like how you start small and add what you need. The option of having a read-only root FS is convenient on flaky SBCs.
Have you tried putting /nix on an filesystem with compression? I've experienced >50% in storage savings with Bcachefs + zstd and dedupe
Good stuff! From the title I thought this would be about sysadmin woes, but this is interesting too :)
Another little addendum: you can trivially create bootable custom NixOS installer images with whatever configuration you want pre-applied[0].
[0] https://nix.dev/tutorials/nixos/building-bootable-iso-image
This reminds me a little bit of trying to slim Docker images.
This is highly relevant for me, will probably end up using some of the same tricks they've done to trim the size of some of my micro VMs.
Could you run ’nix store optimise’ before you remove ’nix’?
Any OS without stable releases and a security team is not a server OS.
NixOS does have "stable" releases every 6 months, but they don't have long term support.
Then you only have Ubuntu, Fedora, Red Hat/CentOS/Alma/Rocky and Suse/OpenSuse.