<gchristensen>
I want an ipxe that understands nix store paths
<gchristensen>
or a kernel that understands nix store paths
<gchristensen>
fetching everything but an ultraminimal startup environment inside Linux, with maximum parallelization
<gchristensen>
don't make me build a fat initrd
<flokli>
gchristensen: wrong channel? But I hear some HPC people run the nix store from NFS ;-)
<gchristensen>
I didn't know where else to go :P
<gchristensen>
why do you tell me scary stories of NFS when I'm telling beautiful stories of the future :')
<flokli>
gchristensen: okay, build a fuse file system that substitutes from the binary cache on-access and mount that as the nix store in initrd :-P
<gchristensen>
I more want, like
<gchristensen>
not sure, a very slim initrd which accepts a store path which it substitutes at the start
<flokli>
but usually initrd is somewhat self-contained, and provides networking modules
<gchristensen>
yep
<andi->
just bake an initrd that does nix --experimental-features "nix-command" $storePath && exec $storePath/bin/init? ;-)
<andi->
add the build in there that I forgot with all the new boilerplate
<gchristensen>
yeah!
<andi->
I want something similar but as initial kexec argument to bootstrap servers that don't support iPXE or nixos out of the box.
<gchristensen>
yeah!!
<andi->
I think for the generic use case being able to supply a URL to a nixos configuration would already go a long way. Having a store path requires a custom binary cache that is populated with each of these images, custom trust settings for the images etc... While what I imagine should be a building block anyone can use. With a custo nixos configuration you can just do whatever you need.
<andi->
You might require to kexec (twice) in the process but that is still better than the current status.
<andi->
I wonder how functional our current netboot image is if we remove the installer bits..
<andi->
A static version of NIx would be amazing.. We could just append that + kexec to the kernel and have a "small" bootstrap bundle
teto has quit [Ping timeout: 244 seconds]
<andi->
Not really keen on looking into dupliacating/touching the code we have to create initrds that strips the nix store path.
<gchristensen>
lovethissss
* andi-
searches for a valid root store path on hydra to kexec into
<gchristensen>
:D
<mdlayher>
it has been a few months and i just wanted to mention how much i love NixOS on my router
<mdlayher>
highly recommended. now if only they'd release a version of this with a Zen CPU
<andi->
do we have a better way to "guess" the initrd of a hydra netboot image than using string matching? i.e. from this output: https://hydra.nixos.org/build/119237890
teto has joined #nixos-on-your-router
<mdlayher>
not that you need a Zen CPU for routing, but it'd be great for Nix builds or anything that needed more punch. in the mean time I use my server with a Ryzen 7 2700 for distributed builds
<andi->
gchristensen: thoughts/requirements on bringing up the networking?
<gchristensen>
IMO safe to assume dhcp
<andi->
bonding, vlans, …
<andi->
but yeah, lets just run dhcp on all interfaces for now..
<gchristensen>
yeah
<andi->
The netboot image without the installer part is still >300MB.. Really hating this
<andi->
So custom initrd..
<andi->
err >400MB
<gchristensen>
isn't it amazing how bulky it is?
<gchristensen>
Nix makes it easy to get exactly what you need, but makes it easy to accidentally need to much :)
<gchristensen>
with debian you have to rethink your life choices when you forgot to add gtk to your netboot
<andi->
it isn't that bad :P
<andi->
but yeah
<gchristensen>
hehe
<andi->
I think I'll see what I can do about shaving off size on the installer image
<cransom>
i really hate running routers with under powered or 'right sized' cpus. it feels like i'm running commercial hardware again where that main, ginormous big, switch/router has a 200mhz processor and a whopping 32megs of ram.
<gchristensen>
+1
<andi->
These days they come with plenty of RAM and just run their old FreeBSD based systems in a QEmu process on linux...
<andi->
not sure if that is *much* better
<cransom>
it's still typically 10 minutes between reboots due to all the hardware initialization that typically happens. i was at a dog and pony show where juniper was explaining how they goofed up and the linux thing was around the corner.
<cransom>
i was disappointed how they handled freebsd. instead of keeping up, they locked in on 4.11 so when it did finally get features they would have really been interested in, it wasn't possible.
teto has quit [Ping timeout: 260 seconds]
teto has joined #nixos-on-your-router
<mdlayher>
out of curiosity, has anybody had luck putting NixOS on a switch? thinking about my router experiences, if I could somehow flash my Ubiquiti switches to run NixOS I'd be so thrilled.
<mdlayher>
i suspect ubiquiti may be a no-go due to proprietary hardware acceleration stuff
teto has quit [Ping timeout: 252 seconds]
<cransom>
not that i'm aware of. the best bets would be on hardware that cumulus/other whitebox os's work on, but i'm sure there's still secret sauce in there that makes it painful.
<mdlayher>
yeah that's what i figured. bummer!
<mdlayher>
thanks for the info. i wonder if pcengines would ever make a switch board like 8+ NICs.
<cransom>
i think it's a very small market segment that are interested in it. small switches are cheap, 1 or 2 nics is usually plenty for routers. but eh, some day maybe.
<mdlayher>
yeah totally.
teto has joined #nixos-on-your-router
qyliss has quit [Quit: bye]
qyliss has joined #nixos-on-your-router
teto has quit [Ping timeout: 260 seconds]
teto has joined #nixos-on-your-router
<andi->
my bet would be that on an MLX spectrum switch it just works :)
<andi->
since there you got mainline kernel driver for >100G hardware routing
<andi->
they are a bit more expensive than those FS switches you can buy but IMHO that pays off if you need the flexibility. These days they also support full tables
teto has quit [Ping timeout: 272 seconds]
<cransom>
i'd be super into having a switch at home that was linux powered, but they are all tuned for the data center. my garage isn't quite hygienic enough for that though.
<danderson>
eeh, they're robust little things switches
<danderson>
mdlayher: for anything above 1Gbps, your sticking point is going to be Broadcom switching ASICs
<hexa->
and broadcom is meh wrt open source
<danderson>
the market for >=10G switching is dominated by Broadcom's line of network ASICs: Scorpion, Trident+, Trident2, Tomahawk, Jericho...
<danderson>
broadcom is "lol no" with open source
<danderson>
at least for the high performance network stuff
<danderson>
there is zero open source support for any of these chips
<danderson>
and you'll need an NDA and $$$ just to get SDK access
<hexa->
which is why andi recommended mellanox spectrum :)
<danderson>
and $$$$$ and a "special relationship" if you want the datasheet with the PCIe registers that you need to implement a driver
<danderson>
yup. Downside is that those haven't made it to the used market yet afaik
<mdlayher>
ugh broadcom ugh
<danderson>
so expect to pay 4-5 figures to get one