<gchristensen>
any suggested reading material? I'm definitely out of my depth
<gchristensen>
I hear hairpin is a better option than split horizon DNS
<Shados>
AKA nat reflection... I have no idea why this has so many names.
<Shados>
And it is better than split-horizon, yes, because some services will only work with IPs and not DNS under the hood
<Shados>
gchristensen: Does the system have a fixed/static WAN-side IP?
<gchristensen>
I wish :(
<gchristensen>
it is "very stable" but not fixed
<Shados>
OK. Makes it a little tricker to get NAT reflection to work. What kernel are you on?
<Shados>
As in version
<gchristensen>
4.19.36 but could go to 5.0.9
<Shados>
Will take me a minute, my stuff is using nftables and I can't quite remember what the module I'm looking for is called in iptables
<gchristensen>
thank you :o
<cransom>
fwiw, i use split horizon dns because i like to see the real source ips in my logs internally. and being that i have native ipv6 as well, i can get into those services without vpn and everything just usually works.
<gchristensen>
IPv6 is too unreliable around here :(
<cransom>
the its been good for me on spectrum/timewarner provided i don't touch it.
<gchristensen>
basically no public wifi here supports it
<cransom>
oh, i rarely find a working v6 wifi, but at&t is workable so i can get to my home boxes via the cell network at the least.
<gchristensen>
ah heh
<gchristensen>
how have you setup your split DNS?
<gchristensen>
anything fancy, or just simple rules and static IPs?
<gchristensen>
simple records*
<Shados>
to 192.168.0.8
<Shados>
OK, so you need a way to tell iptables/linux to DNAT all packets going specific ports on your external IP, regardless of the source of those packets (WAN/LAN). This is easy if you know your external IP a prior, because you can just use a `-d your.ip` match on it. If you don't, then your DNAT rules have to get clever with something like this: iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL ! -d 192.168.0.1/24 -p tcp --dport 80 -j DNAT --
<Shados>
That rule basically says "match all addresses assigned to a local interface, excluding the ones in this subnet", the subnet being your LAN. Which naturally should include only your WAN IP and loopback.
<gchristensen>
oh very fancy
<cransom>
i have very few exceptions really. technically i suppose my split horizon isn't all that split since dnsmasq sets up records for v4 and v6. v6 always works, if there's a v4 i need to nat inwords, i set dnsmasq to respond to that record with its public address and nginx forwards it to the right place
<Shados>
And the kernel version bit doesn't matter, I'd been thinking about nftables again. The functionality on the `fib` module used to do this in nftables was only actually working in a somewhat recent kernel, I think
<Shados>
The only other thing is I think the default NixOS NAT rules only do SNAT on packets going out the external interface, so you'll need another POSTROUTING rule to handle the internal ones. If the device is purely a router and isn't doing double duty as your LAN switch, you can probably get away with just MASQUERADE'ing all traffic from internal IPs to the port forward destionations. If it is acting as your LAN switch, then that would inadvertantly
<Shados>
rewrite source IPs on LAN traffic going directly to those destination ip/port pairs though...
<Shados>
NAT makes me sad >.>
<gchristensen>
hear hear :(
<Shados>
I think you could make the POSTROUTING rule appropriately selective by marking the hairpin traffic you're doing DNAT on in the PREROUTING rule I gave earlier, and then only do the MASQUERADE on marked packets, but I haven't tested that
<clever>
gchristensen: i would prefer iptables-restore over iptables, but that requires a major redo of the firewall system in nixos
<clever>
behind the scenes, linux is using an RCU list, read->copy->update
<clever>
so, every time you want to mutate the fw rules, you must copy the entire fw config, mutate the copy, then do an atomic swap to the new rules
<clever>
running `iptables -A` will get slower with every single invocation, as the rule list gets bigger and bigger
<clever>
iptables-restore will atomicaly do everything in one go
<gchristensen>
if we do that we might as well just go to nftables
<cransom>
if this was still tcp, you can also flip on a socat and layer 4 forward your connections around.
<gchristensen>
god yeah I was just thinking that
<cransom>
i might have done that for a very long time in a galaxy far, far away.
<cransom>
socat might also do udp return traffic too, that is not a place i've needed to go.
<Shados>
cransom: Heh, we have socat doing that for some things at work... network guy got lazy :P
<cransom>
sometimes it lazy. othertimes, it really is just a pain in the ass. because, yeah, nat. v6 is your savior.
<gchristensen>
ok fine I'll v6 ...
<gchristensen>
and deal with the fallout when I have to
<Shados>
clever: Actually, I had an iptables-restore implementation of the NixOS firewall, and rewrote it as an nftables-based one. At the time nftables was still fairly rough around the edges (some modules outright not working etc.), but it's pretty good in recent kernel versions. Needs more documentation though. iptables-restore one: https://github.com/Shados/nixos-config/blob/master/bespoke/services/sn-firewall.nix nftables one: https://github.com/
<Shados>
Each of the ethernet ports has a separate Intel 82583V network chip, and the CPU supports VT-d, so I'm thinking about passing each through to individual VMs for overkill isolation/security
<clever>
Shados: i would more do one vm per vlan :P
<Shados>
No no, it just starts at one per NIC. One totally minimal VM per chip, just to handle the hardware, then those feed off to as many as needed internally for shenanigans. Probably the internal ones can be MirageOS unikernels, for extra coolness/insanity.
<Shados>
First I need to finish getting the Xen dom0 module booting on EFI
<Shados>
Which has been a learning experience. I have learned primarily that grub and Xen are not friends, and apparently basically no one puts them in a room together.
<Shados>
...I've also re-learned that I don't like Perl.
<clever>
Shados: xen also has driver domains, where you basically have a dedicated guest for talking to guests
<clever>
normally, for each device type, you have a backend on dom0, and a frontend in the guest
<clever>
xen_netback deals with the vif45.0's on the host, that tunnel into the guest
<Shados>
Oh yeah, driver domains look like pretty much what I want, lovely
<clever>
and xen_netfront (or e1000, depending on emulation), deals with exposing that to the guest
<clever>
but, this is about the guest<->guest api, rather then hardware<->guest
<clever>
its more to prevent bugs in xen_netback from exploiting the real host
<clever>
also, for hvm guests, there is a neutered qemu instance for every guest
<clever>
a special fork of qemu, that lacks a cpu, and talks to the xen hypervisor to get IO activity
<clever>
so when the guest tries to do IO, qemu gets an event, and can emulate the hardware with existing code
<Shados>
FWIR I remember netback is implicitly trusted by netfront, hence the idea of sticking a mirageos vm with less-trusting netfront implementation inbetween any Linux VMs
<clever>
ive also played with pci passthru in xen
<clever>
i was able to get windows7 booting under xen, with full gpu passthru, once
<clever>
then i realized, the hdd was too small to install a single game
<clever>
and after shuffling bytes around to make room, it never worked again
<clever>
blame that on my pre-declarative days :P (gentoo)
<clever>
biggest problem, is that the windows drivers expect the GPU to be in a very very clean state (like, only the raw bios api has ever touched it)
<clever>
even something as simple as fbcon breaks the state enough that windows cant bringup the card!
<clever>
grub is fine (otherwise, it would break when chainloading win7)
<Shados>
Depending on mobo and card, you can potentially do a uh... I forget the technical term, but you can basically have the PCIE bus power cycle the card
<Shados>
I used to do this to let me re-attach a GPU back/forth between a windows KVM guest and a linux host (on a second X server)
<clever>
so, i needed a special xen/linux param, to allocate the entire GPU to the xen_pciback driver
<clever>
no per-slot reset on this mobo
<Shados>
Ah
<clever>
that might also make the state TOO clean
<Shados>
kek.
<clever>
the bios has to bring it up on power-up
<clever>
there are special headers you can put into roms in the pci address space, and the bios will just blindly execute the code on powerup
<Shados>
I gave up on using Xen for GPU passthrough. Tried it out in the very early days when only Xen was capable of it, but moved to KVM pretty much as soon as that could do the job. Alex Williamson did a hell of a lot of good work making that saner to do.
<clever>
oh, the other major problem
<clever>
once windows has used the gpu once, thas it
<clever>
you must reboot the entire host to reset it
<clever>
so, you now have the stability of windows :P
<Shados>
And yeah, after doing slot resets on the card I *did* have to shove a card firmware/bios image at it to get things to work...
<Shados>
And no, slot resets let me un-attach it from windows OK, just had to step through things in precisely the right order or all hell would break loose
<clever>
i had far far better luck passing usb, ethernet, and audio devices around, at runtime
<clever>
i could play hot-potato with any of those, zero trouble
<Shados>
GPU passthrough is a pain to maintain. Eventually gave up, set up a dedicated windows/gaming box, plugged it into the same monitor, used synergy to share keyboard/mouse, hacked up a script to pipe audio to/from windows and pulseaudio, and figured out how to use the venerable ddccontrol to swap monitor inputs on a keyboard hotkey ;).
<clever>
i just dual-boot
<Shados>
I dual-booted for a long time, but I don't like having to dump contexts whenever I want to switch between them...
<clever>
also, i have a neat usb box, that is meant to share 1 printer between 4 machines
<clever>
it has 4 device ports, and 1 host port
<clever>
the 4 device ports, then go to 4 computers
<clever>
and the 1 host port, is meant to go to a printer
<clever>
then you just push a button to cycle which host has control
<clever>
but who says you cant USB hub a keyboard+mouse thru it? :P
<clever>
Shados: same, thats why i play the windows-only games once a month
<clever>
or less
<clever>
proton and steam have also gotten better
<Shados>
Ah, I was super satisfied when Proton came out
<Shados>
More than a decade ago I predicted to a mate that Valve would both bring Steam to Linux and eventually integrate and improve Wine, so I had a huge "I told you so" moment :3
<clever>
:D
<clever>
a week ago, somebody in #nixos asked in skyrim works
<clever>
so i opened up steam and hit "go"
<clever>
about an hour later, i was back in an old save i had long forgotten, and everything worked, lol
<clever>
i dont remember steam having cloud-sync last time i even played it...
<Shados>
OK, I've gotten Xen + grub working under EFI. Now for gummiboot... which will pretty much be the same, as the multiboot2 route did not work with grub =/.
* cransom
pictures a 2 port kvm but with both ports plugged into the same machine and chuckles.
<Shados>
I think that code may have just bitrotted. I could only find two examples of people actually using EFI grub2 -> multiboot2 -> xen in the wild...
<clever>
cransom: reminds me of idiots who plug in both dvi and vga, then wonder why the mouse keeps disapearing
<clever>
Shados: you need to do what ive done, source level debug :P
<Shados>
I'm not sure it's even a grub issue, though. If I take the multiboot2 route, the serial console shows Xen starting, but it reboots the VM I'm testing in (nested virtualization, heh) part-way through printing the EFI RAM map
<Shados>
Booting xen.efi directly from a UEFI boot entry, or via grub with a xen.cfg works OK, but not grub -> multiboot2 -> xen.gz
<clever>
Shados: what about telling grub to load it as an efi binary?
<Shados>
xen.gz is not an EFI binary. xen.efi is, which is the route that I have working right now.
<clever>
ah
<Shados>
But that way is a bit annoying, as you then need a xen.cfg next to the xen.efi file, with the kernel/init/paramaters specified inside it, and the xen.efi and xen.cfg have to be named that (simplifying).
<clever>
how far does xen.gz get in efi grub?
<Shados>
Like I said, it starts Xen but then that crashes/resets part-way through printing the EFI RAM map during its start-up process
<clever>
Shados: pull up that part of the xen source, and see what its doing in that area?
<clever>
does it show a clear marker when its done printing the ram map?
<clever>
try adding one?
<Shados>
I think technically it is crashing immediately after printing the EFI RAM map, just prior to printing the amount of discovered system RAM, but I'm not super interested in solving the problem. While the currently-working way is more annoying, it's also the only way that will work with gummiboot/systemd-boot as well as versions of Xen <4.9
<Shados>
But hey, if you want to fix it I can upload a broken configuration.nix and accompanying modified nixpkgs :P
<clever>
could maybe have a look at it
<Shados>
Aight, I'll upload some stuff in a bit then
<Shados>
the vm needs to be an OVMF/UEFI one of course, you need 'host-passthrough' cpu for nested virtualization (and hardware that supports nested virt), and I have the serial console logging to /tmp/xen-testing.console.log
<Shados>
The nixpkgs branch includes the changes to make the Xen package build the Xen EFI binary as well, in case you want to test with that additionally
<Shados>
Yeah, I have some stuff that directly builds disk images too, I just find it a bit too slow for development purposes
<clever>
same
<clever>
but linux cant use virtio-9plan when under xen
<Shados>
I just rsync some changes over and rebuild within the host :p.
<clever>
i was going for pure testing of xen unikernels inside nix-build
<clever>
so a persistant disk image wouldnt play :P
<Shados>
Makes sense
<Shados>
It sounds like xen >= 4.9 has support for a 9pfs transport?
<clever>
i wanted the 9pfs from the outer qemu to work
<clever>
but, linux doesnt expect you to run linux in xen in qemu, and talk to the qemu directly :P
<clever>
the problem, is that if linux detectss that its a xen guest, it "cant" also be a qemu guest!
<Shados>
Can't you run the xen 9pfs backend in the host qemu, rather than the native qemu one...?
<clever>
the true host is supposed to lack xen
<clever>
this was meant to be something to let you test xen based guests, without needing real xen hosts
<Shados>
So the xen/9pfs backend that is part of qemu is dependent on being in a xen domain? Makes sense, I guess, albeit annoying
<clever>
thats my guess
<Shados>
yeah, it looks like it is dependent on xenstore and some other xen domain-specific stuff
<Shados>
And the other way would require convincing xen to detect and pass through a virtio transport to a domain
<Shados>
VirtIO is exposed within qemu guests as a PCI device, apparently... can you get Xen to pass that to a domain?
<clever>
all "host" devices are available to dom0 by default
<clever>
the problem, is that when linux beleives its under xen, it doesnt enable qemu things
<Shados>
Oh right
<Shados>
That seems like it wouldn't be too insane to fix
<gchristensen>
I fought with ipv6 for several hours last night
<gchristensen>
eventually did a video call with flokli this morning
<gchristensen>
turns out my problems are from my ISP
Guanin has joined #nixos-on-your-router
Guanin has quit [Ping timeout: 250 seconds]
<gchristensen>
I found a real-life shiboleet
Guanin has joined #nixos-on-your-router
<sphalerite>
gchristensen: ooh, what's that?
<sphalerite>
(I know the xkcd, my question is what the secret is word is)
<gchristensen>
"can you please ask your supervisor what is the maximum prefix delegation your docsis headend supports?"
<cransom>
haa.
<cransom>
that would have been an amazing moment if they didn't skip a beat and said "/56, sir"
<gchristensen>
hehe
<gchristensen>
the person I talked to could n't sort out if 64 -> 56 was an improvement or not
<gchristensen>
and also thought it had to do witd bandwidth
<cransom>
you need all the ipv6s for teh games.
<gchristensen>
hehe
<gchristensen>
anyway
<gchristensen>
I can't convince it to givem e more than /64
<gchristensen>
but they said a /52
<cransom>
a /52 seems slightly odd, but maybe.
<andi->
do you have a PCAP of the request/rplies?
<gchristensen>
pcap no, but I have tcpdump log
<gchristensen>
or I did, and could get it again
<gchristensen>
(or could get a pcap...
<andi->
There were some semantically correct but yet treated invalid requests from some CPEs.. so maybe their vendor has a broken implementation now..
<gchristensen>
I'll get some info in a few minutes. doing some pre-dinner prep
<andi->
no hurries, I'm on something else anyway
<gchristensen>
tcpdump -i enp1s0 -n -vvv '(udp port 546 or 547) or icmp6' -w mycap.pcap lgty?
<andi->
add an -s 1500
<andi->
to capture the full length of the packet
<andi->
*s
<gchristensen>
^ I sent the pcap to andi- directly, not sure I want to send a pcap to the world. if anyone else wants to see it, I can link it to you :)
<andi->
as soon as wireshark starts on wayland I'll have a look :D
<andi->
gchristensen: only see requests for a /64 despite the config you linked
<gchristensen>
...huh
<gchristensen>
tbh I don't fully grok what I'm doing there
<andi->
haven't used dhcpcd in a while.. Looking at the man page..
<andi->
gchristensen: try setting the requested prefix length to 0 since that seems to compute the proper required size and send it...
<gchristensen>
so 2/::/56 -> 2/::/0 ?
<andi->
worth a shot
<cransom>
i blame and credit gchristensen for making me log into my router and doing things where i noticed the disk io was horribad and i found fstrim was not doing its job.