<gchristensen>
clever: have you done something wacky with ONT, 802.1x auth, and ISP routers?
<betawaffle>
paging clever and/or andi-: i was told one of you might be able to help me with my crazy router idea
<gchristensen>
=)
<betawaffle>
i'm trying to do something similar to what a bunch of other people have already done, to some extent. but i'm trying to do it in a different way, and having some trouble. and just generally i need more knowledge about what i'm working with
<betawaffle>
the "crazy" part is i'm trying to use only: systemd, networkd, and nftables, all on nixos
<betawaffle>
basically trying to break ground on a strategy that may not be ready to completely work
<betawaffle>
all while not really knowing enough to quite figure it out all by myself :P
<betawaffle>
i have gotten pretty close, though=
<betawaffle>
so, now for some background/context: i've got an apu2d4 (basically exactly the same as an apu2c4, so y'all are probably familiar with the hardware already)
<andi->
is that .1x a requirement? I dobut networkd will be able to do that
<betawaffle>
i've got gigabit fiber from AT&T, that goes into their ONT mounted on my wall
<betawaffle>
yep, afaict, the at&t router needs to auth with the ONT
<andi->
you'll have to add wpa_supplicant (or similar, iwd maybe) to the mix
<betawaffle>
(that's the only 802.1x, none of my own)
<betawaffle>
so, what i'm doing now is bridging the ONT and the AT&T router through my apu
<betawaffle>
it's configured to forward the 802.1x frames correctly
<betawaffle>
and that all works, i have internet on the other side of the AT&T router
<andi->
and now you want to remove the ATT router?
<betawaffle>
but! what i want is for my APU to have internet directly from the ONT, and just. yes
<betawaffle>
just use the ATT router for that auth it needs to do if i replug stuff sometimes
<andi->
depending on HOW it does authentication it might not be feasible (without breaking into the routers flash)
<betawaffle>
the auth works fine, my problem is injecting regular traffic into the bridge without fighting with the AT&T router
<andi->
ah
<betawaffle>
ie. i have to spoof the ATT router's mac on a vlan
<andi->
I would probably try to remove the bridging at all and have a user space process work as proxy.
<betawaffle>
(add vlan0 to the bridge with the ATT router's mac, which gives me fully working internet and DHCP)
<andi->
That would remove the issue with mac collisions etc..
<betawaffle>
yeah, that's what other people have done. but can i do that proxying in the kernel instead?
<betawaffle>
with nftables or whatever
<andi->
basically l2 dnat/snat? :D
<betawaffle>
maybe? i don't know enough about that
<andi->
you could potentially just drop all the frames but 802.1x frames on the bridge towards the real device
<andi->
not sure what the kernel will think of that hack as the mac address packets go to is local.
<andi->
I usually avoided such l2 hacks if possible.. It is so hacky that I always fear some kernel change will break it and nobody but me notices it :/
<betawaffle>
there was an ebtables rule i found that does that filtering, but i can't figure out how to do that with nftables, and i don't know if that completely solves my issue
<andi->
basically what you want is policy routing for layer2..
<betawaffle>
again, maybe? i don't know quite enough
<andi->
it usually is a way to decided which layer3 routing table (ip r show table $num) should be used based on some set of rules that are applied to all packets. You can see the basic rules with `ip (-6) rule`
<betawaffle>
ah, yep
<andi->
I never heard of a concept like that in linux
<andi->
for l2 packets that is
<andi->
the newer bridge layers might have something for that.. Let me google around..
<betawaffle>
yeah, well... all i *really* need is for my APU to be like a regular router between two ports, and forward *very specific* packets to/from the third port
<betawaffle>
as you said, that can be done in userspace... but why not in kernel space?
<andi->
you can write a module for that if you are up for that :)
<andi->
it is so low traffic and latency tolerant that I would not even attempt writing (unsafe) kernel code.
<betawaffle>
well, i mean linux has all kinds of packet shuffling stuff. it seems like i should be able to put a rule *somewhere*
<betawaffle>
no, definitely not going to write a module
<andi->
true.. then one can still argue that most of the networking configuration magic (e.g. ipv6) should have been done in user space as it was orignally intended (most of us are using the "fallback" implementation these days).. :)
<betawaffle>
hmm?
<andi->
old guy grumping about technology ;)
<betawaffle>
i also don't know how i'd do it in userspace, besides using someone else's python thing i don't understand
<betawaffle>
which doesn't seem like a great idea, on my router
<andi->
you need two, one towards the ONT and one towards the router.
<andi->
not sure if you want to do BPF thought :D
<betawaffle>
yeah
<andi->
the older interfaces should also be around..
<andi->
s/older/simpler/
<betawaffle>
i literally want to match this EAPOL traffic on both ports and forward it to the other, as dumbly as possible
<betawaffle>
except it's possible the router needs more feedback than that... it might need to think it can DHCP and such too
<andi->
that I would fake entirely
<andi->
just send NACKs or something
<betawaffle>
i mean, it may not need that
<betawaffle>
we'll see
<betawaffle>
at the same time as all this, i don't quite understand how to actually configure standard routing between interfaces on linux, the way i would on some off-the-shelf router
<andi->
whats the confusion there? It is basically the same. Whats your background?
<betawaffle>
all my networking stuff is self-taught, and i've never really done much routing at all
<betawaffle>
this is also a learning experience for me
<andi->
on the easiest case it boils down to: enable forwarding on the involved interfaces and add routes for the subnets (usually implicit due to adresses on the interfaces)
<betawaffle>
like, i got this APU to replace my ubiquiti so i could learn this stuff
<andi->
if you already have a firwall configuration set up make sure that you allow traffic to flow in both directions in the FORWARDING chain
<betawaffle>
yeah, so i've basically got that already. but it wasn't quite working with the vlan thing i was doing the other day
<betawaffle>
(i need a vlan to get to the internet)
<betawaffle>
so basically i had a bridge, with three interfaces on it, two of the ethernet ports, and a vlan
<betawaffle>
then i've got *another* ethernet port which needs to be nat'd to the vlan
<betawaffle>
i think that was the part i hadn't figure out yet
<gchristensen>
maybe a graph would be helpful :P
<betawaffle>
yeah, where can i build one?
<andi->
draw.io?
<betawaffle>
k
<andi->
vlan aware bridges is something that I want to use on may next iteration of the home network routing... Still fighting some other issues with that hardware (since October...)
<betawaffle>
ebtables -t filter -A FORWARD -i enp3s0 -p 802_1Q -j DROP
<betawaffle>
ebtables -t filter -A FORWARD -o enp3s0 -p 802_1Q -j DROP
<betawaffle>
that's what i had from the internet
<andi->
ok, that works if the frame is vlan encapsulated
<betawaffle>
but i can't figure out how to translate that to nftables. i can use ebtables temporarily though
<andi->
try to get a PoC working and then make it prettier :)
<betawaffle>
yeah, the AT&T router encaps it in a vlan. and i'm supposed to as well
<andi->
ok
<andi->
I would probably add a VLAN device on each of the physical interfaces just to have less typing/indirections in my rules
<andi->
e.g. att.vlanX and otn.vlanX
<betawaffle>
i'm new to vlans, but i'll see what i can do
<betawaffle>
so... are you saying get rid of the bridge? or keep the bridge?
<andi->
keep the bridge for now
<andi->
but don't bridge the raw devices
<betawaffle>
oh interesting
<andi->
well you probably need to bridge the VLANs tho..
<betawaffle>
so, add vlan0 to each of the physicals, then bridge them?
<andi->
makes things less fragile with other vlans once you add them. The rules can be applied either on the bridge, the vlan interfaces of the raw interfaces if you really must enforce something
<betawaffle>
and i need a separate vlan netdev for each one, right?
<andi->
yes
<betawaffle>
k, sec
<andi->
unlike some hardware routers a VLANs only exist on their physical device.. Hardware vendors sometimes call this "unnumbered (vlan) interfaces".
<betawaffle>
should i give these vlans the nice names like ont and att?
<betawaffle>
or just suffix the device name?
<andi->
I usually suffix the device name after I renamed the devices to fit whatever they should be connected to
<andi->
if you come back in a year you'll have a much better idea of what is going on then
<betawaffle>
ok, so rename the devices stuff like ont0, att0, and lan0? or just ont, att, and lan?
<andi->
thats up to you
<betawaffle>
i know, but i can't decide!
<andi->
I had many many many hours of bikeshedding interface names with co-workers.. I wont get into that again.
<andi->
some common things; will and can there be only one? Don't add a suffix. Can there be more then one? Then always use a two character suffix.... and so on
<betawaffle>
ok, so then i'll go with no suffix for these
<andi->
some people say that going from 01 -> FF is the way to go and as soon as You have `100` all devices must have three characters at the end..
<andi->
:D
<betawaffle>
the other annoying thing is i have to prefix these files with an ordering number for networkd
<betawaffle>
like 10-ont.link or whatever
<betawaffle>
because nixos creates 99-main.network
<andi->
yes.. networkd is an ugly hack since they don't really have a graph of dependencies they can work with
<andi->
IIRC it does 3 rounds of configuring everything and then goes into event driven mode.
<betawaffle>
i'm using networkd because i don't want to use scripts
<gchristensen>
#3 rounds should be enough for everybody
<gchristensen>
^ that is a comment in the code, probbly
<betawaffle>
does the ID_PATH udev property show up differently in udevadm?
<betawaffle>
(it doesn't show up with that name, afaict)
<betawaffle>
andi-: should my lan interface also have a vlan configured? i shouldn't add that to the bridge, right?
<andi->
if your lan is supposed to be routed (and NAT'ed) then it should not be part of the very same (traditional) bridge.
<betawaffle>
ok cool, that's what i thought
<betawaffle>
but vlan, yes? because?
<betawaffle>
(there are no vlans on my regular home network)
<betawaffle>
(yet)
<andi->
then it shouldn't be a vlan
<andi->
just a plain old simple interface
<andi->
btw. I just hacked together a simple C tool that might(! very unlikely!) do exactly what you need for EAP proxying https://termbin.com/eklh
<betawaffle>
nice!
<andi->
no .1x here to test it with at the moment but it didn't crash ;)
<andi->
uhh, i forgot the 2nd interface bind path... currently they both bind to the same socket... m(
<betawaffle>
yeah, well i also have to unplug/replug the att router to even get it to attempt auth
<betawaffle>
but that's a great starting point. just need to nix-ify the build, and make a systemd service for it, right?
<andi->
probably
<andi->
you might also have to chnage PF_PACKET to PF_RAW
<andi->
and deal with the L2 addresses
<andi->
so far it would use whatever is on the interface I guess
<andi->
and at that point I might consider not using plain old C for it due to all the things that can go wrong
<betawaffle>
oh, so my other question is... which interface am i supposed to put the att router's mac address?
<betawaffle>
att will only dhcp to that, afaik
<andi->
with PF_RAW I'd probaby just put it in the software and let it fake it towards the OTN. Let linux not know that it is there.
<andi->
It might work with your APUs mac tho
<betawaffle>
so... whichever interface needs the WAN address on it? maybe ont.0?
<andi->
yeah
<andi->
it might be that if the source MAC for the authentication matches the actual next hop MAC that you do not have to fake it..
<betawaffle>
oh, interesting
<betawaffle>
but how would i get the router to auth with that mac?
<betawaffle>
doesn't it have to sign something?
<andi->
no idea how their system works. The device might identify itself to the operator using it's certificate (and maybe MAC) but maybe the actual BRAS only filters for the source of the auth packets
<andi->
because your router just forwards the frames (and rewrites the L2 header).
<betawaffle>
i did see the mac in Identity: for the EAPOL packets
<betawaffle>
not sure if that's relevant
<andi->
that is something you have to play with then
<andi->
Isn't there some kind of law that says you must be able to use your own router? That passed in germany a few years ago.. Blessed all those people that are stuck with legacy ISPs
<betawaffle>
in the US? i can't imagine us being that progressive
<betawaffle>
i'd love it, but i doubt it
<betawaffle>
honestly i don't understand why att needs 802.1x auth at all...
<andi->
ok, i'll try to remember what I was actually working on before this. Feel free to hihglight m:D
<betawaffle>
as obviously it can be bypassed
<betawaffle>
ok, cross fingers. i'm probably gonna lose connectivity for a bit
<gchristensen>
god speed, waffle
<betawaffle>
whelp, that didn't work
<betawaffle>
i couldn't get the interfaces to get renamed, and the vlan bridge thing didn't seem to work either
<betawaffle>
am i really asking for trouble trying to use networkd for this?
<betawaffle>
gchristensen: how many messages of mine have you seen since you said god speed?
<gchristensen>
4
<gchristensen>
16:10 <betawaffle> whelp, that didn't work
<gchristensen>
16:10 <betawaffle> i couldn't get the interfaces to get renamed, and the vlan bridge thing didn't seem to work either
<gchristensen>
16:15 <betawaffle> am i really asking for trouble trying to use networkd for this?
<gchristensen>
16:35 <betawaffle> gchristensen: how many messages of mine have you seen since you said god speed?
<betawaffle>
ok, good. there were just a couple lost
<betawaffle>
first thing i'm trying to figured out is why my .link files aren't matching
<betawaffle>
ok, figured it out
<betawaffle>
sort of...
<gchristensen>
ooh!
<betawaffle>
well, it's either because of networking.usePredictableInterfaceNames, or because my .link files don't get into the initrd
<betawaffle>
at least, that's what the internet saya
<betawaffle>
says*
<betawaffle>
yep, turning off networking.usePredictableInterfaceNames allows the .link files to work correctly, it seems
<andi->
betawaffle: match using pci path or mac address. That should work
<andi->
networkctl status -a should also tell you which files matched the interface
<betawaffle>
only when i disabled networking.usePredictableInterfaceNames first
<betawaffle>
Feb 24 13:02:04 bifrost kernel: IPv6: ADDRCONF(NETDEV_UP): att: link is not ready
<betawaffle>
Feb 24 13:02:07 bifrost kernel: igb 0000:03:00.0 att: igb: att NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
<betawaffle>
Feb 24 13:02:04 bifrost kernel: 8021q: adding VLAN 0 to HW filter on device att
<betawaffle>
Feb 24 13:02:07 bifrost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): att: link becomes ready
<betawaffle>
what does that 8021q thing mean?
<betawaffle>
(att is the interface connected to the att router)
<cransom>
802.1q is a standard for vlan tagging. vlan 0 just means it's the default vlan, and not tagged
<betawaffle>
andi-: so i discovered why the vlan-for-each-physical-interface thing didn't work. not everything sent by the att router is encapsulated (like when doing EAPOL, the stuff from the router has no vlan)
<betawaffle>
but the responses from the ONT *are* in vlan 0
<betawaffle>
it also seems to do ARP stuff and some IPv6 things outside the vlan
<betawaffle>
but all actual traffic to the internet goes in the vlan, definitely
<clever>
gchristensen: my ONT has the auth baked into it, and the only thing i need to deal with is the limit of 1 dhcp lease
<betawaffle>
ahhrg! i had my ports backwards!
<betawaffle>
one bridging the physical interfaces, the other bridging the vlans
<betawaffle>
the EAPOL/ARPs from the router go over the former, the responses and all internet traffic go over the latter
<andi->
betawaffle: so you got it working now?
<betawaffle>
no, just slightly better
<betawaffle>
also... i'm getting this: kernel: IPv4: martian destination 127.0.0.1 from 127.0.0.53, dev lo
<betawaffle>
no idea where that's coming from
<betawaffle>
systemd-resolved[537]: Got packet on unexpected IP range, refusing.
<andi->
that probably means you have a different MAC addresses emitting 127.0.0.0/8 packets that is not you `lo` device..