eyJhb changed the topic of #nixos-on-your-router to: NixOS on your Router || https://logs.nix.samueldr.com/nixos-on-your-router
NinjaTrappeur has quit [Quit: WeeChat 2.7]
NinjaTrappeur has joined #nixos-on-your-router
<gchristensen> people who know more about hardware than I do, is it possible using ipmitool to open the console and immediately specify the TTY size, or does it require executing stty and syscalls / ioctls on the target system?
<clever> gchristensen: in theory (based on $TERM i think), the remote end should know that it can use ansii escape codes, to ask the terminal emulator
<clever> gchristensen: but serial ports dont do that by default, and assume the size
<gchristensen> and this would be before a terminal is reached (still at login(1)
<clever> not sure, it might be handled by the first curses program or something
<gchristensen> ah
<gchristensen> I'm going to go ahead and assume it just isn't possible :P
<clever> it would also sort of need to re-query when you re-connect
<clever> the window-change signal can notify it
<clever> but ssh sends that out of band
<clever> maybe look into telnet's window change signal?
<clever> or just run pppd + sshd
<gchristensen> hmm
<clever> i suspect that the old days of dialup, and the login prompt, are legacy support, for when you dial-up'd into a unix box, and just signed into unix normally
<clever> and your shell was pppd
<clever> you could re-create that on nixos, and then just tell pppd to connect to xyz, and do the standard name/pw handshake
<clever> then you can "dial" into the serial port, and get a full network link
<gchristensen> aye, unfortunately the other side is ipmitool
<gchristensen> and I can't touch what is running on the host at all
<clever> does ipmitool give you stdio or a gui?
<gchristensen> stdio I would think?
<clever> then you could just pipe pppd into ipmitool, basically
<clever> maybe also configure it to run pppd on the remote end, after it logs in
<gchristensen> can't :/
<gchristensen> it isn't a big deal though, just trying to skip a couple annoying steps :)
<clever> logfiles.2578:execve("/nix/store/7wrj3m2qjjrpc5wv0vbskg1bl6jqyr6w-nixos-enter/bin/nixos-enter", ["/nix/store/7wrj3m2qjjrpc5wv0vbskg1bl6jqyr6w-nixos-enter/bin/nixos-enter", "--root", "/mnt", "--", "/run/current-system/bin/switch-to-configuration", "boot"], 0x7ffde07b68c8 /* 13 vars */) = 0
<clever> starting with this pid
<clever> clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f8efc2fda10) = 7
<clever> --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7, si_uid=0, si_status=1, si_utime=0, si_stime=0} ---
<clever> so pid7 within the namespace, exited with status 1
<clever> and then immedietly after, it execve'd the chroot
<clever> 72 LOCALE_ARCHIVE="$system/sw/lib/locale/locale-archive" chroot "$mountPoint" "$system/activate" >>$PIPE_TARGET 2>&1 || true
<clever> 74 exec chroot "$mountPoint" "${command[@]}"
<clever> which matches up perfectly to these 2 lines
<clever> so we want pid 7, within the guest
<clever> which wont be logfiles.7
<clever> [root@install-environment:/wtf]# grep getpid -r | egrep '= 7$'
<clever> it will be the one where getpid() claimed it was 7
<clever> of which, there are 148....
<clever> [root@install-environment:/wtf]# grep execve $(grep getpid -r | egrep '= 7$' | cut -d: -f1 | sort | uniq)
<clever> everything executed in each of those PID's
<clever> 404, not found
<clever> execve was never ran on activate
<clever> nor did 72 even run chroot!
<clever> definitely looks like a bash internal problem
<clever> [root@install-environment:/wtf]# grep 'exited with 1' $(grep getpid -r | egrep '= 7$' | cut -d: -f1 | sort | uniq)
<clever> logfiles.2584:+++ exited with 1 +++
<clever> aha, exactly one of the pid 7's exited with 1, the code we saw
<clever> getpid() = 7
<clever> openat(AT_FDCWD, "/dev/stderr", O_WRONLY|O_CREAT|O_APPEND, 0666) = -1 ENXIO (No such device or address)
<clever> write(2, "/nix/store/7wrj3m2qjjrpc5wv0vbskg1bl6jqyr6w-nixos-enter/bin/nixos-enter: line 72: /dev/stderr: No such device or address\n", 121) = 121
<clever> wait a moment...
<clever> on your version:
<clever> 66 PIPE_TARGET="/dev/stderr"
<clever> 72 LOCALE_ARCHIVE="$system/sw/lib/locale/locale-archive" chroot "$mountPoint" "$system/activate" >>$PIPE_TARGET 2>&1 || true
<clever> ah, same on mine, i just didnt notice that bit before
<clever> logfiles.2576:execve("/nix/store/lbgpnkcnlyznkasks5qv0smp5m5y1c0g-util-linux-2.33.2-bin/bin/unshare", ["unshare", "--fork", "--mount", "--uts", "--mount-proc", "--pid", "--", "/nix/store/7wrj3m2qjjrpc5wv0vbskg1bl6jqyr6w-nixos-enter/bin/nixos-enter", "--root", "/mnt", "--", "/run/current-system/bin/switch-to-configuration", "boot"], 0x17c0008 /* 13 vars */) = 0
<clever> so nixos-enter, will exec (not fork) unshare, and tell unshare to re-execute nixos-enter with an env var set, in the container
<clever> `systemctl start --no-block doinstall`!
<clever> looks like i need to pre-erase the partitions with that uber scary cmd
<gchristensen> clever: yep, doinstall assumes it is wiped already
<clever> i think for any further debug, i would need to edit nixpkgs some
<gchristensen> what would youneed to edit?
<clever> basically, i need a `sleep 3600` inside the nixos-enter script
<clever> just before the failing chroot on 72
<gchristensen> ah
<gchristensen> clever: we could ... hack it :)
<clever> looks like i can edit NIX_PATH in tools.sh,
<clever> and point to a copy
<gchristensen> yep :)
<clever> Environment="NIX_PATH=nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs:nixos-config=/etc/nixos/configuration.nix:/nix/var/nix/profiles/per-user/root/channels"
<clever> it shouldnt need a nixos-config= entry, i believe
<clever> lrwxrwxrwx 1 root root 1 Jan 1 1970 /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs -> .
<clever> gchristensen: wut?
<clever> oh wait
<clever> thats the standard alias to map <nixpkgs> to <nixos>
Orbstheorem has joined #nixos-on-your-router
<clever> and cp must have a bug!
<clever> it copied cp's ., not the symlinks .!
<gchristensen> wat
<clever> [root@install-environment:/wtf]# cp -r /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs /nixpkgs
<clever> this copied /wtf not nixpkgs
<gchristensen> ...hmm...
<clever> you may play the inception noise, we have a wat inside a wat :P
<gchristensen> :P
<gchristensen> clever: btw I think if you exit right before nixos-install, and then run nixos-install as root interactively, I think it'll work
<clever> i want to inspect the active nixos-enter pid
<clever> with all of its wonky state intact
<gchristensen> cool
<clever> but i cant find nixos-enter in nixpkgs...
<gchristensen> I'm not going to get in your way, you're way smarter than I am at this
<clever> /nixpkgs/nixos/modules/installer/tools/nixos-enter.sh: PIPE_TARGET="/dev/stderr"
<clever> ah
<Orbstheorem> Hi o/. Quick question: Does this channel talk about OpenWRT-supported routers running NixOS or any system capable of running hostapd configured using nix? ^^
<clever> [root@install-environment:/wtf]# find /nixpkgs -name nixos-enter.sh
<clever> lrwxrwxrwx 1 root root 68 Feb 12 22:54 /nixpkgs -> /nix/store/bnxplwraqqf2isfdz34572z5dxddjkzc-nixos-19.09pre-git/nixos
<clever> thats why, grep isnt symlink following
<gchristensen> Orbstheorem: it is a pretty generic channel :) if you have a system running nixos and it is a router, it is on topic :)
<gchristensen> (or a system you want to be running nixos as a router, etc... few rules)
<Orbstheorem> My laptop is a router xD
<Orbstheorem> But I get the gist of it ^^, cool !
<gchristensen> it is a very low traffic channel, so anything nearby to the niche is probably fine
<clever> and that low traffic led to me hijacking it to debug something a bit spammy :P
<clever> nobody to bother
<gchristensen> hehehe
<clever> gchristensen: i'm now in a screen session, `screen -x`
<gchristensen> attached
* Orbstheorem will just stick around for the niche
<clever> gchristensen: due to it using --store /mnt, it has to re-build the os every time, and cant cache the product
<gchristensen> :| ow
<clever> i dont think it respected my NIX_PATH
<gchristensen> clever: I tried to treat screen like tmux :x :)
<clever> ctrl+a [
<gchristensen> yeah
<clever> one more time!
<gchristensen> alllllll that beautiful, spammy output :P
<clever> gchristensen: while this builds, check the man page for nsenter
<gchristensen> reading
<clever> looks like we want `nsenter --all --target 42`
<clever> gchristensen: its still ignoring my custom nixpkgs!
<gchristensen> :|
<clever> reading nixos-install source
<gchristensen> btw, could even remount /nix/store rw :)
<clever> oh wait
<clever> its just getting nixos-enter from the host PATH
<clever> not the target nixpkgs
<gchristensen> ack
<gchristensen> `cp` to the rescue?
<clever> yep, same as the rest
<clever> back to the spam once more
<clever> abort!
<clever> oops
<clever> i overwrote /tools
<gchristensen> hehe
<gchristensen> want me to get it back, or do you know where to go?
<clever> probably not much lost
<clever> i only need the PATH bit
<clever> line 6 is to blame for the abort
<clever> nixos-install has a nixos-enter baked in
<gchristensen> I don't think I made any/many modifications to tools other than strace and maybe an experiment with exec 2>&1
<gchristensen> okay cool yeah zig looks nice :P
<gchristensen> oops
<gchristensen> betawaffle: ^ =)
<betawaffle> :D
<betawaffle> join the #zig channel
<clever> gchristensen: while we wait for this, have you heard about my adventures in rpi lately?
<gchristensen> your custom kernel?
<clever> custom everything
<clever> including firmware
<gchristensen> good grief
<gchristensen> that is awesome!
<clever> i started with the rpi-open-firmware project, which claimed usb doesnt work but linux can boot
<clever> the .config and rpi.dtb where missing and not documented
<clever> after about a month, i got linux back to the state they claimed it was at :P
<gchristensen> omg
<clever> but now nixpkgs supports vc4 cross-compile natively (already in master), and you can build the firmware with nix
<gchristensen> hah!
<gchristensen> you are amazing
<clever> after more work, i fixed usb (it needs a 25mhz clock on gpio42 and gpio29 toggled to reset the usb hub)
<clever> but, systemd and busybox where segfaulting like mad on the rpi3
<clever> yet it worked fine for tiny hand-rolled initrd's
<clever> i then began to investigate jtag, and can now attach gdb to the arm core's and single-step or backtrace
<clever> but, jtag forces the arm into 64bit mode
<clever> and aarch64 was giving me too much trouble, so i temporarily gave up, and switched to the rpi2, since it lacks 64bit support
<gchristensen> hahaha
<clever> rpi2 locked up solid, the very first time linux tries to printk
<gchristensen> lmao
<clever> which was inside some smp functions
<clever> but with gdb, i could breakpoint linux, and single-step my way thru any part
<clever> i initially blamed smp support, and just compiled linux without that, which got nixos fully booting, but it ran at 1/50th the normal speed
<gchristensen> did you find the problem? :)
<clever> further debug with gdb, revealed that it wasnt the smp function, but that printk uses spinlocks (if smp is enabled)
<clever> and spinlocks use ldrex (load exclusive) in arm
<flokli> /o\
<clever> and SMP support wasnt enabled in the cpu (the firmware has to turn it on)
<flokli> clever: I admire your persistence.
<clever> with SMP disabled in the cpu, the L1/L2 cache are also non-functional, leading to the loss of performance
<gchristensen> O.o
<clever> setting a single bit in a control register, fixed the performance, and spinlocks in linux
<gchristensen> that seems surprising
<clever> and by pure chance, that same bit (which has a totally different purpose) also fixed the segfaulting on the rpi3
<clever> so, you can now boot nixos on both the rpi2 and rpi3, with fully open firmware
<clever> usb, i2c, gpio, and mmc all work
<betawaffle> oh yeah, this stuff. i was trying to read about writing my own firmware a week ago
<gchristensen> clever: that is *incredible*
<betawaffle> (for x86, which is probably even more unrealistic)
<clever> gchristensen: in parallel to all of that, i began porting little-kernel to the vpu
claudiii has joined #nixos-on-your-router
<gchristensen> is this for the camera?
<clever> gchristensen: first, i made a start4.elf, which the rpi4 could netboot into, which has working uart, but no arm control
<clever> gchristensen: then i made a bootcode.bin for the rpi 1-3 (and got it working on all 3 models), including bringing dram online
<gchristensen> Connection closed by invalid user devops port 37532 [preauth]
<clever> gchristensen: but bootcode.bin is limited in size (128kbyte max), and to make development simpler, i then added start.elf support for the rpi3
<clever> its on the `sleep 3600`
<clever> ok, screen 1 now, `ctrl+a 1`
* gchristensen watching
<clever> [root@install-environment:~]# nsenter --all --target 18086
<clever> now i'm inside the nixos-enter container
<gchristensen> yep, I saw the dir change to / and the ps is quite short, makes sense
<clever> and i can reproduce the error
<gchristensen> ....hmmm
<betawaffle> is there an ssh session i'm missing out on here?
<gchristensen> yeah, but it isn't easy to share
<clever> it also fails outside the container
<betawaffle> ok
<betawaffle> it's time for me to hit the TV with family anyway
<clever> gchristensen: you could ssh in again, from under tmate, and share the ro link, but the IP's would leak
<gchristensen> betawaffle: I have a problem wher erunning nixos-install inside a systemd unit breaks because /dev/stderr is busted
<betawaffle> funky
<clever> i can see a number of pids with socket 516805 as stderr/out, but nothing reading it
<clever> comparing to a normal service locally...
<gchristensen> here is a ro shell: ssh ro-vPF5EWGfkV2X8BgxhR5D42cQJ@nyc1.tmate.io
<clever> same thing on my local machine, i cant easily see the remote end of the unix socket
<clever> lets try up the proc tree
<clever> wait
<clever> what happened to the parent of /doinstall-start
<clever> why is it an orphan?
<clever> maybe it isnt
<gchristensen> not sure - doinstall.service is still active
<clever> checking each pid...
<clever> trying on my own end...
<clever> -bash: /proc/32435/fd/2: No such device or address
<clever> [root@nas:~]# echo things > /proc/32435/fd/2
<clever> gchristensen: aha, even on my machine, i cant re-open the unix socket the journal gives to services
<clever> the only way to clone it, is with dup/dup2 i guess
<clever> so PIPE_TARGET cant contain /dev/stderr
<gchristensen> wtf
<clever> maybe &2
<gchristensen> what if we patched it to... yeah, just >&2 ?
<clever> and off it goes again!
<clever> while that boots, let me boot my rpi3...
<gchristensen> can you do the &>2 as a variable? lol
<clever> we will know shortly...
<clever> [nix-shell:~/apps/rpi/lk]$ make PROJECT=rpi3-start && scp build-rpi3-start/lk.elf root@router:/tftproot/grey-firmware/start.elf && uart-manager
<clever> gchristensen: part 1, compiles littlekernel as a start.elf for the rpi3, part 2 copies it to the tftp host, part 3 launches a custom terminal client, that pulls DTR high, allowing the rpi to run
<clever> DTR is wired to the reset pin for the entire board
<clever> if i ctrl+c uart-manager, DTR is pulled low before it exits, forcing the pi back offline
<clever> its basically ipmi :P
<gchristensen> :D:D:D
<gchristensen> installation finished!
<clever> the rpi3 has netboot support, dangit
<clever> packet is too fast :P
<gchristensen> machine is booted!
<clever> and we got booted, so all evidence of the solution is "toast"
<gchristensen> s1-large-storage-test login:
<clever> and we only have irc to record our efforts :P
<gchristensen> lol thank goodness for IRC
<clever> the rpi3 can netboot, and will download bootcode.bin from tftp
<clever> i can run lk from that, but its rather small
<clever> and i dont have usb support in lk yet, to boot the next stage
<clever> but, if i use the closed-source bootcode.bin, it will tftp start.elf for me
<clever> which is why this is called the grey-firmware, i'm using a closed-source file as a bootloader
<clever> but that saves me from having to swap an SD card on every compile
<clever> and i can still develop the usb stuff, in the wrong stage, then port it backwards once 90% of the work is done
<clever> gchristensen: ssh ro-LRHDvrKTKuc3LjVfm726ymTXc@nyc1.tmate.io
<clever> gchristensen: let me know when your in
<gchristensen> (sorry, have to step away for like 15 min, clever!)
<clever> will only take a sec
<gchristensen> okay, looking :)
<clever> its booting...
<clever> and booted!
<gchristensen> :O
<clever> that is a port of little-kernel, running on the VPU
<clever> with a fully interactive shell over the uart
<gchristensen> this is such a cool debugger
<clever> from here, i now need to implement:
<clever> * pll control (to set clocks)
<clever> * sdhost control (to read sd cards)
<clever> * arm control (to actually boot the arm cpu's)
<clever> then lk can do everything rpi-open-firmware can do
<clever> but, having proper timers and threads in lk, makes future development much simpler
<clever> rpi-open-firmware is single-threaded and lacks proper timers
<gchristensen> incredible
<clever> yeah
<gchristensen> are you going to upstream stuff or do like a project on your own or is this just for-fun learing?
<clever> ive been pushing everything to github, and i have hydra setup to build everything
<clever> once i figure out the linux .config issues, i could make an sdimage, like the current pi ones
<clever> and then others can test it more
<clever> both rpi-open-firmware and lk also have OTP support, so it can read the board serial#
<clever> rpi-open-firmware will also set the mac address correctly, since the usb nic lacks an eeprom of its own
<gchristensen> nice
<clever> wifi doesnt work yet, i need to figure out the i2c gpio expander
<clever> hdmi doesnt work yet, the power for the entire graphics pipeline is off
<clever> but i have found the i2c gpio expander, just finding it took a few weeks
<clever> ive also found the EDID eeprom over hdmi
<clever> so i can detect a monitor, i just cant feed it any signal