2021-03-08

<clever> also...
<clever> ixxie: there is also a flag when creating the sdimage, to make the fat32 larger, but that needs a reinstall
<clever> ixxie: mostly, but make sure it finishes copying before you reboot
<clever> ixxie: fdisk -l /dev/mmcblk0, mount the fat32 fs
<clever> ixxie: then its not mounted, mount it, and then nixos-rebuild switch
<clever> ixxie: mount | grep boot
<clever> nix-shell will build things for you, as it evals
<clever> so that path wont be in the nix store
<clever> however, that just evals, it doesnt build
<clever> the old api is still stable
<clever> "/nix/store/nm7d7i9jlqf25nwvan7ghlv3jafnbryj-hello-2.10"
<clever> [root@amd-nixos:~]# nix-instantiate --eval -E 'with import <nixpkgs> {}; "${hello}"'
<clever> qbit: the api for the new nix command is in beta and constantly changing
<clever> ixxie: then the bootloader isnt working right, is the fat32 fs mounted to /boot before you run nixos-rebuild?
<clever> ixxie: it only tries to load the firmware at boot, if you reboot, is the firmware still there?
<clever> ixxie: if /boot isnt mounted right, the reboot undoes everything
<clever> ixxie: rebuild switch, and then check again
<clever> ixxie: the firmware should be in here, check if the file it wants is present
<clever> [root@amd-nixos:~]# ls /run/current-system/firmware
<clever> until you --update
<clever> so it will be using whatever nixpkgs the iso itself was made from
<clever> the config says 20.09, but the iso has a copy pre-fetched, and baked into it
<clever> shla: because its using the copy of the channel that was packaged into the iso
<clever> shla: you may need to nix-channel --update before you install
<clever> ixxie: check dmesg again, any firmware errors like before?
<clever> ixxie: does `ip link` show the wifi now?
<clever> shla: it installs whatever the current channel is, so you can change it with nix-channel before you nixos-install
<clever> maybe, give it a try
<clever> ah
<clever> zalaare: there is an option to do that for you
<clever> networking.interfaces.<name>.ipv4.routes
<clever> List of extra IPv4 static routes that will be assigned to the interface.
<clever> zalaare: that looks correct
<clever> hardware.firmware = [ pkgs.raspberrypiWirelessFirmware ];
<clever> ixxie: pkgs.raspberrypiWirelessFirmware
<clever> ixxie: your missing the firmware
<clever> [ 12.996512] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43430-sdio.bin failed with error -2
<clever> yep
<clever> ixxie: this should also show 2
<clever> root@pi400:~# find /sys -name ios
<clever> ixxie: does `dmesg | grep mmc` show 2 mmc busses?
<clever> ixxie: step 1, check if `ip link` can see the wifi
<clever> if its already booting with the nixpkgs build, you can keep that
<clever> ixxie: i was doing that mostly to skip the compile times
<clever> ixxie: https://gist.github.com/cleverca22/7be36648fd3b44d525d9b3071caceec1 this brings wifi up, in the initrd, using nixpkgs but the rpi builds of the kernel
<clever> ixxie: kinda
<clever> but u-boot breaks overlays
<clever> u-boot instead allows the kernel/initrd to be on the /boot dir of an ext4 fs
<clever> the firmware bootload requires the kernel/initrd on the fat32 fs
<clever> yeah
<clever> ixxie: yeah, fkms works on all models
<clever> ixxie: ive also only been able to get graphics to work using the fkms overlay, and the firmware loader
<clever> ixxie: also cehck the X log in /var/log
<clever> ,libraries veleiro
<clever> i prefer using UUID's
<clever> so nixos-generate-config doesnt know to include the stuff providing /dev/tank/root
<clever> but lsblk doesnt understand that / is /dev/tank/root
<clever> lsblk understands the lvm and luks, and knows that /dev/tank/root (an lvm dev) is on /dev/root (a luks device), which is on /dev/nvme0n1p2
<clever> and i think zfs is the breaking point
<clever> i suspect its using lsblk to figure it out
<clever> this config, plus the normal fileSystems stuff, will get it all working
<clever> luks.devices.root = {
<clever> name = "root"; device = "/dev/nvme0n1p2"; preLVM = true;
<clever> initrd = {
<clever> jboy: certain combinations of luks, lvm, and zfs, will break the auto-detection of what devices are needed
<clever> thats involved in copying paths between 2 machines
<clever> cmdAddToStoreNar
<clever> angerman: though, its only used in a single place!
<clever> src/nix-store/nix-store.cc: SizedSource sizedSource(in, info.narSize);
<clever> angerman: `nix-store --verify --check-contents`
<clever> angerman: run it under gdb and `catch throw` to catch it
<clever> never seen it before though, you would need to get a backtrace
<clever> and it somehow didnt have enough bytes
<clever> angerman: this lets you wrap a source (like the pipes in haskell), to create a new source, that can only consume N bytes before creating a fake eof
<clever> if (this->remain <= 0) {
<clever> throw EndOfFile("sized: unexpected end-of-file");
<clever> size_t read(char * data, size_t len)
<clever> struct SizedSource : Source
<clever> src/libutil/serialise.hh: throw EndOfFile("sized: unexpected end-of-file");
<clever> to copy ${firstdrv}/foo.deb $out
<clever> simplest option, just wrap it in a second drv, runCommand
<clever> CMCDragonkai: yep, there it is, line 242 of vm/default.nix
<clever> mkdir -p -m 0700 $out
<clever> CMCDragonkai: if $out is already a directory for any reason, cp puts a file in the dir
<clever> cp out/make/deb/x64/*.deb $out
<clever> can you pastebin the full expr?
<clever> $out is still whatever you make it to be
<clever> CMCDragonkai: set -x will make it more obvious what is happening
<clever> yeah
<clever> CMCDragonkai: toss a `set -x` in, to see what its doing
<clever> options may exist to change that in bash, but ive not bothered looking into them
<clever> mananamenos: same rules as normal, the history gets commited to ~/.bash_history when the shell exits, and read when the shell starts
<clever> CMCDragonkai: if you run `env | grep 62` in the `buildPhase` does that path come up anywhere?
<clever> mananamenos: thats how env vars work
<clever> mananamenos: you would have to run tmux inside the nix-shell
<clever> CMCDragonkai: try raising memSize more
<clever> mananamenos: not sure then, ive not played with zsh much
<clever> CMCDragonkai: `no space left on device` is a disk error, but it could be a tmpfs, try increasing memSize also
<clever> probably
<clever> yeah
<clever> CMCDragonkai: just add it to the derivation you passed to the vm tools
<clever> mananamenos: nix-shell requires bash, because all of the functions within nixpkgs assume you have bash
<clever> so add size=8192;
<clever> its just another attr to the derivation
<clever> CMCDragonkai: or disk size, line 404
<clever> CMCDragonkai: there are also dedicated tools just for dealing with rpm's
<clever> CMCDragonkai: it will use /dev/kvm if it can find it, and fall back to software emulation if it cant
<clever> CMCDragonkai: yeah
<clever> CMCDragonkai: then it gets ran as root in a vm, and has free access to a fake /, but only the products in $out will persist
<clever> CMCDragonkai: run vmTools.runInLinuxVM on the derivation
<clever> angerman: that happens if something in /nix/store was modified after building, you must either `nix-store --delete` it, or `nix-store --repair-path` it
<clever> yeah, anywhere before $HOME gets used
<clever> CMCDragonkai: yeah
<clever> CMCDragonkai: you can just shove this somewhere in the derivation
<clever> mkdir home
<clever> export HOME=$(realpath home)
<clever> yeah, and you can just run nix-instantiate with the same args (if its not -p based)
<clever> nix-copy-closure --include-outputs on a drv file
<clever> if you want the build-time deps of that drv, you need to map over every attr of the drv, and write them with pkgs.writeText
<clever> angerman: other way around, nix-shell always gives you an env that represents a derivation
<clever> rmcgibbo[m]: gccStdenv i believe

2021-03-07

<clever> boom, its systemd
<clever> za1b1tsu: but there is also grep.app: https://grep.app/search?q=Error%20switching%20console%20mode
<clever> za1b1tsu: that will tell you exactly which binary is to blame for the error msg
<clever> za1b1tsu: one fancy trick thats easier on nixos, `grep 'Error switching console mode' $(nix-store -qR /run/curent-system)`
<clever> on both ends
<clever> dmesg would tell you
<clever> its probably a hash of the contents, not a commit sha1
<clever> ive not tried that yet
<clever> but its missing on a diff box, i think it got moved
<clever> Miyu-saki: that filename, is a hash of the url, and where nix used to store that data
<clever> "96e789b05d2bd60881c221dfba38f0fe"
<clever> [root@system76:~]# cat .cache/nix/tarballs/1x6i5cafxg5kx9rc18w2cmq5y8jcn9b49cfq9bpnpjysdhl7j2hw.info
<clever> which gives me an idea...
<clever> Miyu-saki: if its been over an hour since the last fetch, nix will try to re-fetch it, but i think github etag based caching will let it reuse the old tar, if the rev hasnt changed
<clever> yeah
<clever> tarball-ttl = 3600
<clever> [root@system76:~]# nix show-config | grep ttl
<clever> Miyu-saki: that just refers to the tip of the nixos-20.09 branch
<clever> bisect time!
<clever> Miyu-saki: this is something ive used before
<clever> default.nix: # nix build -L -f ~/apps/rpi/rpi-tools/ pkgsCross.armv7l-hf-multiplatform.pkgsStatic.msd.rootDir
<clever> Miyu-saki: you can mix pkgsStatic and pkgsCross together
<clever> Reventlov: hydra says it took 2h 8m
<clever> Reventlov: this is how i changed the temp dir, when on non-nixos, similar can be acheived via systemd.services.nix-daemon
<clever> Environment="TMPDIR=/mnt/tmp"
<clever> [Service]
<clever> pi@pi400:~ $ cat /etc/systemd/system/nix-daemon.service
<clever> ixxie: set it to false, and they go away
<clever> ixxie: aha! config.boot.initrd.includeDefaultModules
<clever> ixxie: yeah, thats the quickest solution
<clever> ixxie: run a `git grep ahci` on nixpkgs/nixos, and see where its being added
<clever> ixxie: all of the search utils hide packages with duplicate .name values
<clever> ixxie: you have to search using tab-completion in `nix repl '<nixpkgs>'`
<clever> ixxie: nope, that tells modprobe to not load it, but the fault is with nixos trying to copy the module into the initrd
<clever> or one of its related lists
<clever> ixxie: you need to remove it from the boot.loader.initrd.availableKernelModules somehow
<clever> ixxie: thats a driver for sata controllers, its not needed, but a recent bug in nixpkgs makes missing modules fatal
<clever> you would need to either paste an edited copy of this somewhere, or overrideAttrs the name
<clever> ixxie: that sounds like the image was for a larger device
<clever> ixxie: SD cards can fail
<clever> ixxie: try linuxPackages_rpi3 ?
<clever> the images need a bit of work, ive been wanting to go over all of them and make sure its all good
<clever> ixxie: by default, the nixos images want to use u-boot, and configuration.nix must be configured correctly, or it wont update things properly
<clever> ixxie: i just use plain zfs + nfs on my nixos nas
<clever> ixxie: *digs*
<clever> ar: or making use of the protective mbr
<clever> viric: with care (or the right tooling), you can then align the gpt and iso partitions up on the same blocks, so one fs is seen by both
<clever> viric: the iso filesystem starts something like 32kb into the image, plenty of room to shove in either an mbr or gpt header
<clever> patagonicus: `nix show-derivation` will translate it to json

2021-03-06

<clever> ah
<clever> if its in the cache, that will re-fetch it
<clever> ronthecookie: run ldd on the binary, then run `nix-store -r` on the relevant missing paths
<clever> back when i still ran gentoo, and had to compile firefox
<clever> viric: ah, rust wasnt even started (i think), when i first had such linker problems
<clever> hpfr: yeah
<clever> hpfr: /proc/PID/maps, shows all libraries mapped by a given pid, mesa should be in that mix
<clever> probably a lack of people using console mode
<clever> fbcon also complicates matters
<clever> viric: the scrollback buffer is lost if you change to a different tty
<clever> midori?
<clever> rather then doing multi-lib or cross, nixos just makes a 100% pure 32->32 env
<clever> nixos doesnt give you that option right now
<clever> yeah, cross-compile from 64->32 is one simpler option
<clever> viric: i had the same issue many years ago, just running the linker alone required over 3gig of ram
<clever> viric: ive not seen any that mount over http(s)
<clever> isaacr24: assuming default.nix is using the overrides on line 31 right, that should work
<clever> viric: sometimes, its simpler to just rip the hdd out, mount it elsewhere, nixos-install, and then boot that back in the target
<clever> viric: ack, irc cut it short
<clever> viric: this gets it down to 502mb before the squashfs
<clever> [clever@amd-nixos:~]$ nix-build '<nixpkgs/nixos>' -A config.system.build.toplevel --arg configuration '{ lib, ... }: { imports = [ <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix> ]; hardware.enableAllFirmware=false; hardware.enableRedistributableFirmware = lib.mkForce false; system.extraDependencies = lib.mkForce []; disabledModules = [ "installer/cd-dvd/channel.nix" "profiles/base.nix" ]; nixpkgs.overlays = [ (self: super: { grub=self.utilli
<clever> isaacr24: can you pastebin that whole file?
<clever> isaacr24: if you are applying an overlay correctly, it should change the time-compat used by everything
<clever> viric: 655mb -> 531mb!
<clever> viric: profiles/base.nix looks like more fat to trim...
<clever> viric: it just does an overrideCabal, with doCheck = false;
<clever> [clever@amd-nixos:~]$ nix-build '<nixpkgs/nixos>' -A config.system.build.toplevel --arg configuration '{ lib, ... }: { imports = [ <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix> ]; hardware.enableAllFirmware=false; hardware.enableRedistributableFirmware = lib.mkForce false; system.extraDependencies = lib.mkForce []; disabledModules = [ "installer/cd-dvd/channel.nix" ]; }'
<clever> viric: down to 670mb now
<clever> system.extraDependencies shaved another 100mb off
<clever> removing the firmware dropped mine from 1.4gig to 900mb
<clever> [clever@amd-nixos:~]$ nix-build '<nixpkgs/nixos>' -A config.system.build.toplevel --arg configuration '{ imports = [ <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix> ]; }'
<clever> viric: swap
<clever> viric: why does this system have so little ram?
<clever> viric: grub shouldnt be needed, try removing it entirely?
<clever> yeah, but a lot of the tools need special options, due to the kernel being more limited in its unpacking
<clever> viric: i ran into similar problems with the cpio used to make an initrd, it needs `-H newc`
<clever> delete!, delete!
<clever> modules/installer/netboot/netboot.nix: else [ pkgs.grub2 pkgs.syslinux ]);
<clever> may need an override, setting grub2 = super.grub2.override { zfsSupport = false; };
<clever> where is grub becoming a dep?
<clever> viric: try the config first? and then override grub2 if that isnt enough
<clever> boot.loader.grub.zfsSupport
<clever> [clever@amd-nixos:~/apps/nixpkgs]$ man configuration.nix
<clever> 9 , zfsSupport ? false
<clever> [root@amd-nixos:~]# nix edit -f '<nixpkgs>' grub2
<clever> so you can skip having to run one
<clever> qemu can also emulate a tftp server
<clever> -net user,tftp=${config.system.build.ftpdir}/
<clever> the .lkrn file, is ipxe packaged to look like a linux kernel
<clever> viric: there it is, -kernel ${config.system.build.ipxe}/ipxe.lkrn
<clever> oh, where did i put it...
<clever> and qemu can run it
<clever> ipxe has a dozen build targets, including what looks like a linux kernel
<clever> yeah
<clever> you can test on a non-root dir first, to figure out what args are needed
<clever> viric: in theory, just do fileSystems."/" = something; and it will just work
<clever> and all 1000 of them started fighting over ram at once
<clever> when i got home, turned the laptop back on, every single df woke up
<clever> and they slowly slipped into swap
<clever> ehmry: every 5 mins, cron would run `df -h`, and it would deadlock, and 1000's of them built up
<clever> ehmry: in the past, ive has nfs deadlock because i shutdown the server (a laptop) and took it with me on a trip
<clever> viric: the qemu test framework is using 9p's virtio backend, to mount a host fs in the guest, and skip any block devs
<clever> iscsi can support multiple writers, but you need an fs that allows it, and a server to coordinate writes
<clever> and iscsi only lets 1 client write at a time, preferably 1 client with entirely exclusive access
<clever> nfs has caused me troubles randomly
<clever> ive also been wanting a better file level network fs
<clever> grub thinks its an internal hdd!
<clever> witout any netboot support
<clever> plain grub can then boot from that block device
<clever> viric: so anything doing legacy bios calls to read the internal disk, will wind up reading iscsi instead
<clever> viric: if you do this in ipxe, it will hijack the legacy bios routines for reading a hdd, and map an iscsi device to it
<clever> sanboot iscsi:192.168.2.11:::1:iqn.2016-01.laptop-root
<clever> viric: oh yeah, and you dont even need to serve the kernel/initrd over ipxe, when using iscsi
<clever> but i'm not sure how much i woudl trust it, it would lead to inode collisions
<clever> nfs does have an option to just truncate the inode numbers to 32bits
<clever> and you just never notice, if your not using something like xfs on a huge disk
<clever> and a lot of packages just didnt opt-in
<clever> it has to be enabled by the configure script (by default)
<clever> if it returns non-zero, it must be the end of the dir!
<clever> viric: because nobody expected readdir() to return an actual error! lol
<clever> viric: so it randomly tells you, file not found, when the file clearly is found
<clever> viric: half of the bootstrap code in nixpkgs is built without that.....
<clever> viric: a 32bit userland then goes EOVERFLOW when you try to ls a dir with 32bit tools, that had been built without large-file support
<clever> viric: nfs will happily carry those inode numbers over the network
<clever> viric: if you are using xfs, with a disk over 1tb in size, then you wind up with inode numbers over 32bits long
<clever> viric: ive discovered that nfs can be problematic when the client is 32bit, and the server has certain fs combinations
<clever> viric: so i can expose a zvol to my windows box, and windows just says "sure" and treats it like another block dev, toss ntfs on, and your apps are all happy
<clever> viric: ive found nbd to be rather unreliable, iscsi surprisingly even has windows support
<clever> viric: and the server end, which is still buggy, it completely breaks the link if nixos-rebuild tries to restart tgtd
<clever> viric: but to create the rootfs itself, you would need to either build a disk image and share that, or connect it to a temporary box, and nixos-install to it
<clever> viric: you would build a config using fileSystems."/" = { device = "UUID=whatever"; iscsi = { enable = true; host="1.2.3.4"; lun = "something"; }; };
<clever> viric: tgtd and iscsi
<clever> yeah, mkforce should do it
<clever> i think its a bit of both
<clever> to avoid compatability problems from using the wrong versions
<clever> but its also mixing in some libs from debian
<clever> the FHS env used by things like steam, just symlinks them to look like a normal /usr/bin, but it still has the normal nix RPATH
<clever> viric: you could just build it the nix way, and then use patchelf to un-nix it
<clever> viric: the gcc that comes with nix has been heavily patched, to actively reject /usr/lib and to add rpath's into the store
<clever> yep
<clever> viric: on my box, gzip, bzip2, lzma, xz, lzo, lz4
<clever> [root@amd-nixos:~]# cat /proc/config.gz | gunzip | grep CONFIG_RD
<clever> [root@amd-nixos:~]# modprobe configs
<clever> viric: havent seen that before, how big is the initrd? put the gz back?
<clever> viric: 20.09 is just using a stray var in a let block, so you cant force it off, but you can still use disabledModules to just omit zfs support from nixos entirely
<clever> viric: this module will try to detect if zfs is in the supportedFilesystems lists, you could maybe mkForce it to false, or disabledModules it
<clever> viric: you can mkForce those back to false, and the firmware will go away
<clever> nix why-depends ./result /path/to/python
<clever> `nix why-depends`
<clever> kill it with fire!
<clever> but if you just add yourself to the kvm group, it should work
<clever> that + means there is an acl in place also
<clever> CFS: but if you look at the default permissions, it may be in a group you could join
<clever> CFS: on nixos, its got write to everybody
<clever> crw-rw-rw- 1 root kvm 10, 232 Mar 4 21:52 /dev/kvm
<clever> CFS: you need write permissions to /dev/kvm