2021-03-06

<clever> nix will automatically copy things when it needs it, and quoting can force it to copy early, and potentially not enough
<clever> you dont need metrics.nix in the store
<clever> and nix will defer the copy until the last second, or possibly not copy at all
<clever> path + "/foo" creates a new path, identical to having just done something like ./path/foo
<clever> "${path}/foo" will copy path to the store, and omit amything in path/..
<clever> if its un-quoted, it remains as a special path object, and doesnt copy to the nix store
<clever> CFS: self.nixos (../vms + "/images/${name}.nix")
<clever> CFS: theres your problem, it only copied the vms dir, stop quoting paths
<clever> CFS: can you pastebin more of the exprs? how was metrics.nix loaded?
<clever> CFS: you need to find whatever is copying that images dir, and modify it to include more, or just not copy the expr into the nix store
<clever> CFS: your file got copied, only up to the images dir, so .. works, but ../.. goes too far
<clever> CFS: /nix/store/3b0qrgrj7r7ms98s3nhb4wgl943jxwia-vms/images/metrics.nix
<clever> CFS: it is
<clever> CFS: if this file was copied into the nix store the "wrong" way, then ../ wont be the directory your expecting
<clever> CFS: can you pastebin the entire error, with --show-trace?
<clever> CFS: where is this file located?
<clever> CFS: look at the command line grafana was being started with, what are its arguments?
<clever> CFS: also, line 12 is completely un-needed, just do `source = dir + "/${name}";`, skip that whole toFile and readFile mess
<clever> CFS: just do a raw `dir = ../../../dashboards/;`
<clever> CFS: first thing, is that the dir on line 3 is used by readDir and readFile, so it doesnt need to be quoted
<clever> viric: you could use disabledModules to nuke this file from the build, and then nixpkgs wont be included
<clever> interesting idea....
<clever> but nixpkgs can be stripped
<clever> viric: the utils dont give that option i think
<clever> yeah, that adds line 300-400mb
<clever> viric: found some fat?
<clever> rip
<clever> CFS: look at the command line grafana was being started with, what are its arguments?
<clever> ,paste CFS
<clever> CFS: that is being passed to grafana
<clever> CFS: the config that nix generated
<clever> CFS: what config file did /nix/dashboards wind up in? can you pastebin that whole file?
<clever> viric: then build toplevel, and run `du -hc --max=0 $(nix-store -qR ./result) | sort -h | tail` on it, and look for fat you can trim
<clever> viric: then add a second attr to the set, to expose bootSystem.config.system.toplevel
<clever> viric: modify that nix expr, so the symlinkJoin is part of a set, so you access it via `nix-build foo.nix -A netboot`
<clever> viric: you can still shrink that initrd
<clever> CFS: that looks like something did /nix/store/hash-foo/../../dashboards
<clever> viric: yeah, you need the whole initrd to run, but you can shrink it
<clever> viric: but you may want to built config.system.toplevel, and get the closure size down
<clever> viric: and that squashfs will remain compressed, so it shouldnt expand any further
<clever> CFS: how exactly does it break?
<clever> viric: and it needs 500+750 to perform that whole unpacking
<clever> if you omit the initrd in the ipxe script, then it should boot further
<clever> viric: yeah, thats cutting things a bit tight
<clever> viric: but ive never tried that on x86, only arm
<clever> viric: if the machine has a real serial port, you could try early printk
<clever> monokrome: there is also initialHashedPassword, but it only works if the user hasnt been created yet
<clever> monokrome: hashedPassword only works if you have set mutableUsers = false i believe
<clever> so you can still get in, without needing to rebuild the tarball to include the right ssh pubkey
<clever> patagonicus: this shell code also copies /ssh_pubkey into the initrd, and eventually the /root/.ssh/authorized_keys
<clever> the kernel/initrd itself, comes from the netboot utils, which packages the whole rootfs up as a squashfs inside the initrd
<clever> so you dont even need a working kexec installed on the victim
<clever> in this case, its the closure of a shell script, containing kexec, a kernel, and an initrd
<clever> patagonicus: the kexec stuff is a multi-stage trick, make-system-tarball.nix generate a tarball with a complete /nix/store containing the closure of anything
<clever> the original intent, was to infect remote servers, without any console access to fix mistakes
<clever> justdoit automates things enough that you cant really mess up
<clever> once your in, if you wipe the disk, then it obviously cant boot, so if you fail to install it right, its bricked
<clever> this module is also installed, it makes the system reboot within an hour, if you fail to ssh into it and gain control
<clever> the biggest problem, is that if the network doesnt work, you loose control of the system
<clever> one single command will wipe and install
<clever> justdoit is also pre-installed on that live env
<clever> wipe the disk and install as usual, like you used a livecd
<clever> build a magic tarball on one machine, unpack it on the un-suspecting victim, and run /kexec_nixos, and boom, your now running nixos entirely from ram
<clever> it lets you repartition the disk, and do anything
<clever> justdoit and my kexec tool is more powerful then nixos-infect
<clever> if you have control of the vm, you can create an entire virtual disk just for swap, and then you can freely throw it out and replace it with a bigger one
<clever> a swap partition is better performance
<clever> if you didnt have a dedicated partition
<clever> and nixos will auto-create the file and swapon for you
<clever> you can set swapDevices = [ { device = "/path/to/file"; size = 1024; } ];
<clever> swapDevices.*.size
<clever> ?
<clever> yep
<clever> and nixos-generate-config will add the swap to hardware-configuration.nix
<clever> ah, that works too
<clever> [root@amd-nixos:~]# man configuration.nix
<clever> you can add your own too, one sec
<clever> `free -m` ?
<clever> how much ram does this poor thing have??
<clever> slap some swap on that sucker
<clever> so its never actually rebuilt
<clever> your nixos-rebuild was likely dying the same way
<clever> check dmesg
<clever> it should be the same as `nixos-rebuild build`
<clever> what if you `nix-build '<nixpkgs/nixos>' -A system && ls -l result` ?
<clever> what does `which --all weechat` say?
<clever> nix-env can never appear in that list
<clever> nope
<clever> monokrome: if you run `nix repl '<nixpkgs/nixos>'` and then eval `config.environment.systemPackages`, do you see weechat?
<clever> now try rebooting, and see if it stays the same or not
<clever> what does `ls -l /run/current-system` report?
<clever> that is also valid
<clever> then you can see whats behind nix-env
<clever> `which --all weechat`
<clever> mount boot to the right place, nixos-generate-config, and then nixos-rebuild switch
<clever> does weechat appear in $PATH when you nixos-rebuild switch, and DONT reboot?
<clever> if /boot isnt mounted when you run switch, then a reboot will basically be a rollback
<clever> if the packages are already built, it will reuse them
<clever> monokrome: nix-instantiate --find-file nixos-config
<clever> monokrome: if $NIXOS_CONFIG is set, that acts as an override
<clever> 1 { configuration ? import ./lib/from-env.nix "NIXOS_CONFIG" <nixos-config>
<clever> monokrome: [clever@amd-nixos:~]$ vi ~/apps/nixpkgs/nixos/default.nix
<clever> monokrome: that looks like the default NIX_PATH and it has the expected nixos-config=/etc/nixos/configuration.nix
<clever> monokrome: by default, it uses whatever <nixos-config> points to, which relies on $NIX_PATH

2021-03-05

<clever> ixxie: and your pretty much done
<clever> ixxie: substituters = http://nas.localnet:8081
<clever> ixxie: hydra.iohk.io's cache, but if you dont have the public key in your end machine, then that wont really impact things
<clever> ixxie: the only real problems it has right now, is no config (upstream is always iohk + nixos), and no gc policy (i just randomly rm -rf the whole cache dir, whenever i want more space)
<clever> ixxie: and on a 6 month scale, its almost 50/50, so its halving my download requirements
<clever> ixxie: cachecache also includes prometheus exporters, which report that i have had a total of 5.3gig worth of misses, and 4.5gig worth of hits in the last week
<clever> Ke: id say pin, but also test the latest regularly, and keep the pin moving
<clever> ixxie: its currently chewing up 39gig on my nas, and lets me re-fetch things at full gigabit speeds
<clever> [root@nas:~]# df -h /var/lib/cachecache/
<clever> Filesystem Size Used Avail Use% Mounted on
<clever> naspool/cachecache 48G 39G 9.5G 81% /var/lib/cachecache
<clever> ixxie: this program will act as a cache for the cache, and automatically fetch all misses from upstream
<clever> ah, right number, wrong unit, lol
<clever> might be 8tb, cant remember exactly...
<clever> ixxie: ive heard that its 80tb per eval...
<clever> no random "but it worked on my machine" problems
<clever> so when others run nix-shell on it, they get the same version, and it behaves the same way
<clever> its better to specify what version of nixpkgs you want, in package.nix itself
<clever> and then you dont need callPackage
<clever> you can also chuck a `with import <nixpkgs>{};` as the first line of your file
<clever> nix-shell -p 'callPackage ./package.nix {}'
<clever> if you instead want nix to build it for you, and to just run it directly in the shell, you want:
<clever> both, will give you an env suitable for building the package
<clever> or if it needs callPackage, `nix-shell -E 'with import <nixpkgs> {}; callPackage ./package.nix {}`
<clever> Ke: `nix-shell package.nix`
<clever> zfs dedup is block level, and complicated by compression and encryption
<clever> `nix-store --optimize` is a far cheaper file-level dedup
<clever> yeah
<clever> mayhew: zfs would only do it if you turned the dedup flag on before you did all of those --update's
<clever> mayhew: `nix-store --optimize` will hardlink all of the duplicate files together, so they dont consume as much space, and the normal flags in `man nix-collect-garbage` can delete old generations, allowing gc to clear it up fully
<clever> mayhew: based on the URL's in `nix-channel --list`
<clever> mayhew: yes
<clever> winsome[m]: as root, `nix-channel --add https://nixos.org/channels/nixos-unstable nixos`, then `nixos-rebuild switch --upgrade`
<clever> winsome[m]: stangely, that commit is newer then where zsa got added, but is lacking the file, try switching to nixos-unstable to est that out?
<clever> Date: Mon Jan 18 09:16:22 2021 +0000
<clever> [clever@amd-nixos:~/apps/rpi/nixpkgs-test]$ git show 88f00e7e12d
<clever> cat: nixos/modules/hardware/keyboard/zsa.nix: No such file or directory
<clever> [clever@amd-nixos:~/apps/rpi/nixpkgs-test]$ cat nixos/modules/hardware/keyboard/zsa.nix
<clever> [clever@amd-nixos:~/apps/rpi/nixpkgs-test]$ git checkout 88f00e7e12d
<clever> what does `nix eval -f '<nixpkgs>' lib.version` report?
<clever> i just copy/pasted your own msg, and didnt notice the typo
<clever> without the s
<clever> + options.hardware.keyboard.zsa = {
<clever> winsome[m]: git logs say it was added on june of 2020, so i would expect to find it in 20.09...
<clever> winsome[m]: i see hardware.keyboards.zsa.enable in nixpkgs master
<clever> yep
<clever> 20 + "/etc/systemd/system-sleep",
<clever> 18 static const char* const dirs[] = {
<clever> 7 This is needed for NixOS to use such scripts as systemd directory is immutable.
<clever> xwvvvvwx: and bingo, this is one of the patch files!
<clever> 149 ./0015-systemd-sleep-execute-scripts-in-etc-systemd-system-.patch
<clever> xwvvvvwx: i'm guessing that systemd may have been patched by nixpkgs, to look in a different place, the nix expr would be the next place to look
<clever> ExecStart=/nix/store/4712fsjq670jsqjrpkkmpfzhi9g1k4nj-systemd-247.3/lib/systemd/systemd-sleep suspend
<clever> $ cat /etc/systemd/system/systemd-suspend.service
<clever> xwvvvvwx: there is a /etc/systemd/sleep.conf file, and you can create .service files via systemd.services in configuration.nix
<clever> nixos*
<clever> xwvvvvwx: nothing in linux will look in /usr/lib, so that file shouldnt have any effect
<clever> nix-hash can hash a file or path, and then use to-sri to convert
<clever> there is also the old `nix-hash`, but it lacks the sri step
<clever> to-sri convert a hash to SRI representation
<clever> $ nix hash --help
<clever> that binary can then do whatever it wants (in this case, probably fuse) to dynamically add things
<clever> "and even based on what binaries are in $PATH"
<clever> ixxie: and the mount program, will try mount.ntfs-3g, when you use mount -t ntfs-3g
<clever> lrwxrwxrwx 1 root root 14 Dec 31 1969 mount.ntfs-3g -> ../bin/ntfs-3g
<clever> [clever@amd-nixos:~/apps/nixpkgs/nixos/modules]$ ls result/bin/ -l
<clever> [clever@amd-nixos:~/apps/nixpkgs/nixos/modules]$ nix-build '<nixpkgs>' -A ntfs3g
<clever> specifically, it will add ntfs3g to your system packages for you
<clever> system.fsPackages = [ pkgs.ntfs3g ];
<clever> ixxie: if boot.supportedFilesystems contains either ntfs or ntfs-3g, then this module will do something
<clever> tasks/filesystems/ntfs.nix: config = mkIf (any (fs: fs == "ntfs" || fs == "ntfs-3g") config.boot.supportedFilesystems) {
<clever> or grep nixpkgs
<clever> google!
<clever> ixxie: nixos also has a boot.supportedFilesystems option, which will auto-install whatever is needed for a given fs
<clever> ixxie: the more types can be added dynamically, based on what modules your loading, and even based on what binaries are in $PATH
<clever> ixxie: `-t ntfs3g` i think? but my memory is fuzzy on that
<clever> you need to umount, then mount with the right -t
<clever> it just lacks the ability to ever write
<clever> certain ntfs drivers are read-only by design
<clever> ixxie: `mount | grep sda1`
<clever> ixxie: looks like standard chmod rules, you just need to `chmod +w stash`
<clever> what filesystem is it?
<clever> ixxie: use sudo to try and modify a file on that fs
<clever> ixxie: i tend to just cut it down to just what matters, just the header and sda1 line is enough
<clever> ,paste ixxie
<clever> ixxie: is it read-only to root as well? what does lsblk say?
<clever> ixxie: probably, but ive not looked into how it works on nixos
<clever> aforemny: pkgs.system
<clever> lush: `nix repl '<nixpkgs>'` and then you can tab complete the names
<clever> cinimod: they get updated/created when you run `nix-channel --update`
<clever> cinimod: there should be s symlink in that dir, one for each channel
<clever> lrwxrwxrwx 1 root root 82 Dec 31 1969 nixos -> /nix/store/fkkaimqi0pql50pch3w2x4znd8z15jx8-nixos-21.05pre273435.0aeba64fb26/nixos
<clever> [root@amd-nixos:~]# ls -l ~/.nix-defexpr/channels/
<clever> lush: you must put the `let ... in` before the `{`
<clever> lush: you cant have a let inside a set like that
<clever> cinimod: and does nixpkgs exist in that dir?
<clever> cinimod: what about `echo $NIX_PATH` ?
<clever> and make sure to umount before you pop it out
<clever> ixxie: yep, same thing, now you can just mount /dev/mmcblk0p1, like you would mount any other disk
<clever> for me, this appears when i pop in a card
<clever> [5828811.872408] mmc0: new ultra high speed SDR50 SDHC card at address 0007
<clever> [5828811.874285] mmcblk0: mmc0:0007 SD08G 7.42 GiB
<clever> [5828811.752095] mmc0: cannot verify signal voltage switch
<clever> ixxie: watch `dmesg -w` output when you plug a card in, does it say anything?
<clever> ashkitten: either set it to some context-specific no-op value ({} or null or []), or use lib.filterAttrs to remove it afterwards, or use lib.filterAttrs/filterAttrsRecursive to remove every attr with a magic value (such as null)
<clever> yeah
<clever> karantan: all nixos modules get passed the same args, the usual ones are config, pkgs, lib, and i think options, but others can be dynamically added
<clever> karantan: i prefer pinning it within the project like that, so if others clone the project, they get the nixpkgs that i know worked
<clever> which is the nixos channel on root by default
<clever> karantan: its the nixpkgs= from $NIX_PATH
<clever> not sure then
<clever> lovesegfault: try without -d?
<clever> lovesegfault: got an example of one from `ls -l` ?
<clever> angerman: once those graphs are working, you can get concrete numbers for if a given change helps or hurts the eval
<clever> `nix-collect-garbage --max-freed 1` would clean them up, then delete the minimum number
<clever> lovesegfault: nix-collect-garbage cleans them up automatically
<clever> angerman: hydra could record how much those went up, for each attr in release.nix
<clever> angerman: nix itself, also has a wide range of performance metrics its counting internally
<clever> angerman: then you can generate graphs like this, that show how long a given attr took to eval (not counting the shared state with other attrs)
<clever> but it could be hijacked, to add eval-time metrics as well
<clever> angerman: and this is how hydra manages metrics on a job, usually posted in the $out
<clever> angerman: this perl code then takes that json, and inserts it into the postgres db
<clever> then add it to the json in job
<clever> angerman: in theory, you could grab a timestamp on lines 146, and 153, and that would represent the cost of the eval
<clever> angerman: lines 152-216, will detect when a given attr contains a drv, and represent it as json...
<clever> angerman: the while loop starting on 132, will loop over every attribute in a release.nix
<clever> angerman: i do have some ideas on how to make hydra profile the eval for you...
<clever> you may also want to play with declarative jobs, to get it building PR's for you
<clever> :D
<clever> angerman: yep, it sees 3 build machines, and the queue is empty, so its got nothing left to do
<clever> infinisil: nice!
<clever> so if the machine isnt showing on that page, it doesnt even know about the machine
<clever> angerman: https://hydra.iohk.io/machines and this page renders that part of the sql
<clever> it will then insert the results into the sql db
<clever> and re-read the files if they have been modified
<clever> every 60 seconds, hydra-queue-runner will check the last-mod timestamp on everything in $NIX_REMOTE_SYSTEMS
<clever> to confirm if things are making it into systemd
<clever> so you need to double-check the actual .service file, as i showed above
<clever> but, that overrides the default, and removes it!
<clever> ive seen somebody set buildMachinesFiles = []; thinking that the merging would make that a no-op
<clever> but, this has tripped other users up before
<clever> angerman: the default changes, based on what other options are set to
<clever> it doesnt default to [], there is a trick going on
<clever> so you can have a 2nd file, that isnt managed by nix, in that env var
<clever> angerman: but hydra is more flexible then nix itself, and can take a : seperated list of those files, allowing auto-scaling scripts to create&destroy build machines in aws
<clever> angerman: nix.machines is what generates the contents of /etc/nix/machines
<clever> angerman: and this will have logs from its attempts to ssh into each one
<clever> [root@nas:~]# journalctl -f -u hydra-queue-runner
<clever> angerman: this tells hydra what file to find the list of remote machines in
<clever> Environment="NIX_REMOTE_SYSTEMS=/etc/nix/machines"
<clever> [root@nas:~]# cat /etc/systemd/system/hydra-queue-runner.service
<clever> ah, already got a default, good
<clever> andoriyu: check other out-of-tree stuff, to see how KERNEL_DIR gets a default
<clever> andoriyu: probably
<clever> andoriyu: $CC contains the name of the cross-compiler, which is already in $PATH, the nix cross-compile stuff sets all of that up for you
<clever> but going any finer then that gets real messy
<clever> then only the kernel&modules are native, but the rest remains cross
<clever> andoriyu: you can get ugly, and do boot.kernelPackages = (import <nixpkgs> { system = "armv6l-linux"; }).linuxPackages; for example
<clever> andoriyu: nix doesnt allow mixing native and cross compiled things
<clever> andoriyu: then the problem is much deeper, you would have to see how the kernel cross-compile works, and then apply that fix to every module package
<clever> maybe all out-of-tree cross compile is busted?
<clever> andoriyu: double-check if you can cross compile the v4l loopback i linked earlier?
<clever> yeah, i'm not sure whats going on there
<clever> not sure where that problem is
<clever> andoriyu: it must use $CC, not gcc
<clever> sed time!
<clever> and then THAT dir, is what is missing a Makefile!
<clever> but its just assuming KERNEL_DIR is /lib/modules/$(uname_r)/build
<clever> andoriyu: ah, line 37 should be respecting that
<clever> andoriyu: i would expect that to work, though kmod isnt being used yet
<clever> andoriyu: and its being built the same as all of the other out-of-tree modules?
<clever> andoriyu: is there a Makefile in /nix/store/ar3m47gr7qgb6dxc21nvknghn955p908-source ?
<clever> andoriyu: you must fetch it from the current boot.kernelPackages
<clever> 95 #extraModulePackages = [ config.boot.kernelPackages.v4l2loopback ];

2021-03-04

<clever> sshow: buildEnv will merge 2 packages using symlinks
<clever> gpg is special, in that adding a key puts it into the gpg keyring, and it will persist after a reboot, which is why i also prefer it
<clever> drozdziak1: `ssh-add -l` will list keys already in the agent, and $SSH_AUTH_SOCK must point to a socket made by gpg, like /run/user/1000/gnupg/S.gpg-agent.ssh
<clever> drozdziak1: and sudo has a flag to leave $HOME "wrong" (pointing to your user, not root)
<clever> drozdziak1: if nix is ran as root via sudo, it wont talk to nix-daemon, and then you can control its env vars
<clever> drozdziak1: with the right combination of sudo flags, yes