2017-11-15

<clever> while its blocked
<clever> joko: can you gist the output of "ps -eH x" ?
<clever> joko: if it ssh's back into itself, yeah
<clever> joko: the flock command in there would limit you to a single job at a time, and the nice is more for when the hydra master is also a slave
<clever> joko: hydra accepts a list of paths to files matching the /etc/nix/machines format, so you can add a secondary one just for hydra, which nix will ignore
<clever> joko: also, hydra can use an alternative file for its slaves
<clever> and the next rebuild will restore that symlink
<clever> joko: if you delete /etc/nix/machines it will revert back to local builds
<clever> joko: so the normal users can still use the build slaves
<clever> joko: nix-daemon needs read access to that keyfile
<clever> goibhniu: good idea
<clever> joko: but grub.enable isnt set, so the grub code never even runs
<clever> garbas: the only thing fishy that i can see is `zpool set bootfs=tank/root/nixos tank`, other then that, all i can think of, is to try EFI grub, rather then efi systemd
<clever> i cant see anything obvious in that first gist that would explain the boot problems
<clever> garbas: line 24 can also be deleted, nixos will auto-detect that based on the config in hardware-configuration.nix
<clever> joko: also, grub isnt enabled, so that line can just be deleted
<clever> joko: and its set via import <nixpkgs> { system = "x86_64-darwin"; }
<clever> joko: stdenv.system would be the target your building for
<clever> joko: what is in your /etc/nix/machines ?
<clever> pmeunier: .override changes the arguments given to the whole file, on line 1, so the x coming into it is changed
<clever> pmeunier: in something like that, it might be better to use .override, which operates at a much sooner level
<clever> rec modifies the values before they reach the first function
<clever> nothing can override things before rec
<clever> ah
<clever> what are you trying to do with those custom overrides?
<clever> the override function had to setup a new .overrideAttrs, against the new version
<clever> i believe
<clever> (pkgs.hello.overrideAttrs (old: { foo = "foo"; })).overrideAttrs (old: { bar = "bar"; })
<clever> you can chain overrides
<clever> pmeunier: overrideAttrs is a special thing between the user-facing stdnev.mkDerivation, and the backend that actualy handles making the derivation
<clever> pmeunier: you have to pass the old hello to a function in this case
<clever> pmeunier: foo = pkgs.hello // { bar = args: otherFunc args pkgs.hello; };
<clever> pmeunier: when creating the bar value, pkgs.hello is in-scope
<clever> then foo.bar is a function
<clever> pmeunier: for example, foo = pkgs.hello // { bar = args: value; };
<clever> pmeunier: you can use // to just merge a set onto something, or .passthru
<clever> pmeunier: no simple way to just override .version
<clever> pmeunier: rec happens before the overrides, so you have to override everything
<clever> adisbladis: nix also has an auto optimize option
<clever> sphalerite: you could also compare lz4 and gzip-9
<clever> avn: hardlinking works for 100% identical files, dedup works better for partially matching, and will depend heavily on the data
<clever> so try once more with only compression
<clever> that might be the size of the dedup table
<clever> simplest thing is to do it again with no compression
<clever> yeah, i'm not sure how to seperate them
<clever> sphalerite: some savings there, about 18gig saved
<clever> sphalerite: can you gist zfs list -t filesystem -o name,used,written,logicalused,compressratio,dedup,compression ?
<clever> 120mb on-disk for the DDT
<clever> ah
<clever> so you must re-write every block for the change to take effect
<clever> same for compression
<clever> dedup only applies to blocks written while it is enabled
<clever> sphalerite: that might be the size of the DDT on-disk, what does "zpool status -D" say?
<clever> sphalerite: what about used vs logicalused?
<clever> not sure, ive been wondering the same about my raspberry pi
<clever> sphalerite: try logicalused
<clever> i think "on" is the recomended one, so upstream can change it to the "best" algo when a better one comes out
<clever> gzip-9 gets better compression, but has a noticable effect on cpu usage
<clever> sphalerite: i believe on is an alias for lz4
<clever> steveeJ: let me grab the docs
<clever> steveeJ: yeah, i just wget the image from hydra and "docker load" it
<clever> dhess: ah yeah, overrides on sub-attributes are messy
<clever> only a single if is needed to deal with not-found
<clever> dhess: now instead of chaining if's, your just looking up the system in a set
<clever> dhess: srcs = { "i686-linux" = ...; "x86_64-linux" = ...; .... }; and then srcs.${stdenv.system}
<clever> dhess: one way you can improve the giant chain of if's on line 12-33, is to use a set
<clever> airpwn is a tool to mitm http requests, even if WEP is at play (and KRACK could also add WPA support)
<clever> yeah, if KRACK is in play, you could mitm the session, and inject a fake reply before the real server sends one to the router
<clever> so the AP must decrypt the packet, re-encrypt with the other session, then rebroadcast
<clever> and in the case of WPA, only the AP knows how to decrypt things from each peer
<clever> sphalerite: i believe everything must go thru the AP when in AP mode
<clever> steveeJ: i believe nix-daemon does something similar, for its sandboxes
<clever> s/count/could/
<clever> it could either keep the original layer.tar files, and just unpack them in sequence for each container, or it count union pre-unpacked ones together
<clever> i think it also depends on what storage driver you use in docker
<clever> and the kernel may not be happy trying to union 200 filesystems together
<clever> but that would leave other docker end-users downloading 200 layers
<clever> which could optimize things better
<clever> there is a different docker/nix project i forget the name of, that makes one layer for every storepath
<clever> yeah
<clever> so zfs dedup is needed
<clever> steveeJ: and if you import 2 such images, they wind up being 98% identical, often with just 5 bytes differing in a bash script
<clever> steveeJ: when nix builds an image, it creates an entire layer, that has the closure of a given store path
<clever> so if i was to spawn 20 containers from the same image, it would be nearly instant
<clever> steveeJ: and then it uses zfs commands to atomicly spawn an entire FS from that snapshot
<clever> steveeJ: i set docker to the zfs driver, so docker makes a snapshot of every image
<clever> steveeJ: the name of the pool
<clever> sphalerite: i also enabled dedup for amd/docker, because i was loading the same image (built by nix) over&over, and they where 98% identical
<clever> amd/docker 597M 12.4M 2.25G 4.18x
<clever> so my store is 131gig of data, but its only using 67gig on-disk
<clever> amd/nix 67.2G 67.2G 131G 2.16x
<clever> this includes both the effects of compression and dedup, i believe
<clever> amd 189G 96K 288G 1.66x
<clever> NAME USED WRITTEN LUSED RATIO
<clever> [root@amd-nixos:~]# zfs list -t filesystem -o name,used,written,logicalused,compressratio
<clever> sphalerite: and one other thing
<clever> and nearly 1gig of disk
<clever> so its using 340mb of ram
<clever> there are 630k rows in the dedup table, each record takes 1.7kb on disk, and 567 bytes in ram
<clever> dedup: DDT entries 630380, size 1.71K on disk, 567B in core
<clever> [root@amd-nixos:~]# zpool status -D
<clever> dedup is elsewhere
<clever> the help output describes most of the fields
<clever> sphalerite: one includes L2, the other doesnt
<clever> arcstat.py -f time,read,dmis,dhit,dh%,mrug,mru,mfu,mfug,mread,c,ph%,pm%,mm%,miss,hits,mhit,mh%,mmis,hit%,eskip,arcsz 10
<clever> arcstat.py -f time,l2hit%,l2read,l2bytes,l2size,l2asize,l2miss,l2hits,read,dmis,mru,mread,c,ph%,pm%,mfu,mm%,miss,mrug,dhit,mfug,hits,dm%,mhit,dh%,mh%,mmis,hit%,eskip,arcsz 10
<clever> and then look at the numbers
<clever> sphalerite: in theory, you can make a new dataset and just mount it to /nix2/, turn on dedup, then just copy all of /nix over
<clever> so reads dont care about the size of the DDT
<clever> sphalerite: i think the DDT is only required for writes and deletes
<clever> sphalerite: yeah, that would need some testing
<clever> sphalerite: that would allow partial duplicates between 2 files to be shared
<clever> sphalerite: markov chain or troll! lol
<clever> dhess: that should work
<clever> yeah
<clever> pie_: and also qt5.callPackage
<clever> pie_: you want stdenv.mkDerivation and { stdenv, ...
<clever> pie_: mkDerivation is an attribute of stdenv
<clever> pie_: oh
<clever> pie_: can you also pastebin the default.nix?
<clever> pie_: can you try it with --show-trace, and then pastebin the output?
<clever> pie_: callPackage should have handled that
<clever> yep
<clever> joko: what about nix-instantiate release.nix -A shipUnit --show-trace ?
<clever> joko: try again after adding --show-trace to nix-repl
<clever> joko: can you paste the output of "nix-repl release.nix" ?
<clever> pie_: nix-build -E 'with import <nixpkgs> {}; callPackage ./default.nix {}'
<clever> joko: can you link your release.nix file?
<clever> joko: load release.nix in nix-repl

2017-11-14

<clever> and now nix cant rebuild when things change
<clever> Lisanna: it stops people from doing impure things like src = "/home/clever/foo"; and they just chmod when it fails to read
<clever> Lisanna: yeah, that sounds good
<clever> and setup user, network, pid, and mount namespaces
<clever> it has to bind-mount every single storepath in the closure of the build, within a chroot like env
<clever> infinisil: i believe its off, but i always turn it on
<clever> so any time i missed the binary cache, the test passed
<clever> yeah, i had sandboxing off at the time
<clever> a nixos sandbox lacks every one of those, so it hard-codes itself to open "unknown" at runtime
<clever> and then hard-code the result into the binary
<clever> turns out, net-snmp will check for /etc/mtab, /proc/mounts, and 3 or 4 other paths, to figure out how your OS deals with keeping track of mounts
<clever> it took a while to track that problem down
<clever> git bisect cant deal with that pattern
<clever> every build i made locally worked
<clever> every build coming from the binary cache was broken
<clever> gchristensen: i once had the fun of trying to bisect a bug, caused by nix sandboxing
<clever> it uses nix copy and runInLinuxVM to mount the image
<clever> mbrock: modify sourceRoot in the postUnpack
<clever> yeah
<clever> mbrock: and stack2nix just generates a whole haskellPackages set, with exactly the closure of the project, and nothing more, on the exact versions stack said to use
<clever> mbrock: the project i linked has at least 10 sub-projects (each with its own cabal file) in that repo, plus many deps that have been forked and are waiting on upstream things like what i linked
<clever> mbrock: it can correctly translate things like this into nix: https://github.com/input-output-hk/cardano-sl/blob/master/stack.yaml#L47-L51
<clever> mbrock: if your using a stack file, there is now stack2nix
<clever> pie_: that URL isnt hydra
<clever> jtojnar: ah
<clever> jtojnar: i know xfce can do it, ive had to do that on a recent netbook
<clever> so no tainting the rest of the machine
<clever> and only that checkout of X will use the new Y
<clever> so you can just put that into the default.nix of X, to modify what Y src it uses
<clever> and it will ignore the users config.nix file
<clever> so you can just do import <nixpkgs> { config = { packageOverrides = pkgs: { ... }; }; }
<clever> mbrock: that argument lets you pass in the value of config.nix
<clever> mbrock: have you seen what import <nixpkgs> { config = { ... }; } can do?
<clever> ahh
<clever> mbrock: then just skip the unpackPhase, and use what git gave you
<clever> mbrock: you could also just git clone foo, then nix-shell '<nixpkgs>' -A foo
<clever> binaryphile: in this case, it can take minutes just to query cache.nixos.org for the full closure of your build, so it helps speed it up a lot
<clever> there is also a TTL option somewhere in nix.conf to make it re-check sooner
<clever> binaryphile: thats the binary cache cache, you can clear it by just deleting /nix/var/nix/binary-cache-v3.sqlite*
<clever> it only goes into pending when hydra actually starts the build
<clever> one thing lost due to that change, is that the github status plugin wont say "pending" when the job is queued
<clever> a proper eval-finished hook (and maybe a for loop over the jobs) would solve that issue and restore that function
<clever> mbrock: and suddenly, it takes 9 hours to run the hooks for every job, lol
<clever> mbrock: but that started to run into problems, because the hooks are handled by running the hydra-notify program (a perl script), on each build, one by one
<clever> mbrock: previously, buildQueued was ran for every single job being queued after the eval has finished
<clever> mbrock: fetchInput is a rather special one, hydra will keep calling it in each plugin, until it returns a non-null value
<clever> and docs for them
<clever> mbrock: i think this is the list of available hooks for plugins: https://github.com/NixOS/hydra/blob/582c39942097539aa595d6945a82c4e25ca16ff1/src/lib/Hydra/Plugin.pm
<clever> ikwildrpepper: ah, ive kept my distance from perl
<clever> not sure if it has an eval hook
<clever> youll need to make an override on hydra to copy that file into the right part of $out, and then it should work
<clever> when certain events happen
<clever> mbrock: hydra will automatically call a given function in every plugin it can find
<clever> joko, mbrock: this is how i was managing nix-push, before i discovered hydra itself was a binary cache: https://gist.github.com/cleverca22/9d6a1e1c8a849c13e78d499c0dfc5eb8
<clever> correction, its symlinked to /var/lib/hydra/ : https://github.com/NixOS/hydra/blob/master/hydra-module.nix#L260
<clever> ikwildrpepper: one thing i noticed, is that hydra will copy it to /var/lib/hydra/ on startup, and i hadnt seen anything like file(...) in the docs
<clever> ikwildrpepper: yeah
<clever> ikwildrpepper: the only way to deal with github tokens in hydra is to put them into the store, hydra doesnt support anything better
<clever> joko: another more extreme thing, that i think is a bit of a flaw in hydra and nix-serve, if i gave you the result of `readlink /run/current-system`, you could download a complete copy of my hydra server (the nixos that powers it), along with any secrets that happen to be in /nix/store/ (which includes github auth tokens)
<clever> the storepath of anything built from that source also counts
<clever> joko: nginx can be configured to allow the binary cache URL's thru, but then anybody that knows the hash of your source can also get the full source
<clever> so its better to just use some network levels firewalls to restrict who can connect
<clever> basicauth will mess with binary cache usage though
<clever> joko: i either a: dont port forward it, or b: https://nixos.org/nixos/options.html#%3E.basicauth
<clever> and also keep in mind, that hydra will share the full source to everybody
<clever> yeah, like that
<clever> joko: somewhat, "sudo -u hydra -i" then "ssh-keygen" and no passphrase, give it access, and then configure hydra with an ssh url in the build input
<clever> larsvm: diff the output of "env" in both terminals
<clever> Lisanna: what about setting it to 0?
<clever> srhb: some devices like the raspberry pi also have a hardware watchdog timer, where you must reset the timer constantly, and if the OS locks up for any reason, the timer runs out, and the hardware forces a reboot
<clever> srhb: the watchdog i had seen last, would check various conditions to see if they are true (is the network up?, is a given service responding correctly?) and reboot the machine if they fail for X seconds
<clever> srhb: test wont touch /boot, so any reboot would rollback
<clever> srhb: i would just use "nixos-rebuild test" and then maybe configure a watchdog service
<clever> and you could just set require-sigs directly
<clever> ah, the default for the require-sigs option is based on if the pubkey list is an empty string
<clever> ah, does that one work?
<clever> i think that turns off signature checking, because it has no known keys
<clever> yeah
<clever> Lisanna: maybe it was --option binary-cache-public-keys ""
<clever> let me see
<clever> ah yeah, there is a problem with nix copy, it expects things to be signed, even when using root
<clever> pipefail
<clever> ah
<clever> run "reset" to restore the terminal defaults
<clever> yeah
<clever> because the hashes are just missing
<clever> but we are using it in the reverse direction
<clever> and repairs the damage on-disk
<clever> normally, it checks the hashes in db.sqlite against the ones on-disk
<clever> and there is no way for it to know what even needs repair, because the hashes are missing
<clever> it doesnt have network so --repair cant work
<clever> that will hash everything and fix those 00000's
<clever> maybe "nix-store --verify --check-contents"
<clever> Lisanna: i think it was "nix-store --verify", you have to run that after load-db, and before copy
<clever> one min
<clever> the fake db lacks hashes
<clever> ah yeah
<clever> :D
<clever> yeah
<clever> Lisanna: check the results of "mount" in the guest, and ls in prevm, you should see an xchg mount point and dir
<clever> Lisanna: youll need to use one of the xchg things to get the file into the vm
<clever> it should at least exist in prevm
<clever> you should be able to throw an ls into the build script, to look around
<clever> i think it does
<clever> make sure the closure-* file exists
<clever> yeah, i think thats the right one
<clever> Lisanna: something in that area, cant remember which one
<clever> so you need that perl script to create a fake backup, that has the closure of a given path
<clever> and the nix sandboxing wont let you access the host db.sqlite, because its impure
<clever> correct
<clever> so nix considers every file in /nix/store to be invalid
<clever> the root shell that you get from runInLinuxVM has full read access to the /nix/store outside the sandbox, but the db.sqlite is empty
<clever> more for manual use, to test and confirm things work
<clever> Lisanna: boot the disk image under qemu
<clever> Lisanna: you can also use https://gist.github.com/cleverca22/090514e1979ca1e573efbbc8f7a2cbc2 and skip the ISO step, to boot the resulting image
<clever> yeah, something like that
<clever> unpacking a tarball to /nix should fix that, i think
<clever> you could do it from within a chroot inside the vm
<clever> any storepath you would want to install with nix-env
<clever> Lisanna: or just "nix-env -i /nix/store/foo" to install things into a new profile
<clever> it would need to point a buildEnv i think
<clever> Lisanna: yeah, you would probably want to chroot into the image, and re-run "nix-env --set /nix/store/foo --profile /nix/var/nix/profiles/per-user/clever/" to fix the generations
<clever> but taking an ubuntu image, and just replacing /nix with a new copy is easy
<clever> yeah, incremental changes to /nix are going to be tricky
<clever> Lisanna: it creates a full /nix directory, with the closure of whatever paths you give it
<clever> Lisanna: you could blow away the /nix of the disk image, then just unpack the tar there
<clever> Lisanna: there is also the make-system-tarball, which makes a tar you can just unpack to /nix
<clever> Lisanna: not sure
<clever> jluttine: this makes a 1tb harddrive image, boots a nixos install ISO in qemu with kvm, then boots it without the ISO
<clever> jluttine: the qemu-kvm command in the qemu_kvm package uses it
<clever> jluttine: kvm is just a kernel feature, that has to be used by something else like qemu
<clever> fearlessKim[m]: even for a nix-shell, it will import it into the store, then point $src to the store snapshot
<clever> jluttine: what would the kvm executable do?
<clever> Lisanna: ah, yeah, you dont need 22
<clever> fearlessKim[m]: that all happens at eval time, before a single build has begun
<clever> fearlessKim[m]: the first time that value enters a derivation, nix will import a snapshot of that directory into the store, and hash the whole thing, and replace the value with a storepath
<clever> fearlessKim[m]: is src refering to a path on disk, or a fetchurl based derivation?
<clever> Lisanna: looks good
<clever> but if you use lib.mkForce, it should ignore the previous values
<clever> jluttine: the type of that option is lines, so it will concat each value together, with a \n seperator
<clever> jluttine: that depends on the type of the option, what is the name of the option?
<clever> and that will make it a valid source for nix copy
<clever> in your case, it has to be ran within the vm
<clever> you then run nix-store --load-db on that backup, to recreate db.sqlite
<clever> line 22 is like -qR, and 26 creates a fake database backup for the given closure
<clever> Lisanna: one minute
<clever> Lisanna: so no paths will ever be valid in the store
<clever> Lisanna: ah, i see, the VM lacks any code to initialize db.sqlite
<clever> ah, let me double-check the source for runInLinuxVM
<clever> try quoting the args for --to
<clever> because you didnt specify a thing to copy
<clever> i think its trying to build default.nix from the current directory
<clever> [pid 217] getcwd("/tmp", 4096) = 5
<clever> Lisanna: line 2, the args for --to are missing!
<clever> Lisanna: wait, somethings not right
<clever> Lisanna: that is strange
<clever> Lisanna: try prefixing that with "strace -f" and then gist the output