2018-03-05

<clever> Lisanna: you may want to analyze the access.logs and see when something was recently used
<clever> you can also just do foo = pkgs.callPackage ./foo.nix {}; within an overrides or overlay, and then pkgs.foo later on
<clever> and callPackage deals with giving it the right args
<clever> srid: you may need (pkgs.callPackage ./foo.nix {})
<clever> remove it from the cache manually, and try to -r again
<clever> with nix-store --delete, you can then delete it, but it remembers that the binary cache had it
<clever> Lisanna: with `nix-store -r /nix/store/foo` you can forcibly download something from a cache
<clever> Lisanna: yeah, there is a chance it may fail there
<clever> and the kernel will finish the delete when the handle is closed
<clever> Lisanna: under most unix based OS's, the file will keep serving after being deleted
<clever> Lisanna: ah, correction, i think that only gets bad if a path is deleted, then re-made: https://github.com/NixOS/nix/issues/1885
<clever> Lisanna: and the ttl to expire things is only applied with nix-daemon starts, in at least one version
<clever> Lisanna: there is a bug in nix 2.0, where it will assume the path is still in the cache, and hard-fail if its missing
<clever> for 1.11, you just have to manually delete the narinfo and the nar it refers to
<clever> NIX_REMOTE and --to/--from in 2.0 all accept store URI's in the same form
<clever> yeah
<clever> Lisanna: ah, i'm guessing you might be able to use NIX_REMOTE=file://... nix-store --delete
<clever> Lisanna: which binary cache?
<clever> tnks: i only found that one today, while trying to find the code handling sigs:
<clever> tnks: with this, there is now 2 signatures in the sigs field, space seperated

2018-03-04

<clever> [root@system76:/etc/nix]# nix sign-paths $(readlink /home/clever/iohk/daedalus/result) -k /etc/nix/secret-key-file
<clever> tnks: so nix-daemon is already signing things after the builds finish, if configured
<clever> tnks: aha, i set secret-key-files in nix.conf, deleted a store path, then re-built it with nix-build, now db.sqlite contains a signature
<clever> tnks: i believe you need write to the directory to grab the right lock files for opening
<clever> tnks: this, as root, is how i view a sig, right now, its an empty value since its not signed
<clever> [nix-shell:/nix/var/nix/db]# sqlite3 db.sqlite 'select sigs from ValidPaths where path = "/nix/store/b46sv85mp79xww7q31fv3cpz0c0zbw2n-daedalus"'
<clever> tnks: cache.nixos.org and friends
<clever> tnks: db.sqlite is the entire state of /nix/store/
<clever> tnks: binary-cache-v3 is a cache of what is present on the binary caches
<clever> Myrl-sak1: also, nix-store --query --roots tells you why its alive
<clever> Lisanna: yeah, that doesnt really work then
<clever> the nix-store man page shows how to generate the pair
<clever> :q!
<clever> or add the right user to trusted
<clever> so you can either --to root@foo, or sudo nix-copy-closure --from source
<clever> but with 1.11,nix-copy-closure with a trusted user on the receiving end, just ignores the signatures
<clever> maybe 2.0 changes things some
<clever> tnks: trusted-users are allowed to import unsigned paths
<clever> ive had no issues with the old nix-serve
<clever> tnks: which fields did you have to fill in?
<clever> ottidmes: got a link to those docs?
<clever> acertain: i think you can nix-shell -p '(lib.attrValues qt5)'
<clever> ottidmes: ah, ive only gotten signing working with nix-serve and hydra
<clever> ottidmes: for the binary caches, try to query the http://host/hash.narinfo file with curl
<clever> ottidmes: oh, is this with local builds, or copying between machines?
<clever> avn_: nix 2.0 saves signatures to db.sqlite when it downloads things, but nix-serve still re-signs everything it serves
<clever> ottidmes: double-check to see what nix-store your using on each user
<clever> ottidmes: is the client 2.0 still? realpath $(which nix-store)
<clever> ottidmes: oh, that error, is your nix-daemon on 1.11 still?
<clever> tnks: yeah, plain http is best for a simple binary cache
<clever> which causes issues with build slaves
<clever> so if you 'ssh user@host' nix works fine, but if you 'ssh user@host nix-store --help' it fails
<clever> related, nix-store isnt always in $PATH for non-interactive shells
<clever> ah, that sounds new
<clever> so you can push/pull closures, and initiate builds remotely
<clever> tnks: and a similar protocol us used for `ssh user@host nix-store --serve [--write]`
<clever> tnks: if nix is ran without root, it uses the nix-daemon protocol over a unix socket to proxy everything over, and the daemon uses local-store
<clever> tnks: if nix is ran as root, it will directly use local-store
<clever> tnks: there is a few more reads before that, which deal with the handshake
<clever> tnks: this is the main loop in nix-daemon, that reads an operation code, and handles it
<clever> ottidmes: not sure
<clever> tnks: one min
<clever> tnks: only in code
<clever> tnks: the same way you can nix-store --query --hash
<clever> tnks: id also check to see if you can query the signature over the nix-daemon protocol
<clever> tnks: yeah
<clever> tnks: i think the code mentioned it being a space seperated list
<clever> ottidmes: kill isnt really needed, but umount/close all filesystems, and sync, before you kexec
<clever> sign things when they are built, and maybe also add extra signatures over time?
<clever> as an example
<clever> the secret for 2017 has been destroyed, so no matter how badly it gets exploited, you can trust the signatures from 2017
<clever> but it would add some timestamp like features
<clever> you would have to share every public you have made, and publish them ahead of time
<clever> just an optional thing you could do
<clever> you could even rotate the key every year, and know for sure, it was signed prior to X, and that secret has since been destroyed
<clever> with 2.0, you nix-daemon can sign when builds finish, then the signature is just in db.sqlite
<clever> i also gain access to your secret key
<clever> what if i exploit nix-serve and gain control of the process?
<clever> one thought i had, nix-serve needs read access to your secret to sign things
<clever> and let cachecache, cache the nix-serve output
<clever> oh, random thought, use the old nix-serve, behind cachecache
<clever> tnks: though its also caching upstream caches
<clever> tnks: right now, cachecache needs a list of upstream caches, but it could also have a mode for /nix/store
<clever> tnks: it could even be an extension to my cachecache project
<clever> tnks: thats also an option
<clever> its probably faster to remake it then to learn perl :P
<clever> rewrite it in c++!
<clever> ah, it just builds the perl subdir of the nix its pointed at, with some perl flags
<clever> line 145 adds them for 2.0
<clever> tnks: one min
<clever> those are safer to just wipe
<clever> Myrl-sak1: that will reset all generations, so nix-env and nixos state is wiped clean
<clever> Myrl-sak1: or try just normal nixos-install
<clever> Myrl-sak1: try a cd / after you chroot
<clever> Myrl-sak1: what exactly does it fail with?
<clever> Myrl-sak1: pong
<clever> zandy[m]: ah, then youll need to wait for lets encrypt caches to expire, then it can retry
<clever> tnks: you can either patch it to use bzip2 -1 which is faster, or just change out the entire compression algo for something faster
<clever> zandy[m]: scroll up some, maybe an hour or 2
<clever> tnks: yeah, its hard-coded to 30, youll either want a package-override, or a reverse proxy that serves a different nix-cache-info and proxies the rest
<clever> zandy[m]: check `journalctl`, scroll to the end and see if there are any acme related errors
<clever> makefu: but nginx has to come up with an example.com cert for acme to work, and acme doesnt reload nginx
<clever> makefu: the acme one runs itself on rebuild-switch, and at regular intervals
<clever> zandy[m]: i believe you need to restart nginx now
<clever> * SSL certificate problem: self signed certificate
<clever> zandy[m]: and also, port 443 is blocked, security groups on aws
<clever> zandy[m]: try systemctl restart nginx, then try the url again
<clever> 2018-03-04 17:21:15 * zandy[m] sent a long message: zandy[m]_2018-03-04_21:21:14.txt <https://matrix.org/_matrix/media/v1/download/matrix.org/nmzidvkqMNFwvCIKxVYPnnQD>
<clever> though ive seen android do some pretty nutty things with git
<clever> they start with different initial commits, so the standard commands dont allow it
<clever> ive tried doing it before, and it refuses to pull branches from other repos, when they lack a common ancestor
<clever> so i can just throw every git repo into a single bare .git dir
<clever> gchristensen: something ive been wanting, is a way to dedup objects between git repos
<clever> maybe leave a gig or 2 free for apps to use
<clever> as long as the size is under ram+swap, it should be safe
<clever> Myrl-sak1: mount -o remount,size=2G /path/to/that
<clever> Myrl-sak1: check df -h, youll see a .rw-store
<clever> Myrl-sak1: pong
<clever> it takes an extra 20 or 30 seconds just to unpack that fat initrd
<clever> rescue-boot gives you that option, while not slowing down normal boots
<clever> ah
<clever> netboot-server configures full network booting with the same images
<clever> MichaelRaskin: rescue-boot puts the kernel&initrd into /boot and adds a grub option for recovery
<clever> MichaelRaskin: oops, wrong tab-complete
<clever> Myrl-sak1: both rescue and netboot do that, the entire rootfs is a squashfs in the initrd
<clever> nix 2.0's copy command can help, but it needs better docs
<clever> if they have similar configuration.nix's, that will get most of what you want
<clever> thats one option, it will get a complete copy of the desktop
<clever> the issue is no bash in /mnt/nix/store/
<clever> and then run nixos-install
<clever> then youll need to let it copy into the tmpfs and swap
<clever> Myrl-sak1: try nixos-install --chroot, does that work?
<clever> ah, that is a bit more tricky
<clever> Myrl-sak1: for example, on the laptop, nix-copy-closure --from desktop /nix/store/foo
<clever> Myrl-sak1: nix-copy-closure from either machine
<clever> Myrl-sak1: you can also modify the netboot to run sshd on bootup
<clever> Myrl-sak1: partially, you have to `systemctl start sshd` and `passwd root`
<clever> Myrl-sak1: i use zfs, which checksums all data and also works in an immutable manner to reduce corruption
<clever> Myrl-sak1: improper shutdown?
<clever> Xianwen: fork the repo on github, clone your fork, edit it on your local machine, push to the fork, and then you can issue a pull request
<clever> Xianwen: and if you add a PR to nixpkgs, for adding cwm, and also add it to https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/x11/window-managers/default.nix#L14 then everybody else can get cwm easily
<clever> which will check for damage, and repair whatever it finds
<clever> there is also nix-store --verify --check-contents --repair
<clever> yeah, --repair-path
<clever> and ive broken my system the exact same way, so dont worry, lol
<clever> nix refuses to even download a path until the deps are valid
<clever> Myrl-sak1: a storepath must not exist without its dependencies
<clever> Myrl-sak1: it will delete the entire /nix/store/hash-library-name and everything that depended on it, recursively
<clever> Myrl-sak1: i dont think you ever said which one on irc
<clever> Myrl-sak1: but if it breaks, you can always plug the cable in again and reboot from the image again
<clever> i dont remember if nixos-rebuild works inside that netboot image
<clever> Myrl-sak1: you can also set those in nix.conf and /etc/nix/machines
<clever> Myrl-sak1: youll need to adjust the /etc/nix/nix.conf on the host, and add the url and pubkey for the cache
<clever> Myrl-sak1: networking.nat does that automatically
<clever> lejonet: lib.optional and the related functions
<clever> Myrl-sak1: networking.nat
<clever> Myrl-sak1: its running entirely from ram, so its safe to unplug the cable
<clever> Myrl-sak1: and refresh the dhcp lease on it
<clever> Myrl-sak1: once it boots, you can optionally just plug the broken machine into any router
<clever> lexwhere: nix-bundle and arx are in my plans
<clever> lexwhere: the nix variant isnt written yet
<clever> Myrl-sak1: just run nixos-install
<clever> lexwhere: i'm actually working on something just like that, that uses namespaces to put it at ~/nix/
<clever> Myrl-sak1: if you dont nix-channel --update often, a single force can delete months worth of generations
<clever> but you did delete everything that refered to the path you where deleting
<clever> yeah, you didnt delete everything
<clever> Myrl-sak1: just leave your configuration.nix as-is, and mount the right partition under /mnt/boot/
<clever> Myrl-sak1: there is also a minor problem, since your booting via legacy, configuring efi will be a bit more tricky, but we can assume that the efivars are already setup
<clever> Myrl-sak1: netboot_server.nix doesnt support EFI currently
<clever> Xianwen: just copy that to /etc/nixos/cwm.nix, replace all dwm's with cwm's, and add imports = [ ./cwm.nix ]; to the configuration.nix, and it should become a valid option
<clever> Xianwen: and it has to be +x
<clever> Xianwen: its called .xsession on nixos
<clever> Myrl-sak1: you may need to enable network booting first
<clever> Myrl-sak1: try that option and see what it does
<clever> pong
<clever> then just mount your existing FS's under /mnt and run nix-install
<clever> pick network boot from the bios
<clever> but the wan one can be wireless
<clever> the server will need 2 interfaces
<clever> Myrl-sak1: network.lan must be set to the NIC facing the broken machine, which should be an ethernet card
<clever> Myrl-sak1: network.wan must be set to the name of the internet facing NIC
<clever> Myrl-sak1: you have to add it to the imports section of configuration.nix, and set the 2 nixos options on lines 35&40
<clever> Myrl-sak1: its a nixos module
<clever> Myrl-sak1: it shares its internet over the same ethernet that netboots the broken system
<clever> Myrl-sak1: thats why netboot_server.nix includes NAT
<clever> Myrl-sak1: which machine is working, which is broken?
<clever> and nat
<clever> Myrl-sak1: this fully configures dhcp, bind, httpd, netboot, tftpd
<clever> Myrl-sak1: do you have other nixos machines?, is a laptop still working?
<clever> same
<clever> Myrl-sak1: oh, id just back them up to another machine, how much is used?
<clever> Myrl-sak1: nixos-install will fix it without loosing any files or your config
<clever> Myrl-sak1: just dd it directly to the root of the usb drive
<clever> Myrl-sak1: the nixos ISO's are already usb images
<clever> Myrl-sak1: that is what --repair-path is for
<clever> Myrl-sak1: if you boot from any install media for nixos, you can mount your FS's to /mnt, and re-run nixos-install to repair it from the existing configuration.nix
<clever> ive done it, it breaks a lot :P
<clever> Myrl-sak1: you shouldnt have forced it either
<clever> Myrl-sak1: echo instead
<clever> tnks: currently, it can only cache the narinfo files
<clever> tnks: so you fire up a cache-cache instance, put it into your binary-caches list, and it will proxy all requests upstream, and cache all replies
<clever> tnks: basically, its a transparent http proxy for binary caches, with support to query several caches behind the scenes
<clever> tnks: and i also have this project: https://github.com/cleverca22/cachecache
<clever> tnks: also there is the irc logs in the /topic, on botbot.me
<clever> tnks: i plan to go over each one, lookup the context, and document it more in that gist
<clever> tnks: all relevant lines from my irc logs: https://gist.github.com/cleverca22/bc86f34cff2acb85d30de6051fa2c339
<clever> tnks: hydra pre-compresses everything before uploading
<clever> tnks: its just a dumb S3 bucket behind cloudfront, all it can do is serve static files
<clever> if running a local cache on your LAN, you may want to make it 30, so you get gigabit speeds
<clever> so smaller numbers come first
<clever> and in general, you want to hit cache.nixos.org rather then a hydra
<clever> tnks: i see a hydra listed at 100
<clever> tnks: look at the priority field in `curl https://cache.nixos.org/nix-cache-info`
<clever> heh, found this in my logs
<clever> 2017-10-18 16:07:40< tnks> clever: when you say "nix copy" it sounds like something new, not "nix-copy-closure".
<clever> so its basically just an rsync
<clever> and streams it directly into the local nix-daemon
<clever> i think that calls nix-daemon to dump a .nar of the path, without compression
<clever> i need to go over `egrep --color 'nix .*copy' irclogs/freenode/#nixos.log` and see what they all do
<clever> i think its in 2.0, its just documented very well
<clever> which replaces nix-push
<clever> there is a variant of it that will creare a directory full of .narinfo and .nar.xz files
<clever> then any dumb http server can handle the cache
<clever> which will pre-compress and pre-sign a bunch of static files
<clever> the only way to improve things is with the deprecated nix-push, or its replacement in nix 2.0
<clever> hydra also does the exact same thing, so no real benefit there
<clever> so it only has to compress once when serving
<clever> wait nvm, it asks the nix-daemon for the narhash when signing
<clever> the db.sqlite signatures would help there
<clever> oh, but it may do it twice?, once to sign, and once to serve
<clever> every time you request an archive, it will `nix-store --dump` the storepath, compress it, and serve that
<clever> nope, thats because it doesnt cache the compressed nar, or even the nar itself
<clever> my plan is sort of the reverse, build a closure in hydra, tar it up, then unpack that tar to a tempdir and import it into a local store
<clever> i'm planning on using it to import a closure from a tarball
<clever> tnks: are you using `nix copy`? i need to get into using it more heavily as well, cross-mount stores and such
<clever> but this one isnt capable of merging, so that is lost if you set it to anything
<clever> tnks: if the config option is entirely missing from nix.conf, it will have a sane default
<clever> tnks: ah, one sec
<clever> tnks: because its in the config section and not default, it will merge this default with whatever you specified
<clever> then nix-serve doesnt even need to know your key
<clever> but another benefit i can see, if you sign all paths after building them
<clever> tnks: yeah, it would still work, just signed with a different key
<clever> sending those signatures instead would be an improvement
<clever> tnks: but nix 2.0 is able to retain the original cache.nixos.org signatures in db.sqlite
<clever> tnks: nix-serve will re-sign any path its serving with the configured key, and only that key
<clever> joko: there is a bug that tries to store the entire os its building in a tmpfs
<clever> joko: add swap and resize the .rw-store directory with `mount -o remount,size=2G /path/to/it`
<clever> tnks: it is now fetching the signature of everything in my store...
<clever> [root@system76:/nix/var/nix/db]# time nix copy-sigs --all -s https://cache.nixos.org
<clever> tnks: db.sqlite now retains the original signature(s) of things, and nix-serve shouldnt be re-signing things
<clever> tnks: with the advent of nix 2.0, nix-serve needs to be fixed i guess
<clever> tnks: aha, there is a new addition to db.sqlite, the ValidPaths table now has a sigs field!
<clever> tnks: what is this....
<clever> return "copy path signatures from substituters (like binary caches)";
<clever> tnks: oh, there is also a NarInfo class, that can convert to/from string: https://github.com/NixOS/nix/blob/561e977f51c1d9ec55e4a70791958d4e214df465/src/libstore/nar-info.cc#L85-L114
<clever> cache.nixos.org lacks garbage collection, so that cant really happen
<clever> but if somebody was garbage collecting their own cache, and the build was not bit-for-bit reproducable, the narhash would come out different, resulting in a new signature for the new nar
<clever> tnks: for the nixos cache, yeah
<clever> return "1;" . $storePath . ";" . $narHash . ";" . $narSize . ";" . join(",", @{$references});
<clever> it generates a fingerprint using the storepath, narhash, narsize, and closure, then signs that string with a secret
<clever> tnks: if your wondering how the signatures are generated, here is a perl example: https://github.com/edolstra/nix-serve/blob/master/nix-serve.psgi#L36-L44
<clever> nixos just entirely ignored config.nix with no obvious cause
<clever> i have ran into nasty problems before when the file behind $NIXPKGS_CONFIG was made, and i hadnt seen the option before
<clever> ottidmes: you can just do nixpkgs.config = /home/clever/.config/nixpkgs/config.nix; in configuration.nix
<clever> yeah, both sets of packages will have the same packageOverrides
<clever> then it will always be whatever config the nixpkgs it came from is using
<clever> so you could do self: super: { unstable = import <unstable> { config = self.config; }; }
<clever> pkgs.config is the config from the parent of the overlay
<clever> so pkgs.hello is the stable version and pkgs.unstable.hello is unstable
<clever> ive also seen overlays/overrides used to set pkgs.unstable to contain the entire nixpkgs from the unstable channel
<clever> it can even be used to just store generic strings or any other nix value inside the pkgs set
<clever> it just gets a new random ip every time you boot
<clever> nick_l: yeah, ive often ran things without an elastic ip or vpn
<clever> tnks: i believe trusted users can just bypass the signature checks entirely
<clever> nick_l: ah
<clever> dmj`: your late to the miso party :P
<clever> nick_l: the ps -eH x output would have told me so, but you didnt give it
<clever> nick_l: maybe its waiting for ssh on a different machine then
<clever> nick_l: what does `journalctl -f` on the remote server say, during the deploy?
<clever> nick_l: what does it output if you re-run the same deploy?
<clever> tnks: i think you also need a special CLI flag to make it sign things
<clever> nick_l: what does df -h and df -i report for the remote server?
<clever> nick_l: what else is in that area?, it sounds like the copy already finished
<clever> nick_l: can you pastebin the `ps -eH x` from the machine running nixops?
<clever> nick_l: what about with |grep nix