<clever>
dhess: what about systemctl status hydra-queue-runner?
<clever>
dhess: what was in that log?
<clever>
ghostyy: i think we should take this to #zfs
<clever>
dhess: ok, what about `journalctl -f -u hydra-queue-runner` ?
<clever>
dhess: that one better not be asking for any features!
<clever>
dhess: yeah, got them mixed up a bit, it also doesnt matter what user you run the command as
<clever>
sometimes after a large snapshot destroy, freeing has a value of 5gig, and your free space slowly climbs
<clever>
that means its not actively trying to free blocks
<clever>
ghostyy: what about "zpool get freeing" ?
<clever>
and the .drv is already evaluated
<clever>
dhess: hydra-evaluator
<clever>
nix should never be in nix-env when using nixos
<clever>
dhess: yeah, thats the problem, you have a 2nd copy of nix in nix-env, removing it should fix the issue
<clever>
ghostyy: try destroying those snapshots and see if the error clears up
<clever>
dhess: the user that had the error
<clever>
dhess: i want the non-readlink version
<clever>
ghostyy: nix fixed the version on the fs, but the snapshot is immutable, so it cant be fixed
<clever>
ghostyy: ah, that could also explain things
<clever>
ghostyy: id check to see if you can just delete some of those then, what does `nix-store --query --roots /nix/store/2cqprl9pz1x6zl1nmjwnciqpvh1a02hi-libX11-1.6.5` say?
<clever>
dhess: what does `which nix-store` return?
<clever>
dhess: nix 2.0 daemon doesnt set NIX_REMOTE, but nix 1.0 client still needs that set
<clever>
Lisanna: do a grep -r on all of nixpkgs for that filename
<clever>
ghostyy: ah, ive never seen corruption with zfs before, can you add -C5 to the grep?
<clever>
dhess: you didnt set `export NIX_REMOTE=daemon`
<clever>
ghostyy: which files are corrupt?
<clever>
dhess: pick one of them
<clever>
dhess: the .drv path
<clever>
colemickens: the user on his gists is Dezgeg, he is a nixos user
<clever>
dhess: open the details for a build that isnt buinding, find its .drv url, and then run `nix-store -r --dry-run` on it, can you pastebin the output?
<clever>
dhess: hydra will never log about a missing feature
<clever>
dhess: is the hydra publicly visible?
<clever>
dhess: what features do the slaves have?
<clever>
dhess: are jobs visible in the machine status?
<clever>
dhess: how exactly is it stuck?
<clever>
colemickens: i cant think of anything obvious, id have to start patching the guest kernel or the host qemu and throwing printf's everywhere
<clever>
Lisanna: ah, then you can pretty much just delete the bash heredoc on lines 3,4,6,7 and your done
<clever>
colemickens: mostly that hydra runs some basic nixos tests, that happen to involve 9plan
<clever>
colemickens: hydra wouldnt let nixos-unstable update if such a thing was broken
<clever>
Lisanna: that lets you quickly iterate thru a list of libraries, and also re-patch something that tries to auto-update and unpatch itself
<clever>
Lisanna: if you run nix-build against one of these, it will generate a bash script, and if you run the bash script on an ELF, it will patch it
<clever>
it may even work with a darwin or windows host
<clever>
so the host kernel shouldnt matter
<clever>
colemickens: when the client tries to read something, it will relay the request to the daemon (qemu), which then uses standard posix stuff (readdir, open, read, write) to perform the same request against the host
<clever>
colemickens: the "daemon" side of the 9plan connection is within qemu, and the "client" side is in the guest kernel, both come from the same nixpkgs (your fork)
<clever>
its using 9plan to mount the host fs into the guest directly
<clever>
colemickens: ah, the problem is with 9plan then
<clever>
ghostyy: `nix-store --verify --check-contents --repair` will test the drive and repair any damage, but you should probably replace the disk asap if its truely failing
<clever>
colemickens: right before the switch_root, do "ls -ltrh /fs/nix/store"
<clever>
not sure then, kind of tricky to debug things in that region
<clever>
you need to add another "set -x" to the stage2 script, ah, and your comment says you did
<clever>
colemickens: the exec command replaces that shell with the switch_root command, so the "set -x" ceases to have any effect past that point
2018-04-26
<clever>
you can upgrade a single-user to a multi-user after installing
<clever>
id just run nix-daemon as root
<clever>
and also enforces the rules so 1 user cant cheat and exploit another
<clever>
nix-daemon manages running everything as a single user to solve the problems
<clever>
that wont inherit to the directories that the nix tools create inside /nix
<clever>
because you cant easily give every user write access
<clever>
you need nix-daemon to do a multi-user install
<clever>
if you have +w to /nix, it never even needs root
<clever>
its not designed to have root, and wants to chown /nix to your user
<clever>
:P
<clever>
and then nix-store --delete
<clever>
i think you can use proot to map the same /home/user/whatever to /nix
<clever>
then you can switch back to the faster namespace
<clever>
id try doing just the --delete with proot
<clever>
yeah
<clever>
you can never delete paths while in a namespace
<clever>
Myrl-saki: that breaks the GC root code
<clever>
Myrl-saki: are you in a namespace?
<clever>
Myrl-saki: i dont think nix has an option for that, you have to --delete it (and everything depending on it), then rebuild
<clever>
only an output
<clever>
you cant repair a .drv file
<clever>
need root
<clever>
yeah
<clever>
Myrl-saki: you want --repair-path
<clever>
thats why ive been saying to run nix-collect-garbage
<clever>
so now a collect-garbage will delete more things
<clever>
deleting the 2nd generation made more garbage
<clever>
and there is basically no way to tell the difference between what it depended on, and all the other unused paths
<clever>
but not what it depended on
<clever>
you also deleted the build itself
<clever>
just run `nix-collect-garbage --max-freed 1g` and you should be good
<clever>
run nix-store -qR on the path that doesnt exist anymore, and try to nix-store --delete each and every one of those things
<clever>
treets: the simplest way to delete more things is to `nix-collect-garbage --max-freed 1g`
<clever>
treets: but not what it depended on
<clever>
treets: it only deletes the path you pointed out and the things that depend on it
<clever>
treets: yep
<clever>
tomberek: yeah, that should be fixed
<clever>
you need to use `nix-env -p /nix/var/nix/profiles/system --delete-generations 2` to make system-2-link not active
<clever>
and the symlinks in /nix/var/nix/profiles/ tell you what storepath each generation is
<clever>
ls -l /run/current-system points to the storepath that is currently running
<clever>
it deletes the 2nd generation of the system profile
<clever>
for that, you need nix-env -p /nix/var/nix/profiles/system --delete-generations 2
<clever>
GC roots cant be removed with --delete
<clever>
treets: when given the same path?
<clever>
tomberek: builtins.fetchgit will just run the real `git ls-something` and do what git normally does to filter it
<clever>
treets: you need to first run `nix-store --query --roots` on those, to see why they are still active
<clever>
?
<clever>
treets: the storepath that system-link-3 points to
<clever>
treets: run an ls on the sw/bin directory of those system links
<clever>
to nixpkgs, under nixos/doc/
<clever>
yeah
<clever>
probably could be added
<clever>
you can also use: nix-env -p /nix/var/nix/profiles/system --list-generations
<clever>
all of the ones starting with system
<clever>
ls -l /nix/var/nix/profiles/
<clever>
the builds live in /nix/store, and nix-store --delete deletes them from there
<clever>
if you want to remove osmething, you have to use nix-store --delete
<clever>
no
<clever>
including anything in the grub menu, those cant be deleted
<clever>
so anything actively in-use cant be deleted
<clever>
it will only be things that lack GC roots
<clever>
treets: `nix-collect-garbage --max-freed 1g` will delete a random 1gig worth of un-needed things
<clever>
treets: why do you want to delete some things?
<clever>
if it has no roots, it can be deleted
<clever>
treets: `nix-store --query --roots /nix/store/foo` will tell you if somebody can be deleted or not
<clever>
nix also keeps track of what is needed and will refuse to let you delete something thats potentially in-use
<clever>
ertes-w: simply add autoreconfHook to the nativeBuildInputs
<clever>
ottidmes: its internally managed by nixops based on the state of libvirtd
<clever>
dx__: /nix/store/f1c5k5vac133n9m8kvnda16m8373zm7y-mutt-1.9.4 is not on the binary cache, so something is different on that machine
<clever>
along with what infinisil asked for
<clever>
dx__: can you pastebin the output of this command on each machine: nix show-derivation $(nix-instantiate '<nixpkgs>' -A mutt)
<clever>
dx__: both machines print the same string?
<clever>
dx__: what does this output on both machines?
<clever>
but its simpler to just do pkgs.writeScriptBin, nix-build it, and nix-copy-closure
<clever>
if you are a trusted user (which you are in single-user), you dont even need the output to match up with the hash of the storepath
<clever>
Myrl-saki: also of note, the nix api basically lets you write to any path in /nix/store that does not currently exist
<clever>
phase b: how to hack your own server!
<clever>
that has a different command on it
<clever>
do you have a 2nd pubkey?
<clever>
oops
<clever>
oh
<clever>
<enter>~. will cause ssh to forcibly hang-up
<clever>
thats what i was just guessing! lol
<clever>
Myrl-saki: and if you run `env` from .profile, you may find the original command in an env var
<clever>
Myrl-saki: yep
<clever>
Myrl-saki: it will ignore whatever ssh asked it to run, so you need to force the command to be a wrapper script that runs nix-store --serve --write
<clever>
its probably /bin/bash -l
<clever>
bash sources different files depending on if its login or not
<clever>
i dont think it counts as a login shell when you do that
<clever>
you just need a line like that in authorized_keys