2019-05-15

<clever> dhess: can you pastebin the dmesg?
<clever> dhess: also, the old /boot/grub/state from before you deploy anything should give details
<clever> dhess: that FS can only mount when booting via EFI
<clever> [clever@system76:~]$ mount | grep efi
<clever> efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
<clever> dhess: i suspect that the nvme aws machines might also be uefi, you should look into what the state of a freshly booted ami is, `nixops deploy --create-only` to stop it from messing with it
<clever> dhess: /home/clever/apps/nixpkgs/nixos/modules/virtualisation/amazon-image.nix: boot.loader.grub.device = if cfg.hvm then "/dev/xvda" else "nodev";
<clever> so if your motherboard lacks drivers in the rom, grub cant read the nvme, and you cant cheat by putting grub on a legacy disk
<clever> dhess: grub assumes the EFI firmware provides nvme drivers
<clever> dhess: and anoyingly, despite grub "supporting" nvme, grub DOESNT support nvme!
<clever> dhess: for extra fun, on real hardware, you CANT boot legacy nvme, because it just doesnt appear over the normal legacy api's, so uefi is the only way to boot it
<clever> nixops doesnt know its aws, and doesnt know what defaults to use
<clever> dhess: oh, that would explain it
<clever> dhess: you can also try just setting it yourself, to the correct value
<clever> dhess: and this table
<clever> dhess: nixops should set it for you, based on the instance type
<clever> so if it was nvme, then incorrectly got changed to xvda, it will correctly want to re-run grub-install
<clever> another is the grub device
<clever> one of the flags is the version of grub
<clever> dhess: and if that file doesnt match, it will re-run grub-install
<clever> dhess: /boot/grub/state has the state of various grub flags
<clever> dhess: yeah
<clever> dhess: sounds like a bug in nixops, the boot.loader.grub.device isnt set right
<clever> Unode: ive been using zfs for a few years, and havent seen any weird corruption like that
<clever> Unode: something like that would turn me off an FS forever
<clever> Unode: it could be more stray nulls in various files
<clever> lrwxrwxrwx 1 root root 11 Feb 12 20:26 /dev/disk/by-label/nixos -> ../../xvda1
<clever> /dev/disk/by-label/nixos 1.5T 456G 961G 33% /
<clever> dhess: dont think so
<clever> dhess: a decent number of them
<clever> MmeQuignon: do you want to open a shell with a built copy of gitlab_blog, or do you want to open a shell suitable for building gitlab_blog?
<clever> sphinx is already in the buildInputs, so you shouldnt have to list it again, enless you also want it in the shell
<clever> if you want both sphinx and whatever the default.nix is
<clever> MmeQuignon: nix-shell -p python3Packages.sphinx 'callPackage ./default.nix {}'
<clever> MmeQuignon: nix-shell -p 'callPackage ./default.nix {}' should work
<clever> Unode: perhaps combined with trying to boot while the disk is full
<clever> Unode: ah, maybe find a config flag in syncthing to make it stop before it crashes into a wall
<clever> Unode: ncdu can also be handy here
<clever> Unode: do you know what caused it to run into the ground?
<clever> Unode: how the heck that does that happen!?
<clever> Unode: eeek!
<clever> cizra: that will fetch nixpkgs from the current dir
<clever> cizra: nix-build -E 'with import ./. {}; libfprint.override { thinkpad = true; }'
<clever> that will probably do what you wanted
<clever> cizra: nix-build -E 'with import <nixpkgs> {}; libfprint.override { thinkpad = true; }'
<clever> cizra: that will import <nixpkgs>, check if that directly has a thinkpad arg (it doesnt), then discard the arg, and basically ignore it
<clever> cizra: that wont pass thinkpad to libfprint
<clever> run500: nix-prefetch can already give you the hash of a rev
<clever> run500: and then every other VCS will want their hashing in nix, and it will get bloated
<clever> run500: it would need a new type of fixed-output derivation, which would require everybody to upgrade nix to be able to build it
<clever> nix doesnt care how it creates $out, so you are free to use whatever you want (curl, torrent, git, svn, carrier pidgeon)
<clever> and the hash keeps them in line
<clever> run500: fixed-output derivations are just declaring the hash of $out, as a promise of purity, to be granted impure network access
<clever> run500: and there is still the issue of nix itself knowing how to validate that it matches a given rev
<clever> run500: yep, but by default, it deletes the .git at the end, before nix validates the hash is correct
<clever> so you dont have any trace of the .git dir
<clever> run500: but fetchFromGitHub just downloads the auto-generated zip from github
<clever> run500: and you would need to clone the entire history to do that
<clever> run500: also, the rev is a hash over the commit, which contains the hash of the previous commit, so you need the entire chain of commits to validate the rev as being right
<clever> run500: the nix itself doesnt understand git, so all it can do is hash the directory
<clever> and maybe have it recover on its own, with a log to warn you
<clever> Unode: should probably file an issue about uid-map corruption being non-obvious
<clever> cizra: fetchFromGitHub
<clever> Unode: that file is mainly so if you delete a user, then later re-add it, it gets the old uid back
<clever> Unode: you should be able to safely delete it, and nixos will recover
<clever> Unode: thats a valid json file on my end
<clever> oooooo
<clever> Unode: tab completion?
<clever> i believe it has always had that name
<clever> ls -lh /nix/store/*users-groups.json
<clever> thats still valid json, so it shouldnt be an issue
<clever> Unode: what does the first line of `hexdump -C /path/to/json | head` report?
<clever> Unode: this will just check all of them!
<clever> [root@amd-nixos:~]$ for x in /nix/store/*users-groups.json; do echo $x ; jq < $x > /dev/null ; done
<clever> and see what it says
<clever> Unode: cat the json | jq
<clever> ls -l /nix/store/*/bin/jq
<clever> Unode: hmmm, do you have jq installed?
<clever> Unode: hmmm, i would just run it directly, rather then sourcing it
<clever> Unode: sure
<clever> youll find a line where it runs perl on that user-groups script, and gives it a json
<clever> Unode: read the `activate` script at that path
<clever> Unode: above that, you should see the path to the nixos build it was running
<clever> Unode: did the journal say which json file was invalid?
<clever> that should get you a link equivelant to ethernet, and from there its all the same
<clever> then you will want to bring the IF up (ip link set XXX up), then run wpa_supplicant with &
<clever> wep or wpa?
<clever> depends on your crypto though
<clever> wifi is harder to setup
<clever> which does it normally use?
<clever> wifi or wired?
<clever> there is also a --repair command, but it will depend on what is corrupt, there are many choices
<clever> Unode: the --verify --check-contents should confirm that
<clever> Unode: that sounds like your nix store is corrupt, so various things are failing on boot
<clever> Cale: there ^^
<clever> make-package-set.nix: callHackage = name: version: callPackageKeepDeriver (self.hackage2nix name version);
<clever> [clever@amd-nixos:~/apps/nixpkgs/pkgs/development/haskell-modules]$ grep -r --color callHackage
<clever> to view the journal
<clever> Unode: oh, `journalctl -b -1` maybe
<clever> Unode: you could try disabling docker in configuration.nix, and do `nixos-rebuild boot` to update things
<clever> Unode: maybe `nix-store --verify --check-contents`
<clever> Unode: export PATH again, as i showed before
<clever> Unode: its reading an absolute symlink from outside the chroot
<clever> Unode: tell it to continue anyways
<clever> Unode: then systemd wont even be ran
<clever> Unode: change the init= to init=/bin/sh
<clever> dhess: the above path is where the last active generation is kept
<clever> dhess: /run is a tmpfs, so current-system wont exist on-disk
<clever> -r-xr-xr-x 1 root root 20377 Dec 31 1969 /nix/var/nix/profiles/system/bin/switch-to-configuration
<clever> dhess: one sec
<clever> dhess: and behind the scenes, `--install-bootloader` just does `export NIXOS_INSTALL_BOOTLOADER=1`, and switch-to-configuration then checks for that var
<clever> dhess: but, you can just run `/run/current-system/bin/switch-to-configuration (switch|boot)` to get the same effect, using whatever is currently active (last deployed)
<clever> dhess: first, you cant really get nixos-rebuild to easily respect the nixops config
<clever> 41 export NIXOS_INSTALL_BOOTLOADER=1
<clever> 40 --install-bootloader)
<clever> dhess: one minute
<clever> rather then silently putting things in the /boot dir of /
<clever> so if i ever forget to mount, it fails hard
<clever> Unode: thats why i like to `chmod 0 /boot` before mounting it
<clever> dhess: `man nixos-rebuild` -> --install-bootloader
<clever> so you need to mount everything that matters where it belongs
<clever> it assumes systemd took care of that already
<clever> nope
<clever> Unode: then try rebooting again
<clever> Unode: boot will update /boot for whatever it just build
<clever> oops, wrong tab-complete
<clever> yangm97: you want boot
<clever> Unode: switch will always fail under chroot
<clever> dhess: ive also had plans to build ssh into the bootloader somehow, either directly into grub, or by booting into a thin distro, that presents a menu, and kexec's into the real distro
<clever> dhess: then reverse those steps to put it back where it belongs
<clever> dhess: if you need to, you can disconnect the root EBS, reconnect it to a second instance, and then mount the device and poke around
<clever> Unode: i would also bindmount them
<clever> Unode: try to chroot again then, using the bash from the system profile, and then `nixos-rebuild build` and see what happens
<clever> cizra: and instead of list, you can also ` --delete-generations 42`
<clever> [root@amd-nixos:~]$ nix-env --profile /nix/var/nix/profiles/system --list-generations
<clever> cizra: thats why i just dont -d, and i manually delete generations
<clever> just mounting it may show some things
<clever> no need to chroot
<clever> Unode: try from the usb stick you booted from, after mounting the rootfs
<clever> Unode: you can run dmesg to see those errors
<clever> yeah, its unlikely to corrupt if it wasnt in use when you ran out
<clever> nix-env profiles will also be lost
<clever> if the db is lost, then so is all of /nix/store, and it will need to re-download/build everything
<clever> and is modified every time that is modified
<clever> Unode: the sqlite db keeps track of what is in /nix/store/
<clever> sqlite is fairly hardened against a lot of things
<clever> Unode: IO errors feels more like a failing drive
<clever> what does dmesg say?
<clever> check dmesg
<clever> that doesnt mean the DB is corrupt
<clever> what points towards it being corrupt?
<clever> though completion mostly comes from bash sourcing the bashrc stuff in /etc
<clever> use the bash from the path i gave above and you should be good
<clever> it turned itself off :P
<clever> 2019-05-15 14:44:52 -!- grahamc[m] [gchristens@gateway/shell/matrix.org/x-tgmttfnyebwhgwkt] has quit [Quit: Idle kick: User has been idle for 30 days.]
<clever> gchristensen: from matrix
<clever> gchristensen: you got booted!
<clever> i'm not sure if its all documented in one spot, i mostly just memorized how everything works
<clever> that will get you 80% of the stuff you expect
<clever> export PATH=/nix/var/nix/profiles/system/sw/bin
<clever> or just ignore nixos-enter, and manually setup the env vars
<clever> you can also try installing nix, and then just nix-env the right thing
<clever> it needs to be installed on the host end
<clever> that will likely fail
<clever> that will chroot, and set the env vars right
<clever> Unode: you want nixos-enter
<clever> oh right, the thing that isnt booting right
<clever> Unode: rebooting which system?
<clever> Unode: yep
<clever> infinisil: my general idea, was to have a single list, containing mac/hostname/ip, and then dynamically generate zonefiles, and dhcp config
<clever> this has code to generate A and AAA records, from an IP
<clever> infinisil: this is something ive been wanting to get back to
<clever> and if it has changed, increment some RUNTIME state, and sed it in
<clever> every time the service starts, diff the zonefile
<clever> one solution ive thought of, is to generate the seq number when bind starts
<clever> and then the dns wont go backwards
<clever> yeah, rollbacks will randomly either go to an old seq number, or a new one (depending on how aggressive you GC)
<clever> infinisil: impurely call `date` ?
<clever> Unode: yep
<clever> Unode: you can also add `single` to the kernel cmdline, then systemd will drop into single-user mode, rather then starting services
<clever> lucus16: what is it printing? and what is the expr returning?
<clever> lucus16: try `nix-instantiate --eval file.nix`
<clever> possibly with --eval
<clever> lucus16: nix-instantiate
<clever> gchristensen: sometimes the reverse, ive had to search thru a dozen pages of irc logs to find that URL before
<clever> not really
<clever> jD91mZM2: and i found that last one, via the "spam" {^_^} just gave :P
<clever> only revs that are in the history of some channel have coverage in the cache
<clever> jD91mZM2: this service also records the history of every channel
<clever> deploy also obeys -I foo=bar, but it wont remember it
<clever> and a comment in my house.nix reminds me what it should be, if i ever loose the state
<clever> i'm using that to pin nixpkgs to a single rev, so it only updates when i choose to update the pin
<clever> jD91mZM2: you can also use `nixops modify` to change any param you gave during `nixops create`
<clever> # nixops modify -d house deployments/house.nix -I nixpkgs=https://github.com/nixos/nixpkgs/archive/dae9cf6106d.tar.gz
<clever> wut? :P
<clever> that would explain things
<clever> although, i was also jamming a 2nd llvm into the buildInputs, rather then changing clangs llvm
<clever> matthewbauer: my understanding is that clang just calls out to llvm to do 90% of the work, but it feels like clang has a whitelist of platforms, and is blocking what should just work
<clever> for a chip with 32kb of flash....
<clever> so i went back to avr-gcc and avr-libc, and eventually got a 90kb firmware blob
<clever> i then realized, i was distracted by rust support AVR via llvm, and realized i wanted to compile plain c code
<clever> if i instead tell clang to emit llvm IR, then i run llvm on that, it works for basic things, but for crypto code, llvm segfaults!
<clever> but, clang doesnt know that, so clang refuses to target avr
<clever> this is enough to get avr "support" in llvm
<clever> llvm.overrideAttrs (old: { cmakeFlags = old.cmakeFlags ++ [ "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR" ]; })
<clever> i recently experimented with AVR support, and found some things
<clever> clang (via llvm) is the only way i can see to feasible target all the things!
<clever> Ericson2314: but wont the compiler need to support the target cpu at a bare minimum?
<clever> Ericson2314: ah, so you could do insane things like cross-compile an ios toolchain that runs on an rpi, using an x86 machine, lol
<clever> kisik21: yep, thats how nix stops you from getting internet access
<clever> kisik21: if the nix sandbox is enabled, your already in a container
<clever> asheshambasta_m: rebuild switch obeys slightly different rules
<clever> { pkgs, lib, ... }: on line 1
<clever> asheshambasta_m: lib.mkForce
<clever> cizra: its a fairly simple task, and its all described in the nixpkgs manual
<clever> asheshambasta_m: yes, if you omit enabled=true; then the nginx.service wont even be generated, and you cant manually start it
<clever> cizra: i think libGLU_combined would be the best one
<clever> ,locate libGL.so.1
<clever> asheshambasta_m: yeah, the docs for `services.nginx.virtualHosts.<name>.listen` do say its a list
<clever> so services.nginx.virtualHosts."example.com".listen = [ { port = 1234; } ];
<clever> asheshambasta_m: ah, it looks like its a list based option
<clever> asheshambasta_m: anything
<clever> then it will still be "enabled" but its not wanted by multi-user.target, so it wont run on boot
<clever> asheshambasta_m: systemd.services.nginx.wantedBy = mkForce []; i think
<clever> cizra: and you want do the patchelf in a nix expression, not against a random path you find in /nix/store/
<clever> cizra: nix-locate, there is also a ,locate in the {^_^} bot
<clever> even some open-source things are handled like that, electron for example
<clever> cizra: yep
<clever> cizra: patchelf
<clever> and that will override defaults
<clever> rnhmjoj: of note, you can set <modulename>.<optionname>=<value> on the kernel params
<clever> and line 200 is where the force-loaded modules get loaded
<clever> rnhmjoj: i dont think the initrd even has its own /etc/modprobe.d folder
<clever> so at the nix level, it allows up to 8 jobs, each doing `make -j8`
<clever> which is 8 for the buildkite machines
<clever> angerman: cores=0, max-jobs=8, so it will auto-detect based on actual core-count
<clever> *looks*
<clever> (propagated)(native)BuildInputs
<clever> angerman: dynamic linking didnt work, at all, so it had to staticly link everything
<clever> angerman: me and taktoa where crazy enough to make it cross-compile QT and gstreamer, before Ericson2314's improvements
<clever> angerman: :D
<clever> so you cant do things like build a windows ghc, that targets arm
<clever> i dont think nixpkgs lets you set 3 arches currently
<clever> but, is it even a cross-compiler, if it targets windows, and runs on windows?
<clever> yeah
<clever> but the error doesnt even mention, "your cross-compiling ghc to run on windows"
<clever> yeah, nixpkgs cant cross-compile ghc to run on windows, because gmp isnt setup right in the cross packages
<clever> yeah, the .name of the failing derivation confirms its the wrong one
<clever> "ghc-8.6.4-x86_64-pc-mingw32"
<clever> nix-repl> x86_64-mingw32.pkgs.haskell.compiler.ghc864.name
<clever> ah
<clever> (totally unrelated to the bug)
<clever> and there is a stray space after the python3 a few lines down
<clever> angerman: this line has a tab character in it!
<clever> my vim is setup to render tabs, so i they stick out like a sore thumb
<clever> theres a tab in the nix!!
<clever> 176 ->>>>>>>dontAddExtraLibs = true;
<clever> angerman: i dont remember exactly what targetPackages refers to in this context
<clever> nix-repl> x86_64-mingw32.pkgs.system
<clever> "x86_64-windows"
<clever> angerman: wut?
<clever> { __raw = true; recurseForDerivations = false; stdenv = { ... }; }
<clever> nix-repl> x86_64-mingw32.pkgs.targetPackages
<clever> siraben: did you enable virtualisation.virtualbox.host.enable = true; ?
<clever> (some of it)
<clever> angerman: and this is where the confusion should end!
<clever> nix-repl> x86_64-mingw32.pkgs.targetPackages.gmp
<clever> error: attribute 'gmp' missing, at (string):1:1
<clever> 158 "--with-gmp-includes=${targetPackages.gmp.dev}/include" "--with-gmp-libraries=${targetPackages.gmp.out}/lib"
<clever> angerman: if i narrow it down to a single attr, i can get the old failure
<clever> error: attribute 'gmp' missing, at /nix/store/sf0phgki1bddmglmglsxd27p5lv164y3-source/pkgs/development/compilers/ghc/8.6.4.nix:158:28
<clever> nix-repl> x86_64-mingw32.pkgs.haskell.compiler.ghc864
<clever> angerman: yeah, its just making it tricky to debug by iterating over all of the attrset
<clever> gentauro: i just leave that on defaults
<clever> angerman: you may want mapAttrs or a related
<clever> gentauro: internationalization
<clever> so its creating thunks that will fail to evaluate
<clever> ah, you have a bunch of overrides, for every ghc version, including versions that dont exist
<clever> error: attribute 'ghc842' missing, at /nix/store/m8sz1nxxq2rm89rjz564r8fk73dn0mr9-source/config.nix:104:26
<clever> nix-repl> x86_64-mingw32.pkgs.haskell.compiler
<clever> yep, i also get that path
<clever> out -> /nix/store/wndwgmxmnx53q6pdnld2l22jic9afn6g-gmp-6.1.2-x86_64-pc-mingw32
<clever> nix-repl> :b x86_64-mingw32.pkgs.gmp
<clever> as a side-effect, hydra would try to (cross) build all of nixpkgs, so this kind of thing shouldnt be pushed
<clever> ive modified jobs= to expose that entire pkgs attr, so i can now view it in repl
<clever> inherit fetchTarballFromJsonWithOverride fetchTarballFromJson x86_64-mingw32;
<clever> ah, and that just fetches pins/iohk-nix-src.json, and calls its default.nix
<clever> importPinned is being given some very basic config flags, the rev/sha of a nixpkgs, and the system/crossSystem
<clever> i'm now going thru the callstack, to see what it says
<clever> and this then gives a backtrace
<clever> nix-build release.nix -A x86_64-pc-mingw32-ghc864.x86_64-linux --show-trace
<clever> angerman: i can reproduce the issue in a plain `nix repl release.nix`