2018-07-24

<clever> as silently as max logging can be, lol
<clever> most programs silently accept too many, and just go to the max log level
<clever> i'm never sure how many v's are needed, so i just spam them, lol
<clever> can you pastebin a censored copy of the log output?
<clever> elvishjerricco: when i ping-store with -vvvvv, i can see paths to the credentials file, and the keyid
<clever> region = eu-central-1
<clever> [default]
<clever> elvishjerricco: ive also got a .aws/config file, that sets the region
<clever> elvishjerricco: any difference if you run ping-store with -vvvv ?
<clever> elvishjerricco: my next step would usually be to get strace logs, but those will include the secret keys
<clever> elvishjerricco: and have you double-checked that the accesskey works when you run it thru s3cmd ?
<clever> when i set things up in that PR, i found it refused to work without the region being set
<clever> elvishjerricco: and is the region set correctly in the hydra config? https://github.com/input-output-hk/iohk-ops/pull/352/files#diff-d164bc05da9475caf99ae31f786560e9R81
<clever> elvishjerricco: and that file is owned by hydra-queue-runner?
<clever> elvishjerricco: so the creds are now in `/var/lib/hydra/queue-runner/.aws/credentials` ?
<clever> elvishjerricco: for which user?
<clever> elvishjerricco: how did you configure the access keys?
<clever> etu: since you have no channels, it made an empty channels profile
<clever> etu: what if you run `nix-channel --update` as your own user?
<clever> etu: use `sudo -i` and then `nixos-rebuild ..` not `sudo nixos-rebuild`
<clever> i'll see if i can poke krisajenkins again about upstreaming the fix
<clever> older though, and different user
<clever> elvishjerricco: i got it from https://github.com/input-output-hk/iohk-ops/pull/339
<clever> but if i dig thru the git history...
<clever> cant remember who i got this fix from
<clever> i see 3 open PR's about prefetch problems
<clever> they forked off a while ago and the hydra one just got old
<clever> i believe so
<clever> elvishjerricco: and line 13 to make hydra use the patched version
<clever> elvishjerricco: you need hydra-nix-prefetch-git.patch and gawk to fix submodules
<clever> elvishjerricco: one min
<clever> i hadnt seen the loopbackip changes, but they appear to be backwards compat, and the old config still works
<clever> if you know my public ip, you can connect to any of those sourcePort's, and it will forward you to the destination in my LAN
<clever> elvishjerricco: this is the bulk of the config for my router at home
<clever> elvishjerricco: it was MUCH simpler back then, and didnt support connections from itself
<clever> ah, nixos didnt have a .loopbackIPs option when i had setup my router
<clever> also, this looks different from how i remember it a few months ago...
<clever> and the 3rd will check all packets after routing has been computed, and will apply an SNAT
<clever> the 2nd will i believe apply to all patches coming into the wan interface, before routing has been applied, and check that its targeting a loopback ip, port, and proto, then DNAT it again
<clever> the first will match all outbound packets targeting the loopback ip, forwarded port/protocool, and use DNAT to redirect it to the destination of the forwarding
<clever> for each combination, it will generate 3 iptables commands
<clever> so for every combination of both of those options, it will run 58-78
<clever> and line 57 looks over every loopbackIPs
<clever> line 51 loops over every forwardPorts entry
<clever> it sometimes helps to skip the description and go to the implementation
<clever> elvishjerricco: if you select an option here, and clicked on declared in, you can see how it works: https://nixos.org/nixos/options.html#forwardports
<clever> elvishjerricco: only connects to its external ip, coming in over the external interface
<clever> elvishjerricco: it wont include connects to localhost, or from the nat'd box itself
<clever> yeah, that can be a problem
<clever> there has been talk of having it improved to parse the old wrapper, and add to it
<clever> but it does technicaly "work"
<clever> samueldr: currently, it makes a mess of ...foo-wrapped-wrapped-wrapped i think
<clever> you can probably now delete the propagated-user-env-packages and not loose anything (other then machine-breaking bugs)
<clever> just remove support for dynamic plugin loading, and wrapProgram the plugin paths, or patch the search logic directly
<clever> but does any such plugin even exist in nixpkgs?
<clever> and it feels like QT is using that, so that you can nix-env -i a plugin and then all apps find it
<clever> but i feel that propagated-user-env-packages sort of breaks the entire point of using nix
<clever> then they wont collide, and both will be available and working
<clever> why not just use $out/lib/qt-5.9.1 ??
<clever> but then the libs break when you mix 5.9.0 and 5.9.1!
<clever> qt uses $out/lib/qt-5.9
<clever> one thing that could be improved on the issue i saw
<clever> havent read over yours yet
<clever> because an old qt app was in the profile, and the new qt from plasma looked in ~/.nix-profile and exploded
<clever> samueldr: i have even seen this break the entire desktop environment (you cant login at all) twice
<clever> then things break hard
<clever> samueldr: and if X and Y want different versions of qtbase, it silently gives only 1
<clever> samueldr: if you install package X with nix-env, it will silently also include qtbase in your ~/.nix-profile/
<clever> samueldr: something else ive seen fail hard, is propagated user env packages
<clever> so you only have to define options and set global stuff in one place
<clever> elvishjerricco: any config you set on defaults, applies to every machine in the cluster
<clever> elvishjerricco: something that may help is the defaults machine
<clever> elvishjerricco: that is strange, i would not expect it to fail like that
<clever> elvishjerricco: note that this will also copy nixpkgs.pkgs, which is likely going to cause more problems
<clever> elvishjerricco: why do you want to refer to another machines nixpkgs config?
<clever> elvishjerricco: can you gist what you are trying and the stack from --show-trace ?
<clever> elvishjerricco: nodes.hostname i believe
<clever> damn, lol
<clever> > pkgs.riot-web.meta.position
<clever> ah
<clever> > pkgs.riot-web.meta.path
<clever> tenten8401[m]: that package looks what you want
<clever> > pkgs.riot-web.meta.description
<clever> tenten8401[m]: i see a riot-web package
<clever> samueldr: let me grab a link
<clever> samueldr: the syntax
<clever> tenten8401[m]: ive pulled up the first commit on nixpkgs, thats not nix ....

2018-07-23

<clever> robstrr: it has been merged into nix 2.0
<clever> __monty__: some names map to several attribute paths
<clever> then sync the config.nix over and install it again
<clever> RetardedOnion: create a config.nix with a buildEnv override, and install that
<clever> RetardedOnion: ~/.nix-profile can only be modified with nix-env, so you cant rsync it to another machine
<clever> bebarker: nix-env only ever adds packages to ~/.nix-profile/

2018-07-22

<clever> it may need to be patched to allow it
<clever> das_j: sounds like it would be better to disable the automatic recompile, and have it compiled once by nix
<clever> das_j: all files must have a timestamp of 1 unix time
<clever> das_j: its part of making everything pure and reproducable
<clever> das_j: nope
<clever> bed*
<clever> without wiping the cache, it will take 30 days to correct itself
<clever> angerman: if you delete soemthing from the binary cache, it will 404, but the positive cache says it exists, so nix will refuse to just build it locally
<clever> angerman: there are also some bugs in nix to do with the positive caching
<clever> nix will automatically recreate that if its missing, and then re-query every binary cache
<clever> angerman: you can also just `rm -v /root/.cache/nix/binary-cache-v5.sqlite*`
<clever> for the remote end, it has to be in nix.conf
<clever> angerman: so if you try something, push, then try again, it wont re-check the binary cache
<clever> angerman: note that both ends will cache the negative replies from the binary cache
<clever> packageOverrides = pkgs: { renoise = pkgs.renoise.override { releasePath = /path/to/something; }; };
<clever> tertle||eltret: line 5 should be
<clever> angerman: if that is enabled, then the build slaves will prefer fetching directly from a cache, rather then letting you upload
<clever> angerman: --option builders-use-substitutes true
<clever> angerman: ah, found it
<clever> auto substitute = settings.buildersUseSubstitutes ? Substitute : NoSubstitute;
<clever> and this gets printed if it blocks for 15mins, lol
<clever> printError("somebody is hogging the upload lock for '%s', continuing...");
<clever> angerman: i also see a seperate lock file for uploading to build slaves
<clever> AutoCloseFD uploadLock = openLockFile(currentLoad + "/" + escapeUri(storeUri) + ".upload-lock", true);
<clever> angerman: and then it can decide between uploading the local copy, or having the remote machine hit up a binary cache
<clever> angerman: also of note, both tools must have the entire set of inputs locally, before they contect the build slave
<clever> tertle||eltret: refer to lines 6 and 28 of my example
<clever> tertle||eltret: you are missing the pkgs: { and } for packageOverrides
<clever> angerman: the default for that flag may differ between the 2 tools
<clever> angerman: there is a special flag, that will allow or dis-allow the remote machine to do the binary cache work on its own
<clever> angerman: one difference that could exist, is the modes for nix copy closure
<clever> they should use the same library behind the scenes, so that shouldnt be possible
<clever> nix build is more silent by default
<clever> angerman: nix-build prints all log output to stdout, which may eat some cpu, -Q will silence it
<clever> yeah, they should be interchangable, once you know how to translate the args
<clever> but once you know the quirks, you can translate between the 2 easily
<clever> i find the nix-build api to be more predictable, but nix build is the only one with the better progress UI
<clever> angerman: nix build uses the free argument as either an attrpath or an expression, based on some weird logic, but nix-build needs -E or -A for that
<clever> angerman: nix-build treats the free argument as a filename, nix build needs a -f for the filename
<clever> sophiag: the automatic GC wont touch generations, so you still have to manually delete those
<clever> tertle||eltret: https://gist.github.com/cleverca22/b8f9f73372fe4585d29706c7a9fd83c8 line 12 is an example doing the same type of thing to steam
<clever> ah
<clever> tertle||eltret: you need an entry in your config.nix packageOverrides, that does renoise = pkgs.renoise.override { releasePath = /path/to/something; };
<clever> tertle||eltret: a: thats a "lovely" packaging idea, b: then how did i install it without a licenses? lol
<clever> any time i reach 3gig of free space on /nix/store, it will start a gc, and it will stop when it gets to 6gig free
<clever> sophiag: i was digging around in the nix source a month ago and discovered the min-free and max-free options
<clever> sophiag: one min
<clever> sophiag: and then youll probably spend an extra 10-20mins re-downloading from the binary cache, depending on your modem speed
<clever> samueldr: by default, a normal `nix-collect-garbage` will delete anything not currently in use, which may include build-time tools, and anything source like
<clever> samueldr: you can also run "busybox strings foo" to run the strings sub-command
<clever> tertle||eltret: the CI system for nix
<clever> the default version in the busybox attribute could be changed, as long as the initrd uses an override to re-enable that
<clever> half the command-not-found results tell you to try installing busybox, which just horribly breaks the machine
<clever> samueldr: busybox probably should also be hidden from nix-index, lol
<clever> hydra probably doesnt build it, so no index
<clever> ,locate renoise
<clever> angerman: so once the process that was using the build slave quits, the lock should be auto-released by the kernel
<clever> angerman: and the locks are opened with O_CLOEXEC
<clever> angerman: ah, on closer inspection, nix uses fcntl(fd, F_SETLK on the lock files in /nix/var/nix/current-load
<clever> tertle||eltret: you need to enable unfree packages in config.nix
<clever> angerman: it looks like its no longer on a tmpfs, but deleting those files should reset its memory of in-use build slaves
<clever> angerman: check `ls -ltrh /nix/var/nix/current-load`
<clever> tertle||eltret: installs the package refered to by an attribute path
<clever> currentLoad = store->stateDir + "/current-load";
<clever> angerman: AutoCloseFD lock = openLockFile(currentLoad + "/main-lock", true);
<clever> its unfree, so it will always be hidden on the website
<clever> > pkgs.renoise.meta.license
<clever> > pkgs.renoise.meta.description
<clever> tertle||eltret: it is packaged, and the above will install it
<clever> tertle||eltret: i just installed it using nix
<clever> tertle||eltret: nix-env -iA nixos.renoise to install the nix package of it
<clever> tertle||eltret: so installing things the non-nix way will always fail
<clever> tertle||eltret: all binaries must be patched by nix before they can run
<clever> sophiag: so its still plausible that the overheat bug was caused in fall of 2017, and it took until summer to notice
<clever> sophiag: but even 17.09 was failing, but thats during winter...
<clever> sophiag: i initially thought it was a recent channel update, because it never went into thermal shutdown prior to this summer
<clever> angerman: so its lost at reboot
<clever> angerman: i think nixos keeps that state on a tmpfs
<clever> angerman: let me find the state...
<clever> angerman: the local machine keeps track of how many builds are being ran
<clever> i initially thougth 80 amps was normal, because i only tested linux, and the math matches up with the TDP of the cpu
<clever> linux is just jumping all over the place while idle
<clever> under windows, with max load, the voltage wont even leave 0.86v
<clever> notice the amps going literally off the scale, lol
<clever> sophiag: the amps and volts on the cpu, when idle, and then starting a stress test to max out all cores
<clever> sophiag: https://imgur.com/piR9KwZ
<clever> or windows is doing something seriously wrong, and can only use 10% of my cpu's power :P
<clever> so either linux is doing something seriously wrong
<clever> i started to suspect a linux problem, after i noticed windows at max load barely used 20 amps in the same monitoring util
<clever> and just doing a rebuild was making it get dangerously close to overheating again
<clever> in my case, the issue is a weird voltage control problem with the cpu, that leads to the motherboard dumping 80 amps into the poor cpu, which then overheats to the point of the machine cutting power
<clever> sophiag: i did have to reduce my own rebuilds down to 1 core
<clever> sophiag: sadly, 17.09 also shows signs of the same problem, so its either a long-standing issue in the linux kernel, or a config issue in configuration.nix itself
<clever> sophiag: with only very minor changes to my configuration.nix, i was able to downgrade the whole machine to 17.09 effortlessly
<clever> sophiag: the power of nixos config is amazing, i switched my desktop from nixos-unstable to 18.03, and then even 17.09 today, trying to narrow down a problem
<clever> tertle||eltret: is it giving an error?
<clever> the comments i read dont mention any cpu stalls, have you tried testing ssh?
<clever> yeah, thats the comment i mentioned
<clever> sophiag: that comment implies that its less to do with what grub did to the GPU, and more what grub isnt forwarding to the linux kernel
<clever> looks like its still "linux"
<clever> linux ($drive1)//kernels....
<clever> [root@amd-nixos:~]# grep linux /boot/grub/grub.cfg
<clever> sophiag: and for the comment a few down about linux vs linuxefi, just read the grub.conf file in /boot and see which one its using
<clever> sophiag: from the first comment, it sounds like a race condition, the systemd unit just needs to wait a bit longer
<clever> for legacy booting, i believe grub uses the same legacy bios calls dos would have used, and i has not messed with advanced stuff like pci passthru for me, so it should also not cause issues with wonky drivers
<clever> sophiag: and if it did cause problems, i would expect every bootloader to have the same problems
<clever> sophiag: for efi booting, there are dedicated functions provided by the firmware for drawing to the screen, i would expect that to not cause any gpu driver problems
<clever> tertle||eltret: add yourself to the wheel group, example linked
<clever> i prefer to always use declarative
<clever> now you can undo the changes to configuration.nix to re-add yourself, nixos-rebuild switch, and possibly run `passwd username` to set the pw again
<clever> tertle||eltret: try ctrl+alt+f1 and then login from text mode?
<clever> tertle||eltret: using grub or systemd-boot?
<clever> tertle||eltret: do you not have a root password?, have you tried booting an older generation from the bootloader?
<clever> petersjt014[m]: looks pretty normal, i would expect that to boot
<clever> petersjt014[m]: you can probably also just `fdisk /dev/sdX` and then run the print command within it (its probably just p)
<clever> petersjt014[m]: list
<clever> petersjt014[m]: if you put the sd card back into a reader and do `fdisk -l /dev/sdX` what does it list?
<clever> turn off the power, remove the card, turn on the power
<clever> petersjt014[m]: if you remove the SD card entirely, then reconnect power, does it blink the same way?
<clever> 2 flashes: The SD Card cannot be read.
<clever> petersjt014[m]: how many blinks is it doing?
<clever> petersjt014[m]: you can lookup what the blink pattern means online

2018-07-21

<clever> dhess: ok, it doesnt appear to be a "recent" problem in nixos, even nixos-17.09 exibits the problem
<clever> dhess: nixos-18.03 still fails
<clever> dhess: trying nixos-18.03.git.d6c6c7f ...
<clever> juhe1: sounds like a bug in nix then, it should have terminated that process automatically
<clever> infinisil: then youll want a custom unpackPhase that will check $src, and decide if it should copy it directly, or run the original unpackPhase and filter it
<clever> infinisil: what are you trying to actually do?
<clever> infinisil: the default unpackPhase only works for tars, zips, and directories, it wont know what to do with a file
<clever> infinisil: youll want to replace the default unpackPhase then, or just switch to runCommand
<clever> juhe1: what is process 20938?
<clever> infinisil: why is the src another nix file?
<clever> juhe1: can you pastebin the output of `ps -eH x` after you ctrl+c the build?
<clever> juhe1: strange, because it should be stopping
<clever> juhe1: is nix-daemon actually building?, check `ps -eH x` closely
<clever> dhess: yeah, above channel still spikes to 80 amps
<clever> dhess: chat logs show i started to have cpu overheat problems around june 10th, the above is june 6ths's nix-channel --update
<clever> dhess: ok, build a downgrade to nixos-18.09pre140958.696c6bed4e8, though it may not be old enough...
<clever> dhess: thats back from early june
<clever> dhess: giving this a try: nixos-rebuild boot -I nixpkgs=/nix/var/nix/profiles/per-user/root/channels-62-link/nixos/
<clever> windows refused to leave the low voltage, even at max load
<clever> its under 10 amps at idle, but the voltage is jumping up&down a lot
<clever> %Cpu(s): 0.2 us, 0.2 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
<clever> dhess: the only thing running in linux right now is my vpn and synergy
<clever> dhess: i see similar spikes of up to 70 amps before the bios has even initialized the gpu, and when sitting idle at grub
<clever> crosshair v formula-z motherboard
<clever> samueldr: yep
<clever> but underclocking has caused pulseaudio to hang
<clever> it can also remotely under/over clock the motherboard
<clever> dhess: yep
<clever> dhess: and measuring at the wall, it draws 100 watts more then cinebench in windows maxing it out
<clever> dhess: and the cpu doesnt support dynamic clocking, so its always running at full clock
<clever> dhess: thats from running `stress -c 8`
<clever> dhess: look at this insanity! https://i.imgur.com/piR9KwZ.png
<clever> dhess: https://i.imgur.com/HmSZ1hu.png strange, windows idles at 0.86 volts, linux jumps up and down a lot
<clever> takeda: mkdir $out
<clever> infinisil: if only one thing provides a given directory, it links the directory, if 2 things provide a dir, it creates one, and links all of its contents
<clever> infinisil: buildEnv links everything by default
<clever> dhess: heh, during shutdown it rises to 1.3 volts on the cpu
<clever> dhess: and you can remotely change the clock freqs
<clever> infinisil: and buildEnv wont do?
<clever> dhess: the motherboard has some monitoring software that runs over usb, and lets me monitor the cpu amps, many voltages, fan speeds, and temps
<clever> hmmm, its only at 0.86 volts on windows...
<clever> 80 amps at 1v is very different from 80 amps at 120v
<clever> dhess: remember, this is 80 amps at about ~1 volt
<clever> dejanr_: its a 125 watt cpu, running at 0.8 volts right now
<clever> so even with a linux host, your uptime is bounded by the stability of windows
<clever> dejanr_: and even then, windows can only use the card once per boot, then the host has to do a full powercycle
<clever> dejanr_: ive had a lot of trouble getting gpu passthru to work, i had to use a special linux param to entirely blacklist linux from touching the pci device, which meant you dont even get a text console to login at
<clever> windows is pulling 10 amps at max load on all 8 cores...
<clever> without a baseline, i just assumed that was normal, it matches up with the watt rating and the voltage in the same util
<clever> i powered up the motherboard monitoring software, and found that the cpu is drawing over 80 amps
<clever> lately, nixos has been having a lot of overheating problems, going into thermal shutdown repeatedly
<clever> sphalerite: ok, something very fishy is going on with my desktop....
<clever> the nix man pages can be a bit confusing
<clever> nix-store {--realise | -r} paths... [--dry-run]
<clever> takeda: your tool will need to fetch the source and parse it as it generates the derivation
<clever> > pkgs.pypi2nix.meta.description
<clever> takeda: something like pypi2nix?
<clever> that also works
<clever> and that allows you to patch the unpacked copy of the source before the configurePhase runs
<clever> the patchPhase runs after the stdenv has copied (and unpacked) $src to the working directory
<clever> pkgs.fetchGit has the option, but it can cause problems
<clever> manveru: fetchGit deletes .git
<clever> mjrosenb: you can add a package override to your test.nix to change the version of the other package
<clever> or adjust them to include nix's version
<clever> mjrosenb: the simplest thing is to run haskell.lib.doJailbreak over it, or just remove them from the cabal file
<clever> mjrosenb: by default, nix ignores the versions in the cabal file
<clever> but thats kind of ugly also
<clever> sphalerite: id just make a struct with a enum and union, or maybe a std::pair?
<clever> mjrosenb: then you want nix-shell -A project.env
<clever> mjrosenb: use nix-shell test.net -A env
<clever> ghasshee: i believe you just add libudev to the buildInputs and then #include <libudev.h>
<clever> ah
<clever> sphalerite: splits exists within a tab on vim, so each tab has its own split layout