2017-07-31

<clever> pie_: try adding pkgconfig to the nix-shell args
<clever> mpcsh: the code in here can also sometimes be used as a hint as to what a builtin is expecting: https://github.com/NixOS/nix/blob/master/src/libexpr/primops.cc#L959-L977
<clever> mpcsh: for example, just run the inner most fetchurl in nix-repl
<clever> mpcsh: and also run the parts in nix-repl
<clever> so it will be doing 2 downloads at eval time
<clever> but since you lack the nix hash, you need to use builtins.fetchurl again to download it
<clever> you can then readfile, fromjson, and build another url based on the version in there
<clever> if you run builtins.fetchurl on the above url, it will check it with some internal caching
<clever> mpcsh: depends on where you get the json from
<clever> Infinisil: you can also just builtins.fromJSON and builtins.readFile then manipulate it with nix
<clever> ixxie: if you are importing nixpkgs in your nix file, then you have to set config in there
<clever> ixxie: it will depend on what your running nix-shell against as well
<clever> kiloreux: strange, i dont see the default.nix in that dir
<clever> ixxie: maybe --arg config '{ allowUnfree = true; }'
<clever> kiloreux: i need more of the ls output
<clever> ixxie: set allowUnfree in your users config.nix file
<clever> kiloreux: and ls -ltrh
<clever> kiloreux: what is the output of nix-build -v ?
<clever> i suspect that kiloreux is using the full script, rather then the stub in the gist
<clever> same path as me
<clever> pull also works, but ive found push auth to be tricky
<clever> very
<clever> srhb: are you aware that you can just run "git clone" on a gist url?
<clever> srhb: your default.nix is one byte longer, as is the script.py
<clever> srhb: can you gist the output of running ls -ltrh and nix-build -v
<clever> srhb: yv4148n89z0j5n57fwz6kwcj640fnvsh ?
<clever> srhb: and if you test that gist?
<clever> kiloreux: what error do you get with that gist?
<clever> works on this end
<clever> can you gist both files?
<clever> builtins.fetchTarball
<clever> fetchtarball, not fetchurl
<clever> oh right
<clever> can you gist the entire file?
<clever> import takes that path, and returns the object (a nix function)
<clever> builtins.fetchurl takes a url and returns a string pointing to the store
<clever> it needs the with still
<clever> you almost never need to delete things from the store
<clever> just leave it
<clever> once you put in the right git rev, yeah
<clever> to isolate it from changes the user may have in ~/.nixpkgs/config.nix
<clever> an example i yanked from my #nixos logs
<clever> import (builtins.fetchurl https://github.com/NixOS/nixpkgs-channels/archive/nixos-unstable.tar.gz) { config = {}; };
<clever> lol
<clever> yeah
<clever> or that
<clever> set the $NIX_PATH env variable
<clever> effect*
<clever> you have to pass the same -I every time if you want it to keep having an effet
<clever> the -I flag doesnt change any config
<clever> kiloreux: you appear to be using nixpkgs f2c4af4e, not a7c8f5e419
<clever> kiloreux: nix-instantiate --find-file nixpkgs
<clever> kiloreux: what does this return?
<clever> nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion
<clever> yeah, its just a string value
<clever> i think
<clever> so even if you unquote it, its still a string
<clever> Infinisil: the nix parser treats all URI's as strings
<clever> Infinisil: looks like it
<clever> Infinisil: one sec
<clever> Infinisil: those should all be reasonably modern
<clever> 1tb Seagate Barracuda 7200.14 (AF) * 3 + Western Digital Caviar Green 1tb
<clever> Infinisil: the raidz1 is over modern drives, the L2 is on a slightly older drive
<clever> Infinisil: raidz1 over 3 drives, with an L2 arch on a 4th to speed up reads
<clever> Infinisil: an example of how slow gc can get
<clever> Jul 31 04:07:36 nas nix-gc-start[29805]: note: currently hard linking saves 86631.22 MiB
<clever> Jul 31 00:17:23 nas nix-gc-start[29805]: deleting unused links...
<clever> nope
<clever> thats what wrapProgram is doing
<clever> its in the search path, it should work
<clever> thats strange
<clever> kiloreux: can you gist the contents of result/bin/output?
<clever> yeah
<clever> tilpner: no real way to do that right now, and calling wrapProgram twice results in an infinite loop
<clever> tilpner: are you trying to modify how it wraps it?
<clever> /home/clever/apps/nixpkgs/pkgs/tools/security/eid-viewer/default.nix: wrapProgram $out/bin/eid-viewer --suffix LD_LIBRARY_PATH : ${pcsclite}/lib
<clever> /home/clever/apps/nixpkgs/pkgs/tools/filesystems/fuse-7z-ng/default.nix: wrapProgram $out/bin/${pname} --suffix LD_LIBRARY_PATH : "${libs}/p7zip"
<clever> /home/clever/apps/nixpkgs/pkgs/development/compilers/rust/cargo.nix: ${stdenv.lib.optionalString stdenv.isDarwin ''--suffix DYLD_LIBRARY_PATH : "${rustc}/lib"''}
<clever> a grep over nixpkgs should provide examples
<clever> i think its this one
<clever> you want to use the --append mode in wrapProgram
<clever> it will eval to the full store path
<clever> by including the exact string, ${gnome2.gtk}/lib/, inside default.nix
<clever> so you want to append ${gnome2.gtk}/lib/ to LD_LIBRARY_PATH
<clever> kiloreux: gnome2.gtk is the attribute path with that library
<clever> which is anti-nix
<clever> that relies on the end-user installing things
<clever> clearing LD_LIBRARY_PATH will only make it worse
<clever> you need to append the gtk lib dir to LD_LIBRARY_PATH
<clever> kiloreux: which arguments did you give to wrapProgram?
<clever> Infinisil: i think the nix-hash command can do it
<clever> and auto only gets cleaned up at gc
<clever> and the autolinks point into an area of /tmp that gets cleaned up pretty fast
<clever> i think nix-push has one derivation per derivation
<clever> ah yeah
<clever> Infinisil: and remember, nix uses a non-standard base32
<clever> Dezgeg: how the hell does that happen.......
<clever> kiloreux: can you gist everything from the console when you tested it?
<clever> and it has to read all of that to give a directory listing
<clever> Infinisil: yeah
<clever> c2d is xfs, the rest are zfs
<clever> ls -lhd /nix/store/.links/
<clever> c2d: drwxr-xr-x 2 clever clever 1.9M Feb 8 2016 /nix/store/.links/
<clever> nas: drwxr-xr-x 2 root root 1.6M Jul 31 14:39 /nix/store/.links/
<clever> amd: drwxr-xr-x 2 root root 1022K Jul 28 22:10 /nix/store/.links/
<clever> i'm only building a small subset of nixpkgs
<clever> tilpner: you ran nix-store --optimize at some point, and it made hardlinks to save space
<clever> causing reads and writes to be interleaved
<clever> nix-collect-garbage also has to lstat every single entry, and conditionaly unlink
<clever> thats from my new hydra setup
<clever> real 4m0.277s
<clever> 1589822
<clever> [root@nas:~]# time ls /nix/store/.links | wc -l
<clever> and thats on a less used nix build slave
<clever> real 0m0.710s
<clever> 30453
<clever> clever@c2d ~ $ time ls /nix/store/.links/ | wc -l
<clever> yep
<clever> LnL: so duplicate files hardlink to the same hash, and share the on-disk data
<clever> LnL: after nix-store --optimize, every single file in the entire store is hardlinked to /nix/store/.links/<hash>, based on the hash of the file
<clever> Infinisil: nix-collect-garbage has to list the contents, and delete everything with a hardlink count of 1
<clever> LnL: and /nix/store has the same problem to a lesser degree
<clever> jluttine: nix-prefetch-url -A foo.src
<clever> too many files in one directory
<clever> LnL: reading /nix/store/.links/ takes hours
<clever> so its still better
<clever> but its only a cost at gc, while dedup would be a cost at every write
<clever> yegortimoshenko: on my NAS, that part of the GC can take upwards of 10 hours
<clever> yegortimoshenko: depending on your filesystem, it can make garbage collection unusually slow
<clever> kiloreux: you want nix-build -I nixpkgs=https://github.com/NixOs/nixpkgs/archive/a7c8f5e419.tar.gz
<clever> i ctrl+f'd for binary, found no definitions, and then looked at the with's
<clever> ixxie: because they used with statements, you guess
<clever> kiloreux: so nix-build continues to use the previous version
<clever> kiloreux: and it doesnt change any config
<clever> kiloreux: that nix-env command tells nix-env to install EVERYTHING in nixpkgs
<clever> ixxie: line 41, i suspect its using chromium.upstream-info.binary
<clever> kiloreux: why do you think its doing the wrong thing?
<clever> kiloreux: nix will detect when things have changed and do the right thing
<clever> kiloreux: 90% of the time, you dont need to
<clever> ixxie: can you link an example?
<clever> hodapp: when you give it a path to a nix file, it just runs the top level value, in the same way as -A
<clever> hodapp: they wont run until you run the unpackPhase and patchPhase
<clever> yeah
<clever> kiloreux: i tested it against a7c8f5e419
<clever> kiloreux: which channel are you building this on?
<clever> kiloreux: my opencv doesnt try to load gtk when i run it
<clever> kiloreux: this is a different so file
<clever> kiloreux: thats not the same problem, thats an entirely different problem
<clever> kiloreux: try closing that shell and re-opening it, other env variables you set might be messing with it
<clever> Phillemann: does it have the ability to detect if the file has already been downloaded?
<clever> Phillemann: if its doing network during a build, its already broken, and that should be stopped first
<clever> nix-build creates a result symlink in the directory you ran it in
<clever> ls -ltrh
<clever> kiloreux: that doesnt start with ./result/
<clever> kiloreux: and what happens when you run ./result/bin/output ?
<clever> kiloreux: can you gist the output of nix-build?
<clever> nixos-rebuild may also accept -v
<clever> rydnr: add -v to the nix-build command and you will see every file its reading
<clever> rydnr: it looks like you customized NIX_PATH and it now ignores all channels
<clever> pie_: i mainly use nix-shell and then ./configure && make
<clever> nix-env is not the way to test things
<clever> kiloreux: i run nix-build on it
<clever> bkchr: use wrapProgram to prepend sshfs to PATH
<clever> list everything you need in the default.nix and nix will fetch them for you
<clever> the entire point of nix, is to make it work without the user having to install other things
<clever> nope
<clever> surprisingly, python.withPackages delt with the opencv libs automatically
<clever> kiloreux: i dont get any errors when i run the script produced by that gist
<clever> it doesnt have the LD_LIBRARY_PATH fix yet, i'm waiting for opencv3 to compile to test that
<clever> no need to mess with PYTHON_PATH
<clever> that is what i have so far
<clever> but you have done neither
<clever> because the nix expression should also be patching LD_LIBRARY_PATH
<clever> thats what the nix expression should be doing for you
<clever> kiloreux: how is numpy getting into the python search path?
<clever> FRidh: all interpreted languages, because we cant patchelf the interpreter
<clever> kiloreux: attribute, not store path
<clever> FRidh: its a problem common to all of java and python
<clever> kiloreux: and what was the nix attribute path for opencv?
<clever> kiloreux: can you add a simple python script that imports the problem module?
<clever> kiloreux: i dont see that changing LD_LIBRARY_PATH any
<clever> gchristensen: and would also open up the option to make it more cross-platform, if we ever get multi-user on ubuntu for ex
<clever> gchristensen: does the above util sound useful?
<clever> gchristensen: sudo update-nix.conf --append binary-caches ... --append binary-cache-public-keys ... --restart
<clever> gchristensen: i was just thinking, what about a util for nix on darwin, that would help manipulate nix.conf and reloading nix-daemon
<clever> thats the kernel to blame
<clever> ah
<clever> sauyon: in this case, the error was because a list containing a set was passed to makeLibraryPath
<clever> cmcdragonkai: this is how i tracked it down: https://gist.github.com/cleverca22/872b21404062218409315a39a080091a
<clever> cmcdragonkai: qt5.full
<clever> qt5.full.out 0 s /nix/store/kg36kcq4kx2m0j33ynz3bjllsd7li4zk-qt-5.8.0/lib/libQt5Core.so.5
<clever> correction, only qt5 is a set
<clever> cmcdragonkai: qt4 and qt5 are sets, not derivations
<clever> cmcdragonkai: can you gist your nix expression?
<clever> yeah
<clever> i'm not sure when the last or next one even is here, lol
<clever> sauyon: mkdir -pv $out/bin/
<clever> same, i just use cp
<clever> cmcdragonkai: the stdenv will cd into it for you, so .
<clever> my NAS was setup before i did that, and now i have to delete several weeks worth of backups just for nix-collect-garbage to have any effect
<clever> i turned off snapshots for /nix
<clever> Infinisil: so it always copies, and cant even share the blocks on-disk (enless i turn on dedup!)
<clever> Infinisil: in my case, / and /nix are seperate datasets in the same zfs pool, and the linux kernel isnt smart enough to understand that
<clever> cmcdragonkai: still simpler to test under nix-shell
<clever> cmcdragonkai: atomic doesnt really matter much in this case, it just makes mv as slow as cp
<clever> if you are using a sandbox, or the FS's dont match up, a move is just cp + rm
<clever> though if your not using a sandbox, and /tmp is on the same fs as /nix/store, a move might be atomic
<clever> Infinisil: for a normal build it wont matter, but it can make testing under nix-shell harder
<clever> copy is usually better then move
<clever> cmcdragonkai: or strace the program to see what paths it tries to open
<clever> dalaing: another simpler option might be to just try changing the nixpkgs channel hydra is using temporarily
<clever> dalaing: then it might have been a problem with the ghc that pandoc linked to
<clever> dalaing: did it fix the problem when it did so?
<clever> dalaing: yeah, hydra will either download it from the binary cache, or re-build it
<clever> yeah
<clever> simpson: have a look at the vid i just linked
<clever> Infinisil: cats will do that
<clever> normally, if you want something off, you just turn it off in the nixos config, and it ceases to exist
<clever> sauyon: the entire enable/disable section of systemd has been disabled, everything is just on all the time
<clever> sauyon: depends heavily on what the os has done to generate the iso
<clever> which was already usb bootable to begin with
<clever> sauyon: those modes break the nixos image
<clever> sauyon: unetbootin has special modes to extract the kernel and initrd and mess with things to make iso only images boot from usb
<clever> yeah
<clever> Infinisil: the nixos iso is a specially made hybrid image, that can boot on usb or an iso, just burn the iso directly to a usb
<clever> the nixos iso can just be dd'd directly to a usb stick
<clever> unetbootin breaks the nixos ISO
<clever> noobly: what about ps aux | grep wpa_suppicant
<clever> noobly: wpa_cli, then status, scan, and scan_results
<clever> noobly: journalctl -f -u wpa_supplicant
<clever> bbl
<clever> dalaing: rebooting is the simplest way to clear them all, something has come up now
<clever> dalaing: ah yeah, nix also checks for roots in /proc, for in-use files
<clever> Kaydee: and it will conflict with the other ghc's that have packages you want
<clever> Kaydee: the main ghc attribute lacks all packages on hackage, so its not of much use
<clever> Kaydee: check the man page for nix-collect-garbage, the --delete-older-than i think
<clever> Kaydee: no more spam, and if things break horribly and you hard-reset, all changes are undone
<clever> Kaydee: "nixos-rebuild test" will activate it without putting it into the bootloader
<clever> sauyon: pastebin
<clever> Kaydee: there is also stack2nix, which can generate a tree of haskell packages for every version in your stack file
<clever> Kaydee: i never experience cabal hell
<clever> Kaydee: i learned nix before stack&cabal, so i just dont use either
<clever> Kaydee: you have to create a new ghc, using ghcWithPackages
<clever> Kaydee: one reason for that, there is no way to tell that ghc what packages it can import
<clever> Kaydee: things like ghc shouldnt be installed, but loaded in a nix-shell
<clever> dalaing: you can also try manualy deleting those hydra roots, and then nix-store --delete the bad path
<clever> Kaydee: but nix-env can search current and root
<clever> Kaydee: poor UI, nix-channel --list only shows the current user
<clever> Kaydee: adding channels to users tends to make things harder to manage
<clever> Kaydee: that tells it to use the channel called nixos
<clever> Kaydee: what does that do?
<clever> Kaydee: and nix-env -iA nixos.ghc
<clever> Kaydee: what does "ls -lh ~/.nix-defexpr/" say?
<clever> Kaydee: they will use roots channels automatically
<clever> mpcsh: services.kbfs.mountPoint = "...";
<clever> dalaing: and are those jobsets set to keep 0 evals?
<clever> dalaing: did you start the hydra-update-gc-roots service?
<clever> yes
<clever> i think you need ip -6 route
<clever> so it wont block you right away
<clever> tilpner: oh, that block script only runs every 10mins
<clever> dhess: cabal2nix will detect if you generate docs, and set the right boolean to activate my changes
<clever> Infinisil: did you link the script on gist?
<clever> tilpner: the logs for named (the dns server)
<clever> i find slim still works for all of my systems
<clever> Infinisil: i was working on this a few months ago: https://github.com/taktoa/hsdm
<clever> its just enabled by default
<clever> yeah, that often :P
<clever> so they update more often
<clever> the -small channels wait for testing, but not full binary cache coverage
<clever> dhess: it appears to be in nixos-unstable-small
<clever> dhess: and with master, and running strip over the binary, it went down to 30mb
<clever> dhess: with master, it was 151mb
<clever> dhess: with nixos-unstable, the closure of a shake script is 1.6gig
<clever> dhess: the docs are now in a .doc output, and can potentially be GC'd out
<clever> dhess: a PR i recently got into master handles that fully
<clever> i run my browser in a terminal, just because it likes to randomly segfault, and i want to see that
<clever> which is the journal for the display-manager.service unit
<clever> and if you launch something from the xfce panel and such, stdout goes to the same stdout as the display manager
<clever> i think chrome is only logging to stdout
<clever> it could also be that they are just testing you, to see if you can join in a later attack
<clever> so no level of source based blocking will stop it, you are blocking based on the intended target, not the real source
<clever> and if your dns server was an open recursive resolver, you would join the DDoS, by sending a 4kb reply
<clever> it sounds like those chinese IP's your seeing, are actually the victims, with a spoofed source ip