2017-08-24

<clever> sphalerite: they have to associate first, but it occurs before the WPA handshake
<clever> what joepie91 just said makes that even simpler
<clever> it is possible to exploit the firmware in the wifi chipset, just by tricking somebody into connecting to your access point
<clever> joepie91: also, broadpwn
<clever> dylanjust[m]: i believe you want to set testTarget = "--show-details=streaming"; in the derivation
<clever> so if you have a url in NIX_PATH, and run 2 nix's at once, they can collide with eachother and crash one
<clever> so even a cache hit involves "writing" the current value back to the symlink
<clever> and it tries to update the symlink to its current value, every time it checks the fetchurl cache
<clever> what if target+".foo" already exists?
<clever> you cant atomicly overwrite a symlink, so replaceSymlink creates a target+".foo" symlink, then uses rename to atomicly replace target
<clever> the replaceSymlink function was broken
<clever> that lock didnt help
<clever> i had to fix a bug with parallel fetchurl calls causing nix to crash
<clever> if both networks are on the same channel, you can use a special command to create a second wlan interface, and then control each independantly
<clever> also depends on the driver
<clever> simpson: i just leave /etc/wpa_supplicant.conf untouched by nix, and do wpa_passphrase name pw >> /etc/wpa_supplicant.conf
<clever> anelson-: for example, nix-shell '<nixpkgs>' -A haskell.packages.ghcjs.acme-kitchen-sink.env
<clever> anelson-: you must use the .env attribute on the derivation when using nix-shell
<clever> more so the caller wont loose the reference unexpectedly
<clever> it adds it to a queue, and the event loop will delete it when you return control back to the event loop
<clever> QT has a this->deleteLater() to prevent that kind of issue
<clever> yeah
<clever> and if the member function never uses "this", you can call functions on a null pointer
<clever> Ralith: for non-virtual functions, the compiler is dumb, and will just set "this" to whatever you dereferenced, even if its invalid
<clever> eval-inline.hh:inline void EvalState::forceList(Value & v)
<clever> eval-inline.hh:inline void EvalState::forceList(Value & v, const Pos & pos)
<clever> and you would have to mess with "this" every time you use it
<clever> the value of "this" would include the type
<clever> oh, yeah, this could pose a problem
<clever> *args[1]->listElems()[n]
<clever> and if its a thunk, it will run it, and apply the above check
<clever> this will throw an error if the Value in [1] isnt a list
<clever> state.forceList(*args[1], pos);
<clever> nearly all access to the Value objects goes thru a small api
<clever> that could work, but due to the lazy evaluation, it might always be too late to apply
<clever> brb
<clever> how would you use them in nixexpr?
<clever> Ralith: how will tagged pointers, that have been copied to 4 different places, handle the type of the object changing?
<clever> then if any of them are accessed, the type held inside the Value changes from Thunk to something else
<clever> it copies the Value* for f, into the slots a, b, c, and all 3 have the same address pointing to the same Value object
<clever> when i do let f = complex_function in { a = f; b = f; c = f; }
<clever> they cant do what nix requires
<clever> yeah
<clever> thats the problem i can see with trying to use them in nix
<clever> so that all references to the function* become references to the first byte of the char[20]
<clever> Ralith: and then i want to dynamicaly replace that function* with a full char[20], without changing the address
<clever> Ralith: but if i start out with a function* that is stored at a multiple of 16, and type embeded into it
<clever> Ralith: i believe a tagged pointer is basicaly taking a char* that was allocated to a multiple of 16, and then using the low-order bits to store some type info?
<clever> Ralith: and nix doesnt know what the size will become, or the back-references
<clever> Ralith: upon running, that mutates into another type, and all references must follow that change
<clever> Ralith: the problem, is that things start out refering to a Value of type thunk, which is a function pointer + argument pointer
<clever> because all allocations are rounded to the next 16 bytes for ex
<clever> Ralith: ooo, yeah
<clever> joepie91: every string in nix contains a list of context, what storepaths it depends on
<clever> Ralith: ah, like putting the type directly before the value?
<clever> thats what list1 and list2 are doing
<clever> joepie91: first, you have the Value object for the string, then you have a normal char* containing the storepath, then you have a char** containing the context
<clever> joepie91: for example, "${pkgs.hello}" is at least 3 heap objects, maybe more like 10 or 20
<clever> joepie91: strings are also fatter then you think
<clever> joepie91: so thats 5 heap allocations
<clever> so [ 1 2 3 ] is 4 Value objects on the heap, a list, and 3 int objects, and the list points to an array of 3 Value pointers
<clever> joepie91: the Value object then contains the type enum, and the int value
<clever> joepie91: the list always containers pointers to Value objects on the heap
<clever> joepie91: all lists greater then 2 in size are a pointer + size, but i believe nix expects a 1:1 mapping between Value instances, and the backing list
<clever> Infinisil: when you delete a given Value object, nix wants to delete the entire list it points to
<clever> Infinisil: and now that i think of it, nix has no way to know that the list is being depended on twice, as different sizes
<clever> Infinisil: no such builtin
<clever> is there a builtin to get the first N elements of a list?
<clever> joepie91: hmmm, let me see...
<clever> so map is just a matter of pre-allocating a list, and then setting out[x] = f in[x];
<clever> but also, the map function knows the output size ahead of time
<clever> yeah, recursively doing list ++ [ a ] will be a bit expensive, as it has to re-copy the pointers each time, and gc the old list
<clever> so the cost is just copying arround $total pointers
<clever> but because the Value objects are immutable, you can share the Value* between the source and destination
<clever> if you want to append 2 lists together, you need to copy the pointers from the 2 lists, into a 3rd new list
<clever> Ralith: yeah, its a contiguous array of Value*
<clever> it makes lists smaller then 3 slightly faster to read, because there is one less level of indirection in ram
<clever> listn, stores a pointer and size within the union
<clever> joepie91: so, as a speed thing, the list1 and list2 types directly store the values in the union, right after the type enum
<clever> joepie91: behind the scenes, nix uses a union type to hold the real value, and a lot of them are pairs of pointers (like a function* and argument*)
<clever> joepie91: nix has 3 types of list, list1, list2, and listn
<clever> joepie91: another random implementation detail
<clever> lol
<clever> joepie91: and if it runs into a black hole, it knows it has gone full circle, and run into its own tail, and thats infinite recursion
<clever> joepie91: also, while nix is evaluating a thunk, it temporarily changes the type to blackhole
<clever> joepie91: so it only has to eval the thunk once
<clever> joepie91: another detail, is that when nix evals a thunk, it dynamicaly changes the type from thunk to the return value
<clever> joepie91: one sec
<clever> joepie91: at parse time, it turns things into a massive chain of classes
<clever> joepie91: which saves nix having to do a strcmp against "name"
<clever> joepie91: so nix can just ask for the attribute at key State.sName
<clever> and certain key names like "name" are specially pre-allocated, and their id# is stored in a global variable
<clever> joepie91: behind the scenes, it looks the keyname (a string) up in a set, and then does int compares while iterating over the key=value pairs
<clever> joepie91: there is one neat optimization ive noticed in nix, all attribute sets use ints as keys
<clever> joepie91: another thought, if i can serialize the un-evaluated thunk, could i cache its result to disk?
<clever> joepie91: if you can merge identical values, you can shrink the heap, and if you can safely merge identical thunks you could maybe speed up cpu time
<clever> joepie91: one thing i was thinking about, if we can find back-references within the heap, could we dedup the heap?
<clever> it may also be that the nixos modules i'm using are overly complex
<clever> but if you just split them up into multiple networks, its fine
<clever> Ralith: within a single deployment, i dont think it can pass 50
<clever> it does
<clever> and it scales linearly
<clever> evaling a single node needs 2.1gig
<clever> joepie91: 10 nixos machines in the nixops, each with 10 nixos containers
<clever> and the GC in nix refuses to use more then 17gig
<clever> joepie91: i discovered that one of my nixops deployment files needs 33gig of ram to eval
<clever> joepie91: where you the one doing the entire syntax of nix?

2017-08-23

<clever> neonfuz1, joepie91: when you do nix-env (-i|-e), it will read ~/.nix-profile/manifest.nix, apply the changes in ram, then build a new profile based on the resulting manifest (which also contains the new manifest.nix)
<clever> |nix-shell)
<clever> joepie91: the home directory of the user running (nix-build|nix-env
<clever> dy2k: youll need to open an issue on nixpkgs
<clever> and nix doesnt allow setuid files
<clever> they put their own copy of ksu into the package, setuid'd
<clever> ahhh, i see the problem
<clever> dy2k: it looks like you just need to fill that in with a random value, and gitlab will use it to encrypt everything in the database
<clever> only the official binary made by mozilla should have that level of trust
<clever> __monty__: i think part of it, is so if nix makes a mistake while compiling it (bad deps or compile flags) and the binary is broken, it wont ruin mozilla's public image
<clever> that goes into configuration.nix
<clever> dy2k: did you try just doing services.gitlab.enable = true; ?
<clever> and because its right in nix-env, there is no way to ever reuse that set in anything
<clever> this is the function responsible for turning the directory into a set
<clever> but you could put other fun stuff in there, like a fetchurl that downloads the latest nixos-unstable from github
<clever> it was more of an experiement to confirm how the logic works, never actually used it
<clever> for nix-shell i just -I nixpkgs=whatever
<clever> yeah
<clever> ah
<clever> and if it finds a path like <name>/default.nix, it creates an attr like <name> = import <name> {};
<clever> basicaly, nix-env will recursively search the .nix-defexpr directory
<clever> so i can now nix-env -iA foo.hello to grab things from master
<clever> tilpner: this file creates a virtual channel called foo, pointing to my checkout of master
<clever> tilpner: i also recently read the source behind nix-env to see how def-expr works
<clever> import /home/clever/apps/nixpkgs
<clever> $ cat ~/.nix-defexpr/test/foo/default.nix
<clever> kini: and also, clang isnt really the fattest thing, it clocks in at about 300mb, but i have a 1gig xen path just sitting there in the store
<clever> tilpner: deleting it will break all channel usage
<clever> tilpner: .nix-defexpr is required to make root's channels get into your user
<clever> --add and --remove dont take effect until you --update
<clever> i'm guessing you forgot to nix-channel --update
<clever> the trailing / is important when using -l
<clever> ls -lh /nix/var/nix/profiles/per-user/till/channels-22-link/
<clever> what is inside that path?
<clever> and since there is nothing to loose, you can also just delete channels and channels-22-link
<clever> then the channels profile should be empty
<clever> does nix-channel --list show anything?
<clever> that part looks normal, what about the nix-env -q command and its output?
<clever> tilpner: can you gist the command you ran and its output?
<clever> 928M /nix/store/nfcbgv1pq0x4vk1gqx04nkjxp6k01614-nixos-17.09.git.ec9a233-x86_64-linux.iso
<clever> 1021M /nix/store/l7xxzq1sxh57100j548f8whwwkip2j96-xen-rootfs
<clever> the du from 10 minutes ago also finished
<clever> tilpner: instead, use nix-env -e to uninstall whatever is causing the reference, then let nix-collect-garbage -d clean up the old generation
<clever> tilpner: you risk loosing everything in the profile
<clever> tilpner: in general, you should avoid deleting symlinks from the profiles directory
<clever> kini: and ycmd links against libclang.so, so all 600mb of clang gets to come along for the ride
<clever> $ ldd /nix/store/g5ikya205nfy0h22a1lkkzs1lb7l9r1j-ycmd-2017-03-27/lib/ycmd/ycm_core.so | grep clang libclang.so.4 => /nix/store/6nzb1s93l7rzaa0s6cflnm4hf59fg2ba-clang-4.0.1/lib/libclang.so.4 (0x00007f37bb25b000)
<clever> patchelf --print-rpath agrees
<clever> my gut says this is rpath
<clever> $ORIGIN:/nix/store/bxi43dv1wfzs62l7z8c2c2vn8i7wq9mr-python-2.7.13/lib:/nix/store/6nzb1s93l7rzaa0s6cflnm4hf59fg2ba-clang-4.0.1/lib:/nix/store/aqk777xlihcys8z3i9f0gblrz4jj82g6-boost-1.62.0/lib:/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib:/nix/store/gzdx6nd99z0fg10amjivl51frf1mj4q4-gcc-5.4.0-lib/lib
<clever> $ strings /nix/store/g5ikya205nfy0h22a1lkkzs1lb7l9r1j-ycmd-2017-03-27/lib/ycmd/ycm_core.so | grep --color 6nzb1s93l7rzaa0s6cflnm4hf59fg2ba
<clever> something in that library references clang
<clever> Binary file /nix/store/g5ikya205nfy0h22a1lkkzs1lb7l9r1j-ycmd-2017-03-27/lib/ycmd/ycm_core.so matches
<clever> grep --color /nix/store/6nzb1s93l7rzaa0s6cflnm4hf59fg2ba-clang-4.0.1 /nix/store/g5ikya205nfy0h22a1lkkzs1lb7l9r1j-ycmd-2017-03-27 -r
<clever> now i grep for the clang path inside ycmd
<clever> and tracing the line up, i find /nix/store/g5ikya205nfy0h22a1lkkzs1lb7l9r1j-ycmd-2017-03-27
<clever> so i just / to search for clang
<clever> that shows the entire tree of deps starting at the root
<clever> oops, doubled up
<clever> so next, i run this command, on any one of the roots the previous command gave: nix-store -q --tree nix-store -q --tree /nix/var/nix/profiles/system-323-link
<clever> kini: i havent looked into why, but you complete me does that
<clever> no mater how much i GC, it will never go away, i must remove it from a config file and reboot
<clever> and its a dep of the currently running nixos image
<clever> its a dep of nixos itself
<clever> so, it is a dependency of something i nix-env -i'd
<clever> /run/current-system
<clever> /nix/var/nix/profiles/system-323-link
<clever> /nix/var/nix/profiles/per-user/clever/profile-314-link
<clever> $ nix-store -q --roots /nix/store/6nzb1s93l7rzaa0s6cflnm4hf59fg2ba-clang-4.0.1
<clever> because its slow, i'll take the previous clang as an example
<clever> that will tell you why its being kept around
<clever> then run "nix-store --query --roots" on the fat things
<clever> this will list every storepath, sorted by usage
<clever> [root@amd-nixos:~]# du --max=1 -hc /nix/store/ | sort -h
<clever> one sec
<clever> there are also more steps you can do
<clever> so you still have some undo's
<clever> there is also nix-collect-garbage --delete-older-than 30d
<clever> but the profiles are protected properly
<clever> it does use the daemon to perform the actual garbage deletion
<clever> )
<clever> with root, it can delete generations from everything (nixos, and nix-env
<clever> without root, -d can only delete nix-env generations
<clever> did you run that with or without root?
<clever> gist the output of: ls -ltrh /nix/var/nix/profiles/
<clever> next, is generations
<clever> just keep in mind, many things are counted twice, since they probably share glibc and such
<clever> both the root profile, and your users profile
<clever> also, check your ~/.nix-profile
<clever> without apparent-size, it counts that waste, and also any savings caused by compression
<clever> i think yours is bigger, because of cluster size, there is waste beause the FS has to round up
<clever> this is if i tell it to ignore compression
<clever> 599M /nix/store/6nzb1s93l7rzaa0s6cflnm4hf59fg2ba-clang-4.0.1
<clever> [root@amd-nixos:~]# du --apparent-size --max=1 -h $(nix-store -qR /run/current-system) | sort -h
<clever> ehhh, more like half
<clever> thats why my clang is 1/3rd the size of yours
<clever> (if enabled)
<clever> also, the above number includes zfs compression
<clever> that is the single biggest derivation in the closure of my nixos
<clever> in my case, the tab-completion plugin in vim depends on clang, lol
<clever> 272M /nix/store/6nzb1s93l7rzaa0s6cflnm4hf59fg2ba-clang-4.0.1
<clever> using that, you can find out why the current generation is fat
<clever> tilpner: this shows the closure of the current nixos build, sorted by size of the contents
<clever> [root@amd-nixos:~]# du --max=1 -h $(nix-store -qR /run/current-system) | sort -h
<clever> lol
<clever> tilpner: one min
<clever> boomshroom: this will override llvm to add ninja to its build inputs, then open a shell suitable for building llvm, with that ninja
<clever> boomshroom: but you could do this: nix-shell -E 'with import <nixpkgs> {}; llvm.overrideAttrs (drv: { buildInputs = drv.buildInputs ++ [ ninja ]; })'
<clever> boomshroom: nix-shell can only do one of those at once

2017-08-22

<clever> then check config.nix into git
<clever> just set a rule that you only ever nix-env -ieA nixos.mystuff
<clever> bendlas: that is basicaly what nix-env does, it manages a buildEnv in ~/.nix-profile/
<clever> so you can add it to the config.nix, and then nix-env -iA it, to skip updating everything
<clever> and nix-env -e firefox to imperatively remove something
<clever> the major difference from the old buildEnv, is that these remain as seperate entities within nix-env, and you can just nix-env -iA nixos.firefox to upgrade something imperatively
<clever> nix-env -iA nixos.mystuff
<clever> { packageOverrides = pkgs: { mystuff = { inherit (pkgs) firefox chromium; }; }; }
<clever> just make an attrset in config.nix, via a package override
<clever> i found a new trick recently, that does almost exactly that
<clever> Infinisil: yeah, thats what killed it
<clever> Dezgeg: because the ninja file doesnt define a cheap way to get the deps of foo.c, only a way to compile and get the deps
<clever> Dezgeg: and that "feature" has made incremental nix imposible
<clever> Dezgeg: but i have also heard that ninja has a "optimization" that does both of those in the same command
<clever> Dezgeg: one way is to run "gcc -M", then seperately run "gcc" normally to build,
<clever> Dezgeg: there are also 2 ways to gather that dependency info
<clever> Infinisil: nix-shell -p foo -I nixpkgs=.
<clever> like default.nix with import ./.
<clever> any possible infinite recursion?
<clever> ah
<clever> Infinisil: did you use nix-env -i ?
<clever> lejonet: yep
<clever> pstn: and privateNetwork also cuts off outgoing, you need nat to re-enable outgoing traffic
<clever> pstn: the privateNetwork part does all protocols
<clever> pstn: the forwarding part is only for incoming traffic, and only does tcp
<clever> lejonet: but i dont think the UEFI can handle vfat inside madm
<clever> pstn: that covers all protocols
<clever> lejonet: the old zfs guide on the wiki showed how to make an mdadm mirror for /boot
<clever> i also switched to nixos from gentoo
<clever> so it rolled back every time you rebooted
<clever> lejonet: and because /boot wasnt mounted, it couldnt update what the "latest" version was
<clever> lejonet: this is the rollback feature in nixos, allowing you to undo almost any change in the system
<clever> lejonet: you need to manualy mount the boot fs to /boot, run nixos-generate-config, and then nixos-rebuild switch
<clever> lejonet: exactly
<clever> lejonet: does "mount | grep boot" show it as mounted right now?
<clever> lejonet: and the contents of /etc/nixos/hardware-configuration.nix
<clever> lejonet: preferably, the -l of every disk
<clever> lejonet: can you gist the output of "fdisk -l /dev/sda ; blkid /dev/sd*" ?
<clever> lejonet: is it mounted on bootup?
<clever> lejonet: do you have a /boot partition?
<clever> lejonet: this sounds like a different problem entirely
<clever> pstn: if it has privateNetworking = true; then it has no internet access, until you configure it seperately
<clever> lejonet: checking the gist ...
<clever> lol
<clever> they just sent me an email, "how are you?"
<clever> and mozilla has ears everywhere
<clever> lejonet: can you make a gist containing the config you are using?
<clever> lejonet: it uses buildEnv, so it should create a directory of symlinks when it needs to merge things
<clever> dash: then it does both things at once
<clever> dash: you could create a 2nd module, that sets nixpkgs.overlays and imports the 1st module
<clever> lejonet: is it giving a warning at nixos-rebuild?
<clever> dash: you can just add custom modules to the imports list
<clever> but the kernel rejects any mach-o with a different dyld path
<clever> the mach-o files have a field containing /usr/bin/dyld, which acts like ld.so
<clever> ahh, then the kernel actively stops us from ever having such features :P
<clever> and every time it updates itself, it unpatches itself
<clever> except, it has to be patchelf'd to run on nixos
<clever> Ankhers: so if they had just kept the phone as packets longer, it could have routed around the hole on its own
<clever> Ankhers: and internet access was fully working, and ran over the same fiber
<clever> do they turn it back into analog somewhere upstream?, and the failure was beyond that?
<clever> Ankhers: and the strange part, is that my telephone goes over fiberoptic, the analog terminates in the basement, how am i hearing static on the line?
<clever> Ankhers: atlantic canada
<clever> the land-lines didnt even have a dial tone or busy tone, just static
<clever> somebody managed to knock out telephone and tv service, for 3+ provinces
<clever> something similar happened here about 2 weeks ago, on a bigger scale
<clever> brb
<clever> what have you tried so far?
<clever> lejonet: how are you trying to install it?
<clever> lejonet: what program are you running that does the conversion?
<clever> lejonet: everything else is in /nix/store/
<clever> lejonet: the only thing in /run is the symlink current-system
<clever> lejonet: what about rxvt_unicode ?
<clever> lejonet: so if you create a package containign share/terminfo/foo and add that to systemPackages, you will get a /run/current-system/sw/share/terminfo/foo