2017-07-03

<clever> systemd-boot, i havent even seen a limit on how many generations it puts in /boot, and i'm not sure if it even has a rollback menu?
<clever> grub supports both
<clever> ah
<clever> if you set boot.loader.grub.efiInstallAsRemovable it should work in a usb stick, and if you also set device, it should with via legacy, as long as you have both a bios boot, and a fat32 efi system
<clever> it should be possible to do both
<clever> no clue
<clever> this lets you cut out locales you dont want
<clever> locales can be trimmed
<clever> and youll get a list of every storepath in order of size
<clever> 599M/nix/store/ndsfpahlmzh8z0ix4d71jkk1npnjj7pp-clang-4.0.0
<clever> 7.2Gtotal
<clever> 1012M/nix/store/8v64zpyci6rgrzzbhc2bfa911yfmnfw5-ghc-8.0.2
<clever> $ du --apparent-size -hc --max=0 $(nix-store -qR /run/current-system) | sort -h
<clever> sphalerite: start by finding the top-level derivation (before it became a .img), and run something like this on it
<clever> it would need the script to read the user, and then nop itself
<clever> ive only ever used system units with the User= on them
<clever> yeah, that would help
<clever> but its better to generate the unit file from nix
<clever> ToxicFrog: if you fix the value of $NIX_REMOTE (copy it from the normal user end to the unit), nix-shell will work again
<clever> and script automaticaly goes into a bash file and winds up in ExecStart
<clever> ToxicFrog: path lets you insert anything into the $PATH of the service (like systemPackages, but not global)
<clever> https://nixos.org/nixos/options.html#systemd.services.<name>.script
<clever> https://nixos.org/nixos/options.html#systemd.services.<name>.path
<clever> and use script rather then execstart
<clever> ToxicFrog: just add nodejs to the path attribute
<clever> and in the false case, it becomes // {}
<clever> this one is more for doing { always = "on"; } // lib.optionalAttrs false { optional = "stuff"; }
<clever> joko: lib.optional can also do that, and has several variables
<clever> :D
<clever> yeah
<clever> so we need to make that line optional
<clever> edit the nixos-install script to "set -x" and then re-run it
<clever> oh right, thats a nixos-rebuild option
<clever> it might still want to build a nix binary, try with --fast?
<clever> disk image
<clever> ed
<clever> sphalerite: then it uses dd to assemble it into a partition
<clever> sphalerite: the rpi stuff does the bulk of the work without partitions, and uses debugfs to edit an ext3/4 image
<clever> and i had the fun of debugging a program that decided it was normal to stat null pointers
<clever> sphalerite: but the libredirect in nixpkgs segfaulted on that
<clever> sphalerite: oh yeah, and beware of weird edge cases, for example, stat() can take a null pointer string without failing too hard!!
<clever> which allows debugging the protocol
<clever> that then allows wireshark to decrypt things, as long as you log one end of the session
<clever> sphalerite: this one intercepts keypair generation in libsodium, and logs the privates to a log file!
<clever> sphalerite: a fairly simple preload lib i made recently: https://github.com/cleverca22/tox_decoder/blob/master/logkeys.c#L36
<clever> sphalerite: LD_PRELOAD?
<clever> yeah, nix master is now all c++
<clever> but yeah, i can see how just staying at equal priority would work
<clever> sounds like rsync is the problem area, it needs to not overwrite
<clever> which you may prefer, since you have the same content as cache.nixos.org anyways, but at a higher speed
<clever> then your cache will take over when there is a conflict
<clever> so you can always edit it and make it a higher priority
<clever> nix-push will only create that file if its missing
<clever> i cheated a bit with writeText, to make it easyer to read
<clever> noffle: and this is an example of using builtins.derivation directly: https://github.com/cleverca22/nix-tests/blob/master/bare-env.nix
<clever> and return a special object that is both an attrset, and a string of $out
<clever> noffle: it will then hash the serialized form of that map, to compute $out
<clever> noffle: basically, any call to builtins.derivation will flatten the attributes down to a map of string->string
<clever> Infinisil: and the builtins.derivation function itself is another nix file, hidden inside nix
<clever> sphalerite: ive memorized a large chunk of the source, lol
<clever> "/nix/store/1mapr9wzalrx2bj2xcinsvmfydfzxrzc-nixos-17.09pre109180.0d4431cfe9/nixos/pkgs/applications/misc/hello/default.nix:14"
<clever> noffle: [clever@laptop:~/cachecache]$ nix-instantiate --eval -E 'with import <nixpkgs>{}; hello.meta.position'
<clever> gchristensen: go ahead :)
<clever> with 3, it gives the same error as chrome
<clever> with 4, it wants me to continue the statement
<clever> i was using 1-------1 to test
<clever> and in chrome, it evals to this
<clever> Uncaught ReferenceError: Invalid left-hand side expression in postfix operation
<clever> it thinks there is more to the statement
<clever> doesnt work in nodejs
<clever> Infinisil: so it parses fine, but will probably blow up at runtime, it expects a ......
<clever> nh2: my idea with the cachecache project, is that you point it to many caches (you would have to nix-push to a normal http server), and then it would multiplex for you, and manage choosing which cache to get something from
<clever> sphalerite: also, <foo> will desugar into builtins.findFile at parse time
<clever> nh2: ahhh
<clever> nh2: ah, so you may want to adjust that, if you want to take over for bandwidth/latency reasons
<clever> sphalerite: there is also nix-instantiate --find-file
<clever> nh2: currently, it can proxy every narinfo request, and cache the result in a hashmap
<clever> nh2: i also started this lastnight: https://github.com/cleverca22/cachecache/
<clever> nh2: oh, i dont remember a priority, but that could explain it
<clever> gchristensen: modprobe was never meant to be setuid root, and fusermount didnt sanitize the env when launching it
<clever> gchristensen: and if you mess with ulimit some, fusermount can fail to even open /proc/filesystems
<clever> gchristensen: but modprobe can load its config via an env variable
<clever> gchristensen: fusermount is setuid root, and if it cant find fuse in /proc/filesystems, it will just system("modprobe fuse")
<clever> gchristensen: this is also how the fuse root exploit works
<clever> gchristensen: this will run false any time the user wants to load dccp
<clever> gchristensen: boot.extraModprobeConfig = "install dccp /run/current-system/sw/bin/false";
<clever> gchristensen: ah yeah, one min
<clever> gchristensen: pong
<clever> noffle: nix-repl or nix-instantiate
<clever> catern: amazon-image.nix has code to do that
<clever> that reminds me, i havent had any spam in ages
<clever> catern: this lets you embed authorized keys into the build, so they just work after you boot it
<clever> root.openssh.authorizedKeys.keys = with keys; [ keys.dual.distro clever.amd clever.laptopLuks ];
<clever> catern: in users.users.root, you can configure ssh keys
<clever> what error does it give, how does it not work?
<clever> schmits: refer to the derivations, like LD_LIBRARY_PATH=${hello}/lib
<clever> nixpkgs may already be doing things with that var
<clever> i learned nix before learning haskell
<clever> and it will be an immutable snapshot of the headers
<clever> but keep in mind, it will import a copy of the include dir when nix-shell is ran
<clever> schmits: add an attribute to the derivation, CPLUS_INCLUDE_PATH = "-I${./include}";
<clever> then it would need the .zip or .tar url from github
<clever> fetchtarball would unpack it after fetching
<clever> ldesgoui: need to call fetchurl on it first
<clever> catern: and the modules are able to set eachothers options and inter-connect
<clever> catern: the default.nix under nixos will merge all of the core modules and the <nixos-config> module together, to create a single config attrset
<clever> not build nova-image.nix directly
<clever> you must put nova-image into the nixos-config, and build nixpkgs/nixos/default.nix
<clever> the example you pasted already has that
<clever> use -A
<clever> which is exactly how configuration.nix works
<clever> and then aim nixos-config at that
<clever> you can also add any of these modules to the imports section of a custom one
<clever> nix-build '<nixpkgs/nixos>' -I nixos-config=./configuration.nix -A vm
<clever> build-vm can also be done with nix-build
<clever> joepie91: possibly of use: https://github.com/taktoa/nix-parse
<clever> :D
<clever> so you dont notice that the . isnt part of the path
<clever> and the errors say set.a.b
<clever> its just an unusual thing that not many will expect
<clever> yeah, { "a.b" = value; }
<clever> which will give fun error messages, because set.a.b doesnt exist
<clever> its an attribute names a.b
<clever> joepie91: there are also some edge cases, like set.a-b or set."a.b"
<clever> joepie91: i only have 3gig of ram on this laptop, so it takes a while to open things
<clever> joepie91: ah, just opened the gist
<clever> nh2: nope, nix expects the entire store as one stream
<clever> joepie91: what about this region?
<clever> 279 %left OR
<clever> joepie91: i think its in the expr_op definition in parser.y
<clever> joepie91: only source i can think of is the parser and lexer files
<clever> lookup the nixpkgs overlays int he docs
<clever> for normal nixpkgs, its in ~/.config/nixpkgs/overlays i think
<clever> but thats for configuration.nix
<clever> 2017-06-26 21:45:59< Infinisil> You can however clone the nixpkgs-mozilla repo, then link to it with nixpkgs.overlays = [ (import path/to/mozilla/overlay/rust-overlay.nix) ];
<clever> i think thats in the mozilla overlay
<clever> its usualy simpler to use the rust thats already packaged into nixpkgs
<clever> there is a --print-rpath
<clever> yeah
<clever> the gist i linked can do that
<clever> you must edit the rpath with patchelf, to point to zlib
<clever> installing libraries with nix-env will never make them work
<clever> nope
<clever> i have used nix-index, and it works well when you know the path
<clever> so the rustup binary will keep working
<clever> and as long as you keep that result symlink, nixos wont delete the libraries
<clever> that bash script can be ran on an ELF file to patch it
<clever> _habnabit: if you run nix-build on this file, you will get a result symlink pointing to a bash script
<clever> _habnabit: lol, zlib is in my example
<clever> nixos doesnt have a /lib64 dir
<clever> yeah, you need to --set-interpreter it to the right value
<clever> patchelf, oops
<clever> _habnabit: try with patcheld --print-interpreter
<clever> that will give you a shell that can compile things using libfoo and libbar
<clever> nix-shell -p gcc libfoo libbar
<clever> and installing the compiler/libs wont help, and sometimes makes it harder to see the problem
<clever> nearly everything compiling related breaks if not used in nix-shell
<clever> _habnabit: are you using nix-shell?
<clever> nh2: could be another force push, ive been having problems with those lately
<clever> hodapp: but if you delete line 17, you get a full xorg window to the VM
<clever> hodapp: https://gist.github.com/a32177e4b2b8b7c1e443b6d0fa102827 with this, i can fire up a VM that reproduces the issue, i had set line 17 to false because i was running it over ssh
<clever> then you can take advantage of the optimization ngingx has already had
<clever> nh2: it may also help to mix the 2, use a slower daemon with a full cache of everything, then a nginx over that, that can quickly serve recently used data
<clever> nh2: and will be configured to prefer a cache.nixos.org signature, when one exists
<clever> nh2: something ive been thinking of, is a custom daemon, that can query several binary caches, then cache them all in one spot, and serve them up
<clever> nh2: at least on cache.nixos.org, the files are immutable, but when you take different cache providers into account, they may each have different narinfo contents
<clever> and f isnt always going to give the same out for the same in
<clever> and the output is generated via output=f(input)
<clever> ah, yeah
<clever> catern: CA storage?
<clever> now i have to unpack it to use the content, and then i just delete the zip
<clever> its the same as serving .zip files in a torrent
<clever> that just encourages people to delete it from the ipfs store, and then nobody benefits from it
<clever> so you need 2 copies of everything you install (one in /nix/store, one in the ipfs store)
<clever> and related, the object must remain in the ipfs storage dir, in blocked form (not usable by nix) to share with others
<clever> catern: so if nobody asks for a given object, it just doesnt get mirrored, and it dies off
<clever> catern: another limitation of IPFS, is that you only store the things you asked the network for
<clever> so when it gets the nar request, it knows exactly what path to pack up on-demand
<clever> for example, hydra uses a narpath that is identical to the storepath, without even a .nar at the end
<clever> and the naming of the nar file depends on the server
<clever> and the narinfo files act as a map between things
<clever> yeah
<clever> and your "lookup" function is gcc
<clever> its more of a hash(source) = compiled_binary database
<clever> and the binary cache stuff isnt based on the hash of the content
<clever> for it*
<clever> so you need to know the hash of the content to ask for me
<clever> ipfs is mainly a hash(value) = value database
<clever> nh2: and it currently has a bug, where it doesnt save the signatures from cache.nixos.org, so anything you get from nixos, is re-signed by your key
<clever> nh2: and ensure the public half is in the nix.conf file
<clever> nh2: you need to generate a keypair (nix-store man page), then use --key-file on nix-push so it can sign things
<clever> hodapp: now i can just debug it locally
<clever> hodapp: i can confirm the same problem in a vm when i use the config from the gist
<clever> so are the narinfo files
<clever> nh2: what is the contents of nix.conf on that machine?
<clever> hodapp: testing...
<clever> nh2: dont see one, youll need to use one of the many extraConfig options to insert it at the right place
<clever> copy it somewhere, change display_errors, and then manualy run php-fpm with the right end, and the new php.ini path
<clever> hodapp: if you read the php-fpm command from the systemd unit, youll find the current php.ini path
<clever> hodapp: did you ever try it with display_errors = on in the php.ini?
<clever> i dont think nix can read narinfo+nar files from a local dir
<clever> gchristensen: as long as you serve the original .narinfo files from cache.nixos.org, the signatures stay intact, but you would need to host the unpacked torrent on http, and set that as a binary cache
<clever> also, when i check the A records for cache.nixos.org, i can see 8 ipv4 addresses behind the domain
<clever> and since you also have nginx acting as a mitm for cache.nixos.org, you can see it trying again on another cache
<clever> checking the http access logs can reveal what its doing
<clever> nix-daemon may ignore them entirely
<clever> and if you dont, the narinfo files just wont be signed
<clever> you have to pass it a key with --key-file
<clever> and nix-push will re-sign things, so the narinfo will be wrong
<clever> i think nginx has an index of what it has downloaded
<clever> but just throw out the bytes as it goes
<clever> another thought, is to write a custom client that will recursively browse the narinfo tree and fetch every nar via the cache
<clever> ah
<clever> almost sounds like cloudfront needs more work?, do we know anything about the internals behind how it works?
<clever> but i do have ~5 nixos machines at home, and a local cache to share between them would help
<clever> and maybe being in your region
<clever> i would expect cloudflare caching and ec2 stuff to not be that much better then cache.nixos.org, other then it being an isolated instance
<clever> ah
<clever> ah
<clever> i just run my own recursive cache
<clever> nh2: i also noticed you use 8.8.8.8, but i recently had issues with 8.8.8.8 claiming some servers i use dont exist
<clever> ah, i see a diff
<clever> nh2: i aslo need to test our your nginx nixos cache when i get home, ive got gigabit inside the lan so it should help with the latency of common things
<clever> nix-store -r /nix/store/foo
<clever> yeah
<clever> integrateddynamics: 3251ms
<clever> my log file says things like this
<clever> tconstruct: 722ms
<clever> they are split between the modern mc and the old 1.7 stuff
<clever> what version of MC?
<clever> your forge might be too old to make it
<clever> mine was in ~/.MCUpdater/instances
<clever> yeah, you can limit the search to there if your using ftb
<clever> joepie91: try this cmd, find $HOME -name loading-log.log
<clever> joepie91: do you see a loading-log.log file?
<clever> the fml logs may say which mod is to blame
<clever> i remember it being more like 10mins
<clever> or fixing the initialization code in every mod
<clever> and i think its heavily single threaded
<clever> most of my time is spent in the cpu anyways
<clever> boomshroom: it works even on the non-oracle jvm
<clever> boomshroom: you can also run jvisualvm from the oraclejdk to debug the java process
<clever> it can help to just put everything into one gist
<clever> joepie91: nice

2017-07-02

<clever> the server thinks your a different player
<clever> the only thing i can think of that would cause that, is if you where forcing the game to play offline
<clever> boomshroom: and if i forget to even play the game for a week, i can login to a surprise of more stuff, or game-breaking lag from an out of control farm, lol
<clever> (aka, having to leave the entire client open)
<clever> boomshroom: i also prefer a private server so chunkloaders dont cripple my system when i'm not actually playing
<clever> brb
<clever> boomshroom: it might have been generated somewhere, you can try asking Baughn next time he is on
<clever> :D
<clever> spinus: oh, that option looks handy
<clever> boomshroom: that does some things involving modpacks in curse
<clever> boomshroom: you may want to read over the nix files in https://github.com/Erisia/builder
<clever> spinus: bump the state version, and then fix whatever breaks
<clever> boomshroom: and nix-prefetch-url can accept a nix file and attribute path
<clever> boomshroom: i believe with the name attribute of fetchurl, you can just set the name
<clever> spinus: stateVersion controls a range of things that arent backwards compatible, like the postgresql format on-disk, what type of ssh host keys sshd uses, and a few others
<clever> magnetophon1: you may want to compare it to the new versions of the kernel stuff in nixpkgs
<clever> spinus: the whole point of that option, is so that nixos knows what state your nixos was originally installed in
<clever> spinus: it should never be bumped
<clever> and report a bug to musnix
<clever> magnetophon1: it might be simpler to just use an older version of nixos, try 17.03 or 16.09
<clever> magnetophon1: that sounds like the recent cross-compiler changes
<clever> then system is still intact
<clever> magnetophon1: and does system-75-link exist?
<clever> magnetophon1: what does this say about system?
<clever> ls -l /nix/var/nix/profiles/
<clever> there shouldnt be any link or dir called system-profiles
<clever> magnetophon1: you need to leave 2 links, the un-numbered one, and the one it points to
<clever> spinus: the one i linked should have most of them
<clever> spinus: this only puts the qemu drivers into the initrd, but doesnt force weird config onto things
<clever> spinus: the qemu-vm.nix in nixos tries to also share the store, but i have a more stripped down one
<clever> magnetophon1: yeah
<clever> spinus: you may need to include the kernel modules in the initrd, one sec
<clever> only the ones starting with system show up in grub, and even after you GC them, you need to rebuild the grub config
<clever> magnetophon1: there are 4 to 5 profiles most of the time, nixos, channels (root, your user), nix-env (root, your user)
<clever> 2017-07-02 18:42:34 < magnetophon1> I have a couple of system profiles that I'd like to get rid of completely. How can I do that?
<clever> spinus: ahh
<clever> spinus: is partitioned=true; and installBootloader=true set?
<clever> catern: not sur eabout that, havent been keeping up wtih how the config is changing behind the scenes
<clever> and then it can skip the site entirely
<clever> but if you properly delcare the hash, it can be cached to tarballs.nixos.org
<clever> yeah
<clever> boomshroom: but the website could change the tarball and hash at any point, and then it ceases to be reproducable
<clever> and thats usualy a bad design in general, because the version can change without warning