2016-12-09

<clever> lewo: i usualy put the prestart on the unit that cares about the keys, generaly in the same module file
<clever> lewo: normaly, id use a prestart on the systemd unit, auto-generate random keys before starting up
<clever> Rotaerk: or put the customizations into the config argument, it supports overrides
<clever> Rotaerk: you can also aim that at your fork of nixpkgs, and include customizations
<clever> Rotaerk: and the pkgs instance is told to ignore ~/.nixpkgs/config.nix, to further enhance the reproducability
<clever> Rotaerk: pkgs2 can update, but all its used for is to download a fixedPoint version, which goes into pkgs
<clever> Rotaerk: pkgs2 = import <nixpkgs> {}; fixedPoint = pkgs2.fetchFromGitHub { owner = "nixos"; repo = "nixpkgs"; rev = "something"; sha512 = "something"; }; pkgs = import fixedPoint { config = {}; };
<clever> to sync it up
<clever> which also allowed sharing the state with other people
<clever> none that i know of, somebody that i talked to a few months ago kept the nixops state in git (with some gpg crypto over it)
<clever> ( and ) can force it, but antiquoting it inside a string also works
<clever> so you cant do the obvious ./. + "/" + project + ".nix"
<clever> so ./. + "/" == ./. (the trailing slash is dropped)
<clever> but after each append, nix will normalize the path
<clever> paths can have strings appended to them
<clever> Rotaerk: ./. + "/${project}.nix" i think
<clever> simpson: i have also previously ran a linuxfromscratch router before, and manualy turned my laptop into a router a few times with entirely manual iptables and ip commands
<clever> eacameron: i have written wireshark decoding plugins, heh
<clever> having done the same thing in my nixos router, it was simple to do again
<clever> and 10 forwards things to the container
<clever> 18&19 need to be unique within that new subnet
<clever> simpson: line 8 has to be the external ip of the host, on the LAN, and line 12 has to differ from the lan subnet
<clever> simpson: i just setup NAT when i was doing that, one sec
<clever> but it does dhcp and nat, so that makes it a router, right?
<clever> i call it a router, but its a dual-socket rack-mountable server, with 3gig of ecc ram, and 3 gigabit ethernet ports
<clever> ive got nixos and hydra on my router, and its simple to start/stop
<clever> heh
<clever> so i was half right, but didnt think about that possibility
<clever> ah
<clever> gchristensen: except, due to a util-linux->systemd->kexectools dependency, fixing kxectools is practicaly a mass-rebuild
<clever> gchristensen: almost done the tarball
<clever> that causes the entire application to be compiled for mac, then pushed to the linux server
<clever> i have also seen other nixops users spam with import <nixpkgs> {}; everywhere, because they didnt know how to use callPackage
<clever> probably so the stuff its uploading to the linux server doesnt wind up being mac
<clever> so if your on a mac, line 4 provides mac packages
<clever> line 4 loads nixpkgs again, this time without setting system
<clever> Rotaerk: its just a modified form of with import <nixpkgs> {};
<clever> Rotaerk: that forces nixpkgs to do a 64bit linux build, and overrides pkgs.nix to use nixUnstable
<clever> it goes into the same config as allowUnfree
<clever> c74d: allowUnfreePredicate is a function, that takes a derivation name and returns true or false
<clever> c74d: yeah, let me find it
<clever> kk, i can automaticaly tar it up to make things go faster
<clever> want to give it a try?
<clever> so we just need to copy over a build of the userland half from nixpkgs
<clever> yep, its enabled in the kernel
<clever> CONFIG_KEXEC=y
<clever> does /proc/config* exist?
<clever> does that rescue shell have kexec in the path?
<clever> just do the kexec, inside the rescue shell it already has!
<clever> oh, and another thought, the target server may not have an os yet
<clever> think about how much simpler it becomes
<clever> my kexec drops you into a rescue ramdisk, with full nixos
<clever> gchristensen: it sounds like that is forcing the server into a rescue OS, that lacks nix, and then does funny things
<clever> gchristensen: also, it sounds like this would be a great template for my kexec+nixops thing
<clever> gchristensen: it sounds like somebody automated the linode guide from the wiki?
<clever> mkdir -pv /mnt/etc/ssl/certs && cat /etc/ssl/certs/ca-certificates.crt > /mnt/etc/ssl/certs/ca-certificates.crt
<clever> then just copy the contents of the ssl file over
<clever> ah
<clever> which drops you into a shell, with all the mounts setup for you
<clever> i would have just used nixos-install --chroot
<clever> oh, thats strange
<clever> ah, nix-store shouldnt do that on its own, but if you know where the chroot is, just copy the contents of the file over
<clever> to find which process originaly made the err
<clever> then you can just grep "Curl error 77" logfiles*
<clever> strace -s 3000 -ff -o logfiles nix-store --realize /nix/store/q7d0217f5mf2pbw37y389x06y8ha5355-etc
<clever> strace will cut things off a lot
<clever> oh, and -s is often of use
<clever> -f will follow a fork, -ff will append the pid to the output name (resulting in logfiles.1234), which makes it much much easyer to follow
<clever> may need to tar them up, move to a working machine, untar, and then run gist -p logfiles.* on them
<clever> gchristensen: and then try to gist every file it made
<clever> gchristensen: strace -ff -o logfiles nix-store --realize /nix/store/q7d0217f5mf2pbw37y389x06y8ha5355-etc
<clever> can you pastebin your console?
<clever> does that file exist on the target?
<clever> 77 Problem with reading the SSL CA cert (path? access rights?).
<clever> which may need to be inserted into .bashrc or .bash_profile, since ssh may not set it for non-login shells
<clever> gchristensen: error 60 is ssl, export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
<clever> imperative solutions are evil!
<clever> if the schema in hydra-init is out of sync with the code, it will happen every time
<clever> stepcut: i'm guessing the db schema is out of sync with the code
<clever> the something else can be in-lined with -E
<clever> nix-build -E 'with import <nixpkgs>{}; callPackage ./default.nix {}'
<clever> with a slight tweak, that could run on bootup, and now its death&rebirth! :P
<clever> and if i run "justdoit", the hdd is nuked, and nixos is installed
<clever> with these files, any machine on my LAN that tries to netboot will land at a nixos prompt
<clever> insta-format!
<clever> only scary thing, is that you need to be real carefull where you aim the thing
<clever> gchristensen: with a few more changes, my kexec thing could nixos-ify any host in one go, and leave it without a configuration.nix file, then you can target it with the none env in nixops
<clever> gchristensen: the kexec idea also opens up routes that arent currently available, like partition table control
<clever> gchristensen: in theory, it will work on any hoster that obeys the MBR loader
<clever> gchristensen: in theory, you just need to ssh into the root account, inspect some state (ip addr mainly), build the kernel+initrd, scp, execute, and wait for ssh to come back, then it behaves like it booted from the install ISO
<clever> gchristensen: i have thought about how to integrate my kexec idea with nixops
<clever> ive done that, it deleted the active nixos generated, lol
<clever> only thing i can think of is that you did some naughty things in your early nixos days and manualy deleted things
<clever> not sure then, i would expect GC to do it
<clever> avn: what does this output: ls -lh /nix/store/003x4zwb9794iydhbclxl0h4cixq1vq0-cmake-2.8.12.2.drv.bz2
<clever> can you paste the path for one of the larger/older files?
<clever> avn: my router goes a lot of builds, and also GC'd regularly, that dir is 55mb
<clever> looks like you just want -K
<clever> if you stop it before it reaches the install phase, then it never made $out
<clever> nhooyr: can you gist all of the output of nix-build?
<clever> i am currently on git revision a24728f of nixpkgs
<clever> /nix/store/k984k6xncihdsiajy99wk82gn3zhhy7x-nixos-17.03pre95306.a24728f/nixos
<clever> eacameron: realpath $(nix-instantiate --find-file nixpkgs)
<clever> Leo`: purity, some builds behave differently depending on the mod-time of files
<clever> so it looks like i installed it on nov 1st at 8:46
<clever> and adb is only present in 295, 296, and 297
<clever> http://pastebin.com/c1eGkZJL in here, you can see the exact date+time each generation of my profile was made on
<clever> that will just show if it is installed, not when
<clever> c74d: try playing with ctrl+z and fg on the qemu process, that will halt everything its doing, and resume it
<clever> so your only seeing what the guest has done to its virt block dev
<clever> heh
<clever> correction, every single record was a read
<clever> [610566.130202] z_wr_int_5(1127): READ block 332197568 on sdd2 (16 sectors)
<clever> [610566.131166] z_null_iss(945): READ block 304 on sdc2 (8 sectors)
<clever> c74d: i also have zfs, and i can see a lot of write activity from its internal threads
<clever> and the next time you try a build, it will clean it up
<clever> i think it always keeps it, but it doesnt register it as valid
<clever> oh, the out folder
<clever> -K
<clever> nhooyr: <nixpkgs/pkgs/foo/bar/remove-tools-1.7.patch>
<clever> ah
<clever> gchristensen: do you know if hetzner uses dhcp or staticly configured IP's on each box?
<clever> so i could hear the cpu load changing in real time
<clever> this reminds me of an issue on one of my older laptops, the capacitors near the cpu would make an audible noise, that is synced to the cpu usage
<clever> and part of that is the logger saving dmesg, which creates more messages
<clever> normal stuff for a nixos machine
<clever> or just not set a timer to disable it
<clever> you can leave it on for longer

2016-12-08

<clever> are you sure its the hdd?, it could be the cpu fan
<clever> c74d: nothing at all?, you should see idle activity from other things
<clever> gchristensen: and 30mins later!
<clever> gchristensen: i was curious about how https://gist.github.com/cleverca22/48c4254508ecc10c213e97c2dd8f9502 does in a real-world test
<clever> ah
<clever> gchristensen: how do you get nixos on hetzner?
<clever> options = ''--max-freed "$((20 * 1024**3 - 1024 * $(df -P -k /nix/store | tail -n 1 | ${pkgs.gawk}/bin/awk '{ print $4 }')))"'';
<clever> so it always goes to 10gig free at midnight
<clever> i recently put that hydra thing into my desktop as well
<clever> lol
<clever> this is mainly to recover the data from the drive
<clever> gchristensen: its best to also give it a path to a log file that will survive a reboot, so you can resume the copy
<clever> gchristensen: it will skip over the bad sectors, and read as much data as it can, then return to the bad sectors when its done, and re-read them repeatedly
<clever> gchristensen: one thing of use, if you have the room, is ddrescue
<clever> [610567.403299] chromium(27280): dirtied inode 26243778 (?) on tmpfs
<clever> [610567.494423] skype(16707): READ block 175141152 on sdd2 (168 sectors)
<clever> and with this, you get a 5 second data sample
<clever> [root@amd-nixos:/proc/sys/vm]# echo 1 > block_dump ; sleep 5; echo 0 > block_dump
<clever> including which files
<clever> you can then grep that to track down any kidn of usage
<clever> c74d: if you write a 1 to /proc/sys/vm/block_dump with echo, the kernel will list EVERY filesystem/block access to dmesg
<clever> chattered: https://github.com/mcpkg/mcpkg-server/blob/master/root.nix is some maven stuff i had worked on
<clever> for nix-env, i think the command is nix-env -f /home/joko/playground/nixpkgs -iA screen
<clever> nix-build will just show the error
<clever> nix-build is better for testing, nix-env will silently ignore errors and skip incompatible packages
<clever> joko: nix-build /home/joko/playground/nixpkgs -A screen
<clever> joko: and nix-env doesnt use the search path
<clever> just git clone nixpkgs and -I nixpkgs=foo
<clever> because most use platforms.linux
<clever> if you add it to the lib/platforms.nix for linux, that should cover 90% of the packages
<clever> it needs to be in the .meta.platforms for a package to allow building
<clever> probably
<clever> which isnt compiled for that arch
<clever> you may need to write a new stdenv, since nix will want to use the gcc in a bootstrap.tar.gz
<clever> havent heard of those before
<clever> then it just uses the version you put in the string, exactly as its built
<clever> you can also replace derivations in systemPackages and similar, with raw strings pointing into the store
<clever> comment it out temporarily?
<clever> why is it unable to rebuild?
<clever> that only works if you review the git history your signing, somebody could have yeah
<clever> yeah, thats another issue
<clever> with commit signing, that commit wouldnt be signed, and it would set off more alarms
<clever> gchristensen: and then waited until he pushed it up
<clever> gchristensen: and they added an extra commit under his name, that introduced a privledge escalation exploit
<clever> gchristensen: one reason to still sign things even with github, ive heard of an incident where somebody got into the dev box of a linux maintainer
<clever> spacekitteh: then bash will show you every single command it runs, after variable substitution
<clever> spacekitteh: another massively helpfull thing, do "set -x" in the build hook, before the stuff your having trouble with
<clever> and since its a raw text file, it wont go thru the nix parser
<clever> you would basicaly define a package as makeSetupHook {} ./emake.sh; and then it will source that shell script if you add it to buildInputs
<clever> and then only things that add it to buildInputs will be rebuilt, avoiding the mass-rebuild
<clever> it could be possible to make an emake buildInput, that purely defines an emake function
<clever> and now you can copy/paste the contents of the main build hook, and modify it to target something else

2016-12-07

<clever> savanni: in vlc, i just went to capture device, punched in /dev/video0, and now i can see myself
<clever> savanni: try testing the camera in a simpler program, like vlc
<clever> savanni: sounds like maybe the browser lacks pulseaudio support?
<clever> savanni: and when hangouts is active, which mic is it set to on the recording tab
<clever> savanni: what do you see in pavucontrol ?
<clever> savanni: what happens if you try to use hangouts?
<clever> if somebody sends you a file, you get a link to skype.com
<clever> file transfers arent even p2p or built into the linux client anymore!!
<clever> and its a closed source mess that M$ has been steadily ruining
<clever> .... skype is the only thing i can think of right now
<clever> and i never got it working in the end
<clever> simpson: i tried updating it a couple months ago, they radicaly changed the dynamic libraries it used, sp the patchelf stuff had to be redone from scratch
<clever> d
<clever> then any attempt to use the /boot on root, will fail har
<clever> one way to prevent this, chmod 0 /boot when /boot isnt mounted
<clever> one person had 2 /boot partitions, and the wrong one got mounted
<clever> /boot wasnt updating, so it basicaly rolled back on every reboot
<clever> the last guy with this problem, hadnt GC'd yet, and couldnt figure out why some boot related options where not working
<clever> so it just stops booting
<clever> and after they nix-collect-garbage -d, the generation /boot wants to load no longer exists
<clever> related, i have seen people break the config and /boot stopped mounting
<clever> andrewrk: your welcome
<clever> you should see a major difference in ls -lShr /boot/EFI/nixos
<clever> and then nixos-rebuild switch again, to re-generate /boot
<clever> with root, it can also get rid of old system profiles
<clever> without root, it can only delete old versions of the nix-env profiles
<clever> so you still have 36 versions of system left laying around
<clever> looks like you havent ran nix-collect-garbage -d as root
<clever> next step, ls -ltrh /nix/var/nix/profiles/system*
<clever> but we can see you have room for 14 generations, and the bulk of it is the initrd
<clever> not yet, we dont know which one is the one you need to boot
<clever> andrewrk: and then ls -lShr /boot/EFI/nixos
<clever> andrewrk: to start with, lets confirm where its all going, can you pastebin the output of "du -h /boot | sort -h" ?
<clever> so your only option is to GC them entirely, and then run rebuild to trigger line 117
<clever> systemd-boot lacks the option to limit how many generations go into /boot
<clever> andrewrk: you should be able to look in configuration.nix to see which bootloader is enabled
<clever> i personaly use grub on all of my machines
<clever> if your on grub
<clever> andrewrk: try setting boot.loader.grub.configurationLimit to a lower number like 10, and doing a rebuild
<clever> simplest would be to have the map concat a fixed number of spaces
<clever> lula_: related: https://github.com/NixOS/nix/pull/580
<clever> lula_: "double quoted strings"?
<clever> lula_: lib.recursiveUpdate will do a deep merge, but still overwrite the leaf nodes it encounters
<clever> lula_: // will merge at the top most level, and overwrite dups
<clever> erlangDirty = pkgs.erlang.override { enableDirtySchedulers = true; };
<clever> enableDirtySchedulers is an argument to the file, not the derviation
<clever> so you want .override, not overrideDerivation
<clever> i see the problem
<clever> from github
<clever> msd: can you link the default.nix of erlang?
<clever> kk
<clever> 7 year old machine, heh
<clever> its compressing as fast as the poor old cpu can go, lol
<clever> marusich: in about 2mins i'll have another example for you, fully automated install
<clever> and the netboot is more for when you control the LAN and want to automate things to varying levels
<clever> marusich: like a server your renting from the cloud
<clever> marusich: the main use for the kexec stuff, is cases where you cant just plug in a usb or a cdrom
<clever> marusich: once this second test is done, i can gist it all
<clever> marusich: for netbooting, you dont need the kexec or tar, just a couple files on an http server and ipxe configured
<clever> i am currently writing another netboot example that would massively automate it
<clever> so you just unpack the tar and run it
<clever> but it could be further modified, to just be a tar.gz, with 3 files and relative paths
<clever> with the curl | bash script
<clever> it would need to be a distro like ubuntu, that has had nix jammed onto the side
<clever> the chainloading is something i did previously, before i had the kexec idea
<clever> the initrd was only about 300mb, and contained a full nixos install
<clever> yep
<clever> i linked the testcase for it by mistake
<clever> marusich: http://hydra.nixos.org/job/nixos/trunk-combined/nixos.netboot.x86_64-linux the job i meant to link earlier
<clever> but nix-build always takes a path to a nix expression file, and often -A to say which attribute to build
<clever> i tend to just insert it randomly into the command line
<clever> yep
<clever> so i was using a local copy of nixpkgs
<clever> but at the time, it was broken in nixos-unstable, and i was filing a PR to fix it
<clever> but most users will just let nix-channel handle that
<clever> in this case, i specified nixpkgs with -I, so it will resolve to /home/clever/apps/nixpkgs/nixpkgs/nixos
<clever> marusich: so its referencing this file, via the search path: https://github.com/NixOS/nixpkgs/blob/master/nixos/default.nix
<clever> <nixpkgs/nixos> will reference the 'nixos' directory in nixpkgs (and then the default.nix inside it)
<clever> <nixpkgs> will pull up nixpkgs in the search path, <nixos> will pull up the pkgs for the channel called nixos, and do the same thing
<clever> lets see...
<clever> but you could easily add a bash script for more automated installs
<clever> and all it adds is an ssh key, so i can remotely get in, then you just fdisk and nixos-install like normal
<clever> the configuration2.nix is reusing the kernel+initrd from the PXE stuff, for use with kexec
<clever> wait, thats the test of netboot
<clever> this is the main hydra building that netboot i linked previously
<clever> the hydra instance is over on the router
<clever> i had manualy ran nix-build on that server, since its the one ipxe uses
<clever> marusich: yep
<clever> and here is a shell script, that will kexec a nixos install image, no need to touch the bootloader at all
<clever> and thats exactly what kexec can take!
<clever> and at this point, i have a kernel+initrd, that just boots into a ramdisk with nixos-install
<clever> as if it had a cd in the tray
<clever> and now every machine that netboots lands in a nixos install image
<clever> and because i already have ipxe configured in my LAN, it was just a matter of adding chain http://c2d.localnet/netboot/netboot.ipxe to a default clause
<clever> marusich: this netboot image is very simple: https://github.com/NixOS/nixpkgs/blob/master/nixos/release.nix#L106-L127
<clever> marusich: i have also been experimenting with other ways of installing nixos lately
<clever> looks like xorriso handles all of the hybrid magic
<clever> but a proper MBR is 512 bytes
<clever> the bin file mentioned in line 357 is 432 bytes long
<clever> -r--r--r-- 4 root root 432 Jan 1 1970 result/share/syslinux/isohdpfx.bin
<clever> not sure on that answer
<clever> which leaves 32kb of nulls at the start, where other stuff can be snuck in
<clever> yeah, the iso filesystem starts at offset 0x8000
<clever> marusich: so its both a valid MBR partitioned disk, and a valid cdrom image
<clever> marusich: part of the trick, is that the iso filesystem starts at a non-zero offset, and they are jamming an MBR partition table in sector 0
<clever> marusich: i think it uses both isolinux and systemd-boot, depending on how it was booted
<clever> just need to make sure the 500 isnt in use by anything else