2017-02-16

<clever> copumpkin: you might as well be using php to configure that part of services, lol
<clever> copumpkin: reading the source, i think .serviceConfig and .unitConfig are entirely untyped, so the option framework is left to defaults
<clever> copumpkin, ikwildrpepper: yeah, i think nixos just takes a raw attrset, and will merge non-conflicting keys
<clever> mbrgm: the serviceconfig is weird, mkforce doesnt work on prestart
<clever> ikwildrpepper: nice
<clever> ikwildrpepper: 5mins after they asked for the pw, they noticed i had pre-configured the IP and ssh was already up
<clever> ikwildrpepper: its nixos, i dont think they would be able to figure it out :P
<clever> ikwildrpepper: i gave them a .vmdk file to replace my vm with, and they asked for the pw so they could apply that
<clever> eacameron: the datacenter IT guys normaly login manualy via the console, and update the config
<clever> eacameron: one of the datacenters ive been dealing with lately has similar but worse setup, you need static ip's configured, but it has no metadata service
<clever> or other methods to install
<clever> eacameron: it also means you cant mess up the config when using nixos-infect (if you turn off the code thats potentialy buggy)
<clever> eacameron: yeah, imports = [ ./systemd-digitalocean/module.nix ]; in configuration.nix
<clever> and if you clone the machine via the DO control panel, it automaticaly gets the right ip, no need for nixos to rebuild things
<clever> eacameron: i believe you just put the module.nix into imports, and you get an IP, thats it
<clever> eacameron: cant find it on a github search or chrome history, checking my irc logs now
<clever> eacameron: hmmm, where was it now....
<clever> eacameron: yeah
<clever> so you essentialy control GRUB over the internet
<clever> and if you do take control, you can pick which generation to kexec, or kexec into the rescue system
<clever> and if it hits a timeout, it will kexec nixos
<clever> so on bootup, it runs a custom bootloader, that listens on the network for control
<clever> eacameron, copumpkin: my second idea, is to make a linux bootloader, that runs as a stripped down kernel+initrd+gui app
<clever> copumpkin: so you get a very cheap repair ANYTHING option, as long as /boot is intact
<clever> you essentialy are embeding the install cd in /boot
<clever> this can also be of use for local systems
<clever> so when it breaks, you open a ticket and tell them to boot the rescue option, then you can ssh into that and repair it
<clever> #1 (for datacenters with more manual stuff), put the kernel/initrd from my kexec tool into /boot and grub.cfg
<clever> but this goes into 2 more ideas i had this week
<clever> in the case of DO, it doesnt sound like they have any tools to allow recovery
<clever> eacameron: i would expect that to sanely fail when it encounters luks and skip it
<clever> eacameron: ive heard that when you clone a droplet, DO will try to mount the rootfs, and overwrite /etc/network/interfaces and /root/.ssh/authorized_keys
<clever> eacameron: then you can ssh into that, and do whatever you want to the hdd
<clever> eacameron: but with the kexec tool i linked, you can boot nixos in ram, without any changes to the MBR
<clever> eacameron: yeah, with nixos-infect
<clever> eacameron: so you are forced to keep whatever partitions DO made for you
<clever> eacameron: and then builds nixos under ubuntu, and overwrites the MBR
<clever> eacameron: and this script appears to do the same thing i did to a gentoo system, it just jams nix onto the box with curl https://nixos.org/nix/install
<clever> eacameron: and somebody else in #nixos wrote a module that properly queries the DO metadata at boot time
<clever> eacameron: the network stuff on lines 81-89, ive heard its buggy
<clever> oh right, i think i read one of those scripts
<clever> i was thinking that DO worked the same as AWS, just clone a base-image managed by the nixos foundation
<clever> eacameron: that would essentialy give nixops support for every datacenter that boots a user-controled kernel
<clever> eacameron: so you point nixops to the ssh on an ubuntu server in any datacenter, and nixos just takes over by force
<clever> eacameron: i have considered mixing my kexec trick with nixops before
<clever> eacameron: this code maps over all filesystems, and extracts the .fsType from each
<clever> eacameron: and i believe DO lets you clone the machine after that, so you just have to configure nixops to use that as the base
<clever> eacameron: in theory, you could use kexec to get nixos running from ram, then format the hdd and make a zfs based image
<clever> eacameron: my kexec trick may work
<clever> for example, making a usb stick that lacks zfs, but will mount zfs drives manualy
<clever> eacameron: the only time you need to boot.supportedFilesystems = [ "zfs" ]; is when you want to force zfs support in and not define what to mount via zfs
<clever> eacameron: so you need to pre-make an image with that zfs config
<clever> eacameron: but nixops doesnt handle the partitioning/formating
<clever> eacameron: as long as its in the fileSystems attrset, it will just work
<clever> eacameron: yeah
<clever> and any filesystem with requiredForBoot also gets put in boot.initrd.supportedFilesystems
<clever> joko, eacameron: nixos will use add the .fsType of every entry in fileSystems to boot.supportedFilesystems
<clever> 20-23
<clever> eacameron: i have an ext4 /boot partition with no crypto, and the initrd contains the zfs drivers
<clever> eacameron: nope, that just puts a copy of memtest in /boot and adds it to grub.cfg
<clever> joko: havent tried the new boot.initrd.ssh stuff yet
<clever> eacameron: nixos, zfs, lvm, and luks
<clever> eacameron: and lvm
<clever> zero chance of data in the garbage being recovered
<clever> gchristensen: it also gives you peace of mind when the drives get retired
<clever> eacameron: line 20 of configuration.nix opens the luks, line 16 opens the zfs for root, and the rest is unrelated stuff
<clever> copumpkin: i suspect that crypto only works against physical intrusions into the datacenter, assuming the attacker doesnt also steal the box of keys
<clever> copumpkin: yeah, if they can access your aws console or the vm, they can just read the block thru the decryptor, and it does nothing
<clever> eacameron: pretty easy, let me dig up the config for my laptop
<clever> and there is a boot.initrd.ssh that recently got added, to let you ssh in and answer luks prompts
<clever> eacameron: unlocking luks on bootup is a seperate issue, but it should be solveable the same as a local machine
<clever> eacameron: the bulk of what nixops does is just nix-copy-closure, so as long as the fileSystems attrset is right and you can ssh the machine, it will keep working
<clever> qknight_: i think nixops uses something similar, so you can refer to the ip of other boxes in the deployment
<clever> qknight_: this is how pkgs winds up in the args for modules
<clever> qknight_: you can also add to the arguments that nixos passes to EVERY module
<clever> if you copy the module, then youll need to rename it to make it a new service
<clever> arianvp2: so an mkForce on the prestart entry i believe
<clever> arianvp2: modules cant easily be overriden, simplest thing is to mkForce the setting that the module was configuring
<clever> should be simple to test
<clever> and you need to force tcp via 127.0.0.1
<clever> depends, some services like mysql take "localhost" as an alias to mean "use the default unix socket"
<clever> to ensure it can at least start
<clever> i plan to make a nixos test that boots my services under qemu
<clever> oh, lol
<clever> so its stuck on self-signed example.com certs until you give it a boot
<clever> ive also found that nginx wont restart after acme gets certs
<clever> what error does 16.09 have?
<clever> it might be the impurity that is state!
<clever> at least in 16.09, the directory is root-only
<clever> drwxr-xr-x 3 root root 3 Jan 24 14:26 .well-known
<clever> [root@chipuppoker:~]# ls -lha /var/lib/acme/acme-challenge
<clever> ah
<clever> arianvp2: broke how?, last i ran it, it worked fine
<clever> fhoffmeyer: ^^^
<clever> makefu: that lets you selectively control what fixup does, without loosing all of it
<clever> makefu: there are also switches to turn those steps off

2017-02-15

<clever> Ralith: simplest thing i can think of is to run du on /nix/store, pick something fat that you dont like, and then nix-store --query --roots and optionaly nix-store --delete
<clever> Ralith: part of the fun in that, is that a large object can be shared between many roots
<clever> devoid: havent looked into how that part of nixpkgs works, so i cant do much more right now
<clever> bbl
<clever> so you need to rename your install, give it a public domain, and make it publicly accessible
<clever> main issue with trying to test modules like that though, is that github will often try to call back to you over http, and that always goes to the real nixos.org
<clever> so it can be updated if its ever lost/changed
<clever> and maybe for nixos, leave a comment with directions on where that secret has to be configured in github
<clever> contrapumpkin: and then i just dont include passwords.nix in git
<clever> contrapumpkin: i usualy handle secrets by doing let passwords = import ./passwords.nix ; in { .... foo = passwords.servicename; }
<clever> yeah, that should be using a mirror list
<clever> nix-repl> linuxPackages.kernel.src.urls
<clever> [ "mirror://kernel/linux/kernel/v4.x/linux-4.9.6.tar.xz" ]
<clever> bendlas1: it will rename the output to match its actual hash, and tell you what the hash actualy was
<clever> bendlas1: it basicaly just runs the normal fixed-output builder (including its mirror support), and treats the hash failure differently
<clever> bendlas1: there is a new addition, nix-prefetch-url '<nixpkgs>' -A hello.src
<clever> pareidolia: thats defined in "man systemd.exec"
<clever> pareidolia: so if you try to start container@foo, it will run the container@.service unit, with %i replaced by "foo"
<clever> pareidolia: containers and tty's are the main uses ive seen
<clever> pareidolia: i think it means you can spawn many instances of it
<clever> pareidolia: same place as it would land at on a normal system
<clever> # ls /etc/systemd/system
<clever> pareidolia: all of the nixos ones should be defined within https://github.com/NixOS/nixpkgs/tree/master/nixos/modules
<clever> pareidolia: not sure
<clever> but for nixos, you define that dependency within the nixos config
<clever> i think its so systemd will auto-install x when you install y's module
<clever> pareidolia: those other modules will have to be defined/loaded directly in the nixos config
<clever> pareidolia: nixos doesnt allow imperative changes to systemd units, so the install Also option means nothing
<clever> pareidolia: probably this one
<clever> services.udev.packages = [ packages.zfsUser ]; # to hook zvol naming, etc.
<clever> but another option is just throwing the whole file at systemd
<clever> pareidolia: ah, in this case its not being generated via systemd.services
<clever> lrwxrwxrwx 1 root root 99 Dec 31 1969 zfs-mount.service -> /nix/store/szjafmlahqnkj0wv1yfyncn995d789wz-zfs-user-0.7.0-rc3/etc/systemd/system/zfs-mount.service
<clever> zfs-mount.service:[Install]
<clever> nix-repl> options.systemd.services.description
<clever> "Definition of systemd service units."
<clever> and under the options tree, you can find the same info nixos-option uses
<clever> pikajude: you can inspect any nixos option you want, including the options other modules internaly setup
<clever> pikajude: nix-repl is also of use, http://pastebin.com/j8tnQGfa
<clever> oh yeah, forgot about that one
<clever> pikajude: an attacker can just point it to a custom config file, which changes where it finds fuse.ko
<clever> pikajude: but modprobe wasnt designed for setuid use, and it obeys env variables for config
<clever> pikajude: ntfs-3g is setuid root, so it can do fuse mountings, and if fuse isnt in the kernel, it just blindly runs "modprobe fuse"
<clever> pikajude: the ntfs-3g driver recently had a bug that allowed non-root users to load kernel modules
<clever> pikajude: config.system.path is the attribute path for the derivation that winds up at /run/current-system/sw
<clever> pikajude: yeah, ive broken units a few times because of that
<clever> pareidolia: every key=value pair put under serviceConfig will be put into the [Service] section of the unit, with zero typechecking
<clever> MoreTea: fail2ban
<clever> pikajude: it will be in both sw and the setuid wrappers
<clever> pikajude: havent been following that recent pr, but yeah it is moving a bit
<clever> pikajude: all setuid programs on nixos must live in this directory
<clever> sudo is /var/setuid-wrappers/sudo
<clever> that could also be done with iptables, feed it a fake one, that will build rules up in a temp file
<clever> then when all the installs are done, it runs a single ldconfig pass
<clever> MoreTea: it replaces ldconfig with a wrapper, that sets a flag and does nothing
<clever> MoreTea: that sounds like what apt-get does to fix ldconfig issues
<clever> MoreTea: plan b, just run the old/slow docker iptables command after the restore, the bulk of the firewall is in place
<clever> MoreTea: plan a, fix the docker stuff to have hooks that can insert things into the rules file before you restore
<clever> pareidolia: search for "systemd.services" in https://nixos.org/nixos/options.html
<clever> should have looked for one when i tried to write that pr a few months ago
<clever> ah nice
<clever> iptables-restore atomicly changes the entire list in one sweep
<clever> and also, during that time, the firewall rules are partialy built, and an attacker might be able to take advantage of it
<clever> MoreTea: but because of this, every iptables command you run takes more and more time
<clever> MoreTea: to avoid blocking network activity when changing rules, iptables uses an RCU list, so it copies the entire firewall list, modifies the copy, then atomicly swaps a pointer that the networking stack reads
<clever> MoreTea: the iptables command isnt meant to be ran in a script like that, it has some large cpu usage issues
<clever> MoreTea: one thing id like to see is for it to use iptables-restore
<clever> MoreTea: that opens it on every interface, in this case i only wanted it on the VPN
<clever> the nixos firewall doesnt allow you to configre things on a per-interface option, so i just manualy inserted an iptables command into the fw script
<clever> so just doing imports = [ ./snmpd.nix ]; starts it, and allows it thru the firewall
<clever> pareidolia: this file will now automaticaly open a hole in the firewall over the VPN
<clever> pareidolia: another advantage of making things like this into a module is this: https://gist.github.com/cleverca22/c2aa60d65ee6000dc6e4643a8a58a23c#file-gistfile1-nix-L24
<clever> pareidolia: this sshd example can optionaly listen on the ports listed in cfg.ports
<clever> pareidolia: oh yeah, odd
<clever> pareidolia: a random example of a socket + service
<clever> and i can reuse the same files between many machines
<clever> in my systems, i keep seperate files, like vim.nix and snmp.nix, then i can choose which features i want by adjusting imports
<clever> imports takes a list of modules, which have the exact same syntax as configuration.nix
<clever> you can either add it to the imports list, or merge it into the same file
<clever> which reminds me, i was about to add snmp to 2 of my servers
<clever> pareidolia: https://gist.github.com/cleverca22/c2aa60d65ee6000dc6e4643a8a58a23c is the nixos way to create systemd services
<clever> pareidolia: i think all this does is put it in some env variables and add some udev hooks
<clever> pareidolia: then i ran into a second issue, i accidentaly used nix master, so the nix in nixos is too old to handle db.sqlite!
<clever> pareidolia: main issue, is that gentoo files in /bin /usr and /etc usualy break nixos, so i had to rm -rf all of them
<clever> pareidolia: and boom, now nixos is booting on the gentoo rootfs!
<clever> pareidolia: this builds nixos on gentoo's /nix/store, then tells nixos to overwrite the MBR and re-install grub
<clever> pareidolia: after that, it was pretty much just nix-build '<nixpkgs/nixos>' -A config.system.build.toplevel && touch /etc/NIXOS && NIXOS_INSTALL_BOOTLOADER=1 ./result/bin/switch-to-configuration boot
<clever> which downloads it purely from the binary cache
<clever> so i opened hydra up in lynx (not even xorg worked), grabbed the storepath for a new nix build, and used "nix-store -r /nix/store/foo-nix-1.2.3"
<clever> i couldnt just nix-channel --update, because the nix was too old to handle the new nixpkgs
<clever> pareidolia: what would you do?
<clever> it was barely able to boot, but it had a year old copy of nix installed in /usr/local
<clever> and then i had to leave in the middle of that, with half of gentoo missing
<clever> i was in the middle of backing up the gentoo, by moving all its files from /mnt to an nfs server
<clever> but i had an even bigger adventure after that
<clever> nope
<clever> i was doing it purely over the network, because gentoo was still holding on to the hdd
<clever> ah
<clever> so grub boots from iscsi, thinking it a local bios hdd
<clever> ipxe hooks the legacy bios api for the hdd, and routes it over iscsi
<clever> i had used ipxe to do the bootstrapping
<clever> probably
<clever> i can also boot my laptop over iscsi
<clever> about a month ago i changed the laptop from gentoo to nixos + zfs + lvm + luks
<clever> its just so much faster
<clever> but i now install everyhting on it with nix-env, lol
<clever> i still have 1 remaining gentoo install
<clever> but now that nixo is taking over all of my systems, they all look identical!
<clever> the older systems lacked the fancy shell colors nixos has
<clever> prior to nixos, the biggest hint was the color of the shell
<clever> heh, neat idea
<clever> heh, ive shutdown -h the wrong box before because of that
<clever> and line 184 specialy depends on all of your interfaces
<clever> 16.09 has different dependencies
<clever> pareidolia: 16.09, or unstable?
<clever> pareidolia: which channel are you on?
<clever> pareidolia: oh, one min
<clever> and on my system, it doesnt depend on any network interfaces
<clever> pareidolia: strange, "systemctl list-dependencies" and "systemctl list-dependencies avahi-daemon" show entirely different lists on my end
<clever> so the time it starts at means nothing for the primary IF
<clever> mbrgm: it is starting, on the 2nd IF, which isnt plugged in
<clever> Jan 27 23:34:40 chipuppoker.com systemd[1]: Started DHCP Client.
<clever> Jan 27 23:34:40 chipuppoker.com dhcpcd[27568]: eno2: waiting for carrier
<clever> ah,without the --
<clever> systemctl: unrecognized option '--list-dependencies'
<clever> mbrgm: yeah, just a fixed delay sounds best
<clever> so you would want to confirm you can at least reach one IP
<clever> so even if the static ip is configured and the ethernet has a link, the switch may be blocking your traffic
<clever> mbrgm: then the bios re-does the link, causing that 20 second timer to restart, and it would never boot
<clever> mbrgm: i have had issues with SPAN on an old switch, the link was dead for the first 20 seconds it was online, and the network boot in my bios gave up too fast
<clever> mbrgm: about all i can think of is to confirm you can ping some known-good ip before it starts
<clever> mbrgm: ah
<clever> mbrgm: i also ran into a similar issue when i was fixing my vpn a few days ago, i just made the vpn retry every 10 seconds until it came online
<clever> mbrgm: check the output of journalctl -u dhcpcd -u keepalived and confirm if it waited for dhcpcd to fully start
<clever> mbrgm: the systemd depencies are a bit weird, it has different modes to start a then start b, or to wait for a to fully start up, then launch b
<clever> pareidolia: strange
<clever> mbrgm: i dont think the journal is logging targets, so i cant see when it thinks network.target is up, but that feels like it should do what you want
<clever> pareidolia: is try removing the wireless from configuration.nix maybe?
<clever> Guest77080: build-use-sandbox in https://nixos.org/nix/manual/
<clever> pareidolia: ah, it probably wants at least one configured network up before starting avahi
<clever> Guest77080: nix-build can do that, but not nix-shell, and nix-build can only do it when the sandboxing is enabled
<clever> pareidolia: what does the journal say about other units?
<clever> Profpatsch: the entire nix-build is sandboxes, so locale likely cant find things in the context of a build
<clever> mbrgm: network.target would probably be better in the static ip case, let me check one of my static servers
<clever> Profpatsch: the sandbox probably isnt letting it open this
<clever> open("/run/current-system/sw/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
<clever> pareidolia: what else does it say in journalctl?
<clever> mbrgm: just properly list it as dependency in your own service
<clever> mbrgm: http://pastebin.com/FD1qkwZM since systemd didnt declare it as started until after it got a lease, i'm guessing yes
<clever> but if i had a network based boot menu, i could re-run the installer and try again
<clever> k0001: main risk with the kexec hijack method, is that one mistake can lead to http://i.imgur.com/s23nqih.jpg and its bricked
<clever> netboot*
<clever> k0001: i have also loaded a modified form of those netbook images using kexec, which lets me hijack any linux server and convert it
<clever> k0001: it wouldnt take much to just leave a copy of that in /boot, and add it to grub.cfg or the new bootloader
<clever> k0001: using the netboot, you can boot any box from the ramdisk, and use nixos-install to repair/reinstall things
<clever> k0001: and nixos already has some other modules, to make a kernel+300mb initrd, containing a fully working nixos system
<clever> k0001: and then it will kexec your choice, or the default
<clever> k0001: i had an idea today, to make a custom bootloader, that runs as a stripped down kernel+initrd, and presents a bootloader menu over the network
<clever> k0001: in some situations, nixops will let you re-provision the machine, and depending on the datacenter features, you may be able to recover it, but thats not 100% covered
<clever> k0001: one problem i forsee with both nixos and nixops on remote servers, is that you sort of loose the ability to do rollbacks from grub
<clever> tm_: a: you want nix-env -iA, b: you should never install compilers/linkers when using nix, you want nix-shell
<clever> but about 5 minutes after that, they noticed i had pre-configured it and it was already up
<clever> then they wanted the name/pw so they could configure the static ip, i dont think they can handle nixos ....
<clever> so they could just replace the entire disk and its fixed
<clever> i knew it would take too long to walk them thru fixing it, so instead, i made a .vmdk image using qemu, and sent them a URL to it
<clever> yeah, i thought that as well
<clever> what is my poor linux server running on....
<clever> yeah...
<clever> yeah
<clever> pareidolia: http://i.imgur.com/s23nqih.jpg check the right edge very closely
<clever> and c:\ was visible just off the edge of the screen
<clever> and when i broke the VM ~2 days ago, i did get a screenshot
<clever> i have to open a ticket up and wait for a PHOTO of the monitor
<clever> pareidolia: the latest datacenter i have had to deal with doesnt even have the BMC plugged in, so no remote serial/console
<clever> pareidolia: grub 2.0 does have serial support, and i have used that in the past, but this relies on the datacenter routing the serial somewhere
<clever> this lets you kexec into a nixos install image, on any linux distro, and it runs entirely from ram
<clever> and another thing ive been working on, https://github.com/cleverca22/nix-tests/tree/master/kexec
<clever> Profpatsch: it would always load the images available on /boot, but the network just gives you control again
<clever> in the event it gets no reply over the network, it goes to the default
<clever> pareidolia: the basic idea, is that grub always loads a special kernel+initrd pair, that has a very thin boot menu in the initrd, that will check the network for a UDP packet, then kexec the real nixos kernel+initrd
<clever> pareidolia: havent learned scheme yet
<clever> pareidolia, garbas: ive also got plans for something similar/more insane, replacing grub with either haskell or rust, and allowing control over the network
<clever> lassulus: oops, wrong L name
<clever> LnL: both of those run with the entire rootfs in ram