2017-10-23

<clever> sphalerite: also, when nixos-rebuild tries to fetch the nix that obeys your configuration.nix nixpkgs.config, it wont include qemu-vm.nix, and then it throws errors that those options are invalid
<clever> seequ: that module is only in scope if you use build-vm, so all the docs will omit them
<clever> seequ: yes
<clever> its now 6 hours into a 3rd attempt
<clever> sphalerite: and after the 2nd failure, something else snuck into the queue long enough to keep the logs alove, ENOSPC
<clever> sphalerite: when hydra restarted the job, it destroyed all logs, so i had no way of finding the cause of the failure
<clever> sphalerite: ive blown 9h21mins building gcc, twice yesterday
<clever> EFI would likely work differently though
<clever> tanonym: as long as the nixos grub can read the ubuntu / to read out kernels
<clever> tanonym: that could also work, just dont give ubuntu a /boot partition, and leave it using the /boot directory of /
<clever> tanonym: either seperate /boot's or disable the script that manages grub config
<clever> tanonym: you will want to make sure they dont overwrite the nixos grub config when they do kernel updates
<clever> sphalerite: one of those, i dont use darwin much
<clever> etu: yeah, i can see efi being better, because you dont have to configure one as a master, and chainload the others
<clever> ive also heard that the apple uefi supports HPFS+ for the efi system part
<clever> so i had to entirely disable EFI booting to get legacy grub to even start
<clever> etu: its almost like the boot priority is within the CSM module, and CSM only runs if EFI fails
<clever> etu: if EFI is on, it always boots first, and it ignored the boot priority configured in the bios
<clever> etu: one other issue i ran into with my main desktop
<clever> but oddly, ive helped a number of users in here sucessfully install with efi
<clever> legacy boot partition on gpt, on every machine
<clever> etu: personally, ive never once gotten uefi to work
<clever> etu: so if you wipe the disk first, you can never remove that bootloader from the efi vars
<clever> etu: ive also heard of some, that dont let you delete a given bootloader if the UUID it refers to is missing from the disk
<clever> etu: i think it also depends on the UEFI implementation, some dont give you a choice between the registered bootloaders
<clever> seequ: lol
<clever> seequ: line 10-38 shows how it gets generated
<clever> seequ: its directly in the store, and only the systemd service file has the path
<clever> sphalerite: you can also trigger this with <break>o over the serial port
<clever> o will just cut power, without even doing a sync
<clever> sphalerite: any of these special letters can be printed there, and will take immediate effect
<clever> [1062186.379382] sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(V) show-blocked-tasks(w) dump-ftrace-buffer(z)
<clever> [root@amd-nixos:~]# dmesg | tail
<clever> [root@amd-nixos:~]# echo h > /proc/sysrq-trigger
<clever> sphalerite: oh, now i remember how to power off and reboot from the shell
<clever> nice
<clever> yeah
<clever> tanonym: /exit
<clever> yeah, that sounds more likely to do something
<clever> sphalerite: i dont see sync in the kernel flags
<clever> it would have to be patched
<clever> yeah
<clever> sphalerite: there is also something in stage-2 that can record the boot progress to that file
<clever> exec > >(tee -i /run/log/stage-2-init.log) 2>&1
<clever> stage-2 is also what mounts the root rw
<clever> sw/bin/init is raw systemd, and omits a lot of the bring-up stage-2 does
<clever> sphalerite: this one
<clever> [root@amd-nixos:~]# less /nix/var/nix/profiles/system/init
<clever> sphalerite: try /system/bin/init, no sw
<clever> sphalerite: are you using the nixos stage2?
<clever> aha
<clever> Documentation/kdump/kdump.txt for further details.
<clever> a way to allow loading ring0 code at runtime, but then disable it for hardening
<clever> ah, kexec_load_disabled + modules_disabled
<clever> so it can inspect the ram nixos left behind
<clever> and boot the stock os as a crash kernel
<clever> sphalerite: for example, after you fail to boot whatever (as long as you can read the nixos FS), kexec over to the stock kernel & os
<clever> sphalerite: but having it in the nixos kernel may also be of use
<clever> sphalerite: oh, and is kexec available?
<clever> i can see it being useful for hardening, or to deal with broken bootloaders
<clever> sphalerite: and you can even tell the kernel to ignore params given by the bootloader
<clever> sphalerite: another potentially fun kernel option, you can embed kernel params into the kernel binary
<clever> in my case, i was using a packageOverride, so it was affecting the inputs of the existing linuxPackages sets
<clever> sphalerite: yeah
<clever> sphalerite: some, one min
<clever> sphalerite: nixos stage-2 will also try to remount it rw fairly early on
<clever> but given that it can find init, the root is mounting (possibly read-only)
<clever> sphalerite: there is also rootfstype, and rootflags you could play with
<clever> sphalerite: one sec
<clever> sphalerite: nice
<clever> olejorgenb[m]: https://nixos.org/nixos/options.html#docker.enable
<clever> Orbstheorem: and that e thing i think is only for grub
<clever> Orbstheorem: boot.kernelParams = [ "param1" ];
<clever> nix-repl> builtins.unsafeGetAttrPos "fetchurl" pkgs
<clever> { column = 3; file = "/nix/store/q71jxrnm2a49s8hzfirm4ssri1x0lqqg-nixos-18.03pre118328.6d86fcb86d/nixos/pkgs/top-level/all-packages.nix"; line = 181; }
<clever> hyper_ch: nixos doesnt support dkms
<clever> nix-store -qR /run/current-system | grep zfs
<clever> hyper_ch: of the package or on-disk?
<clever> hyper_ch: hello
<clever> sphalerite: does lsmod show anything when running the original OS?
<clever> sphalerite: ah, that could become an issue
<clever> sphalerite: are you booting the same kernel it normally runs with?
<clever> brb
<clever> yeah, a new /boot for nixos would be safer i believe
<clever> this code relies on grub, but it could probably be adapted to systemd-boot
<clever> fearlessKim[m]: it puts the entire nixos installer into /boot, at the cost of ~300mb
<clever> fearlessKim[m]: i also recently wrote this: https://github.com/cleverca22/nixos-configs/blob/master/rescue_boot.nix
<clever> fearlessKim[m]: 512 or more
<clever> fearlessKim[m]: the rebuild will probably delete others, now that it see's the GC
<clever> fearlessKim[m]: try deleting one or 2 of those files, then redo nixos-rebuild boot
<clever> srhb: it might copy the new kernels in before deleting old kernels from /boot
<clever> fearlessKim[m]: garbage collection can help, but you will want to make it bigger
<clever> fearlessKim[m]: 96mb is pretty tiny, thats bound to run into issues
<clever> ahh
<clever> why are you trying to that? i seems like a fairly difficult thing to setup
<clever> but things like xinit and startx, skip the login screen
<clever> and the DM then drops root for its login page, and also for the user after they have logged in
<clever> ghostyyy: systemd runs the display manager as root
<clever> ghostyyy: there is still the problem that xorg needs root to access the gpu, and nixos lacks the setuid binary
<clever> dang
<clever> fearlessKim[m]: there is also clangStdenv.mkDerivation
<clever> but i have been working on some experimental things, to build the entire nixops deployment under hydra
<clever> the simplest thing is to just build the large packages, load them up with callPackage and ensure you use the same nixpkgs and foo.nix in both hydra and nixops
<clever> hydra doesnt understand nixops
<clever> yeah
<clever> so now i have 32bit x86, 64bit x86, darwin, armv6, and armv7 builds of nix
<clever> dhess: with the right application of things like map, you can generate a variant of each job, for each platform
<clever> how you generate the set, and what platforms it contains, hydra doesnt care
<clever> and hydra will just build them
<clever> and hydra expects that nix expression to return a set of derivations
<clever> yep
<clever> so hydra is building the same thing, against 2 different channels
<clever> and over here, it refers to nixos-unstable-small
<clever> in the example-master jobset, it refers to nixpkgs master
<clever> the <nixpkgs>
<clever> so line 3 refers to the nixpkgs defined in the inputs
<clever> every input is in NIX_PATH, and fed as an argument to the default.nix
<clever> the 'nix expression' line refers to the entry point in a given input (listed below)
<clever> this is an example hydra project
<clever> ah, i havent heard anything about fully automatic updating
<clever> and when it does nix-copy-closure to push the changes out, it will obey the remote nix.conf
<clever> dhess: behind the scenes, nixops just uses nix-build to build things on the deployer, so it obeys nix.conf as normal
<clever> dhess: it can easily pull from a hydra cache
<clever> services.hydra.enable = true; does most of the setup
<clever> useSubstitutes = true;
<clever> hydra = {
<clever> services = {
<clever> yep
<clever> it can also do arm builds on raspberry pi's if you wanted
<clever> builder@192.168.2.126 armv6l-linux,armv7l-linux /etc/nixos/keys/distro 1 2 big-parallel
<clever> and you can set nix.buildMachines in configuration.nix to generate it
<clever> nixos and hydra just assume that file is at /etc/nix/machines
<clever> exactly the same file
<clever> an example config from /etc/nix/machines (nixos can generate this file)
<clever> builder@192.168.2.15 i686-linux,x86_64-linux /etc/nixos/keys/distro 3 4 big-parallel,kvm,nixos-test
<clever> yeah
<clever> and that the user it ssh's to can import unsigned storepaths (it either needs to be the single user, or be trusted in nix.conf)
<clever> all hydra requires is that "ssh user@darwin nix-store --version" can find nix-store, and that it can build things
<clever> correct
<clever> dhess: yes
<clever> try removing that line and see if it still works
<clever> uhhh, yeah, thats not right
<clever> thats just going to waste disk space
<clever> build-dir isnt supposed to be installed
<clever> ah, thats ths shrink-rpath hook
<clever> viaken: run "nix-store -l /nix/store/hash-foo-1.2.3" to pull up the logs
<clever> viaken: can you pastebin the entire error?
<clever> what derivation is doing this?
<clever> its only used at compile time
<clever> thats a .o file, it cant be patcheld'd
<clever> viaken: what does "file" say about that file?

2017-10-22

<clever> sphalerite: dang!
<clever> sphalerite: also, what is the build-cores of each in nix.conf?
<clever> nice
<clever> try setting it to null
<clever> yeah
<clever> sphalerite: the platform asserts stop you from reading super.gtk-sharp-2_0 in any way
<clever> kuznero: util-linux defaults to the .bin output
<clever> heading off to bed now
<clever> ${libuuid.out}/lib/
<clever> and util-linux is the one that has libuuid.so
<clever> kuznero: libuuid is just an alias to util-linux
<clever> libuuid.out 0 s /nix/store/aswq968ln90nrykgi7902lg80vmibygf-util-linux-2.29.2/lib/libuuid.so
<clever> kuznero: by adding ${libunwind}/lib to the rpath list
<clever> kuznero: you have to add libunwind to the rpath
<clever> kuznero: buildInputs dont show up in the rpath when you patchelf
<clever> let the spam ensue!
<clever> sphalerite_: and they are back!
<clever> you need to open a shell with "sudo -i" then "nix-channel --list"
<clever> which breaks "sudo nix-channel --list"
<clever> sudo on darwin sets $HOME wrong
<clever> as root
<clever> and it must start with the name of a channel from "nix-channel --list"
<clever> ldlework: you need -A
<clever> sphalerite_: one min
<clever> sphalerite_: what does /proc/cpuinfo say?
<clever> sphalerite_: it will dynamicaly adjust based on cpu load
<clever> sphalerite_: ah, it can go a almost twice as fast as an rpi
<clever> sphalerite_: how many mhz on the chromebook?
<clever> so you have to lock the entire x86 machine to one type
<clever> but the bigger issue is that the ELF header saying v6 or v7 is too complex for binfmt-misc
<clever> which qemu-wrap can set
<clever> i think there is a CLI arg you can pass to do that
<clever> and then things fail when moved to a real v6, that lacks v7 support
<clever> openssl (i think) will notice the v7 support when doing a v6 only build, and force v7 opcodes into the product
<clever> one minor problem ive run into, qemu emulates a v7 with v6 backwards compat
<clever> :D
<clever> i use either "ps -eH x" or pstree for that kind of output, but they show the entire system
<clever> thats mostly IO bound
<clever> so it will be even to double, depending on how hyperthreading handles qemu-user
<clever> ah
<clever> sphalerite_: also, how many cores in the chromebook vs laptop?
<clever> even if qemu is 50% the speed, you can still build at an effective 150%, by building across both machines with nix build slaves
<clever> and cluster that build!
<clever> sphalerite_: but you could now repeat the qemu-user stuff against all your x86 machines, and then add them all as build slaves
<clever> hmmm, chrome is still coredumping
<clever> Oct 22 16:26:20 amd-nixos systemd-coredump[22563]: Process 19452 (chromium) of user 1000 dumped core.
<clever> nice
<clever> and because its staticly linked, it can probably just be (import <nix-misc> {}).qemu-user
<clever> sphalerite_: nix.sandboxPaths
<clever> build-sandbox-paths = ${toString cfg.sandboxPaths} /bin/sh=${sh} $(echo $extraPaths)
<clever> sphalerite_: the closure of qemu-user has to be added to build-sandbox-paths
<clever> sphalerite_: oh, right, nix sandbox, the qemu isnt present
<clever> thats better
<clever> Battery 0: design capacity 7800 mAh, last full capacity 3542 mAh = 45%
<clever> iqubic: i have 3
<clever> its % is better, but its capacity is much worse, lol
<clever> wait no
<clever> hmmm, this battery looks a bit better
<clever> Battery 0: design capacity 2400 mAh, last full capacity 1543 mAh = 64%
<clever> iqubic: thanks for reminding me to check, this battery is toast
<clever> Battery 0: design capacity 5200 mAh, last full capacity 1741 mAh = 33%
<clever> ldlework: so that kind of mess cant happen
<clever> ldlework: nix knows when the overrides have changed, and will use a different storepath for the overridden and non-overridden version
<clever> ldlework: why do you want to force a rebuild?
<clever> sphalerite_: i think updating to chrome 62 fixed the problem
<clever> seequ: it was nearly 10 minutes of people ping-timeouting
<clever> its a bouncer
<clever> iqubic: matrix got an exception to allow it
<clever> iqubic: those are all running on a single machine
<clever> adamt: looks like it
<clever> matrix is at it again...
<clever> MichaelRaskin: which isnt present in the version 60 i'm currently running
<clever> MichaelRaskin: and when linking the source, i noticed there is what appears to be a fix: https://cs.chromium.org/chromium/src/content/common/sandbox_linux/sandbox_linux.cc?q=LimitAddressSpace&sq=package:chromium&dr=CSs&l=434
<clever> MichaelRaskin: so it can never even attempt to use enough ram to run
<clever> MichaelRaskin: in this case, i have 32gig of total ram, and 64gig of swap, but the process is setting a fairly low 2gig limit on itself
<clever> MichaelRaskin: and one of my extensions needs >2gig for a single tab
<clever> MichaelRaskin: the problem i ran into, is that chromium sets a data size limit of 2gig on every render process
<clever> sphalerite: cant think of any, see what happens if you just copy the pointers
<clever> kk
<clever> Tekmo_: it may cause problems down the road, when the drv's it references get deleted
<clever> ehgads, i thought chrome was nearly done building when it claimed 700/800!!
<clever> [488/28739] STAMP obj/components/base32/base32.inputdeps.stamp
<clever> sphalerite: ah, nice, feel free to open a PR
<clever> Tekmo_: .drv files are supposed to be storepaths that have deps, and the hash of the contents (in non-fixed mode) matches the storepath
<clever> sphalerite: ah
<clever> Tekmo_: that could be a problem
<clever> Tekmo_: ah, but does it have the same path after being added back?
<clever> sphalerite: so map $HOME/nix to /nix, on systems where you lack root
<clever> sphalerite: how hard would it be to patch it to fake /nix instead?
<clever> sphalerite: ive been thinking, the fhs userenv stuff is able to use mount namespaces without root, to fake /lib and /bin...
<clever> and --add/--add-fixed only allow things that have no deps
<clever> Tekmo_: ah, the tricky part that the CLI api wont allow, the .drv must depend on the input .drv's
<clever> Tekmo_: this is where the nix language tries to write it to the store: https://github.com/NixOS/nix/blob/master/src/libexpr/primops.cc#L743-L744
<clever> what about nix-store --add-fixed?
<clever> Tekmo_: it might be simpler to call builtins.derivation with the right args, and let nix generate it
<clever> it generally helps for name to contain the version, but you can also add a pname that lacks the version
<clever> jluttine: this is also a thing you can use
<clever> { name = "package-name"; version = "1.2.3"; }
<clever> nix-repl> builtins.parseDrvName "package-name-1.2.3"
<clever> jluttine: some languages use pname for that
<clever> and 62 does have the fix
<clever> so, version 60.0.3112.90 is bork, nixpkgs master is at version 62.0.3202.62
<clever> sphalerite: but the interesting part, is that the comment perfectly describes my problem, and that whole if statement is missing from the version i'm currently running
<clever> sphalerite: so when it gets shifted by 33 bits, it doesnt overflow
<clever> sphalerite: i believe that forces gcc to treat the literal as a 64bit, even though 1 could fit into a 32bit
<clever> error: cloning builder process: Operation not permitted
<clever> sphalerite: and interestingly, lines 429-436 dont exist on the copy i'm running in nix
<clever> its been on my todo list for 2 years...
<clever> i still need to file a PR for that
<clever> that adds a new nix.conf field, a list of platforms the cpu "supports"
<clever> yep
<clever> sphalerite: in nix-misc
<clever> sphalerite: the presense of /dev/kvm inside the nix sandbox is an impurity, that controled if the problem happened or not!
<clever> sphalerite: and because i have kvm enabled on my machine, i could never reproduce the problem
<clever> or rather, dies due to an illegal instruction
<clever> then cryptonite in haskell segfaults
<clever> sphalerite: so a qemu without kvm, lacks sse3 support by default
<clever> sphalerite: turns out, 64bit qemu emulates the very first 64bit cpu, one that lacks sse3 features that are now considered standard
<clever> sphalerite: ive even made an x86-64 qemu-user, on x86-64, and it was able to reproduce problems qemu-system-x86-64 had, without having to emulate the full kernel
<clever> sphalerite: and it sorta worked! (pulseaudio threw an exception though)
<clever> sphalerite: whats even more crazy, i just typed in the x86-64 arch one day, and built the same nix expression on a raspberry pi
<clever> but qemu-user just maps guest threads to host threads, and can use all the cores
<clever> sphalerite: also, qemu-system-arm doesnt have SMP support, so its limited to 1 core
<clever> sphalerite: yeah, because it wont need to emulate a kernel
<clever> sphalerite: aha, tracefile.25892:prlimit64(0, RLIMIT_DATA, {rlim_cur=2147483647, rlim_max=2147483647}, NULL) = 0
<clever> sphalerite: and it fails 100% of the time when trying to load a given page
<clever> sphalerite: found part of the issue, chromium is using ulimit to limit itself to 2gig of ram
<clever> iqubic: its linuxPackages at the 2nd one
<clever> iqubic: what line did you add to configuration.nix?
<clever> sphalerite: i'm currently fighting a very weird chromium bug, [pid 13045] mmap(0x9448d800000, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
<clever> JaakkoLuttinen[m: run "nix-store -q --tree" on the root .drv that nixos-rebuild prints
<clever> tobiasBora: nothing you install with nix-env can be found by the compiler
<clever> usbhid, and a driver for the usb controller i believe
<clever> ghostyyy: you need to include usb drivers in the initrd, boot.initrd.availableKernelModules
<clever> iqubic: boot.kernelPackages = pkgs.linuxPackages_4_12; for example