2017-06-17

<clever> jake_: just dup over the symlink at /dev/fd
<clever> jake__: just normal cd dev and cd .. should also work the same
<clever> jake__: oh, it probably needs bash, not sh
<clever> ben: this generates a tarball containing a full nixos container, and some bash scripts to enter it from another distro that lacks systemd
<clever> ben: ive also been working on something related lately: https://github.com/cleverca22/nix-tests/tree/master/container-generator
<clever> and you have a choice between different builds
<clever> normally, the absolute path to that is embeded into grub.cfg
<clever> ben: and to make /run exist, you must run the init script in /nix/var/nix/profiles/system/init
<clever> ben: boot.isContainer will turn off the kernel and a number of other things, might turn off too much though

2017-06-16

<clever> jake__: pushed
<clever> jake__: aha, the contents list works better then storeContents
<clever> the default is not clear
<clever> boomshroom: it was made optional on april 16th
<clever> boomshroom: is programs.command-not-found.enable set?
<clever> its supposed to be pre-installed on nixos
<clever> boomshroom: what source are you reading that says it should be there?
<clever> jake__: and thanks to a surprise ghc build, that tar took over 10 minutes to build, lol
<clever> i'll tweak the symlink args and see if i can change that
<clever> boot should probably work when copied, let me patch that
<clever> enter should be copied
<clever> 89 cp -v /nix/var/nix/profiles/system/enter /enter
<clever> are you using your for loop or my mknod list?
<clever> may need to strip a dev out of the paths
<clever> done the split
<clever> i'll do it on this end
<clever> yeah
<clever> when i missed that, it deleted some state vital to xorg, and i lost the ability to open any new windows
<clever> --one-file-system will protect you if a --bind mount is missed
<clever> jake__: about all i can think of is to make a bash script that does: kill $(cat foo/pid); chattr -i foo/var/empty; rm -rf --one-file-system foo/
<clever> and on gentoo, it spits my username out half a dozen times, and no other user
<clever> yeah, the users command doesnt work on my nixos machine either
<clever> which flag did you pass to nixos-rebuild?
<clever> isNormalUser handles home for you
<clever> no need to set home
<clever> pushed those
<clever> networking.firewall.enable = false;
<clever> the guest shouldnt be messing with the host firewall
<clever> jake__: i think nixos-install handles it by using mount --bind, rather then cp
<clever> as long as the nameservers dont change, you only have to do it once
<clever> the name should be nixos
<clever> need to add a channel with nix-channel --add
<clever> it will need a /etc/resolv.conf to make the dns work
<clever> what error?
<clever> done
<clever> ah yeah, then i need to delete both of the C's
<clever> pushed again
<clever> yeah, just added :$PATH
<clever> t
<clever> but i was testing on nixos, so the paths happened to exist on the hos
<clever> it copies the root from the target pid
<clever> yeah
<clever> nsenter -r handles the chroot for you
<clever> i wasnt thinking of that
<clever> nope
<clever> enter changes less, so i should be able to just cp it
<clever> oh right, absolute symlinks everywhere
<clever> currently, you need to cd into the container root, and run ./enter
<clever> jake__: try adding "set -x" to the start of the script, and then confirm which mknod is the issue
<clever> yeah
<clever> pushed
<clever> so the #! is wront
<clever> when i made boot1, i forgot, its not in the chroot yet
<clever> g
<clever> oh, i think i see the issue
<clever> jake__: is the current dir the root of the container?
<clever> jake__: which path in it is the issue?
<clever> mellowmaroon: just nix-shell -p jdk
<clever> jake__: pushed a fix, nix-build -A tarball
<clever> jake__: oops
<clever> :D
<clever> mmmmm, spam
<clever> nixos*
<clever> but nixps will override that, and pass in a reference to nixpkgs.config
<clever> by default, nixpkgs will load that config.nix file
<clever> yep
<clever> env/build/shell obey ~/.config/nixpkgs/config.nix
<clever> mellowmaroon: that will change it for nixos, but not nix-env/nix-build/nix-shell
<clever> jake_: then check to see what is still in that namespace after you kill systemd
<clever> and what mount namespace it has in /proc/<pid>/ns/mnt
<clever> jake_: next time, check pstree or ps -eH x, and confirm which bash is the parent of systemd
<clever> ps aux | grep boot
<clever> it will probably have boot in the arguments
<clever> i switched that round, so proc and dev will automaticaly vanish
<clever> jake_: check to see if the bash running the main boot script is still running
<clever> jake_: oh right, the bash at the top, above the systemd, will be part of the mount namespace
<clever> just run nix-shell -p oraclejdk8, and dont install either one
<clever> and this is why you should use nix-shell, and never install compilers
<clever> Judson: did you modify the unpackPhase?
<clever> jake_: can you gist the error, the contents of /proc/mounts, and the output of "ls -l /proc/*/{root,ns}" ?
<clever> jake_: that will list the root dir of every process, do any stand out?
<clever> jake_: try doing ls -l /proc/*/root
<clever> jake_: then its probably not mounted anymore
<clever> Judson: yeah, its relative to whatever directory you left as active
<clever> try umount with the absolute paths?
<clever> jake_: what about "cat /proc/mounts" ?
<clever> jake_: do you see them in "mount" output?
<clever> jake_: i believe the kernel should kill everything automatically
<clever> do you cd in postUnpack?
<clever> but it can probably be both
<clever> by default, i believe its relative
<clever> its just a var pointing to wherever it got unpacked
<clever> nix stores the root of the source in $sourceRoot
<clever> and update $sourceRoot and the current dir
<clever> yeah
<clever> Judson: by default, unpackPhase will basicaly just tar -xvf, and then go with whatever the root dir in the tar was
<clever> jake_: pushed more changes, this isolates the mounts more
<clever> Judson: have a look at how other go things are built, like ipfs
<clever> jake_: it does that a lot, i tend to just ignore it
<clever> but if bar isnt a directory (or doesnt exist), foo gets copied directly to bar
<clever> if bar is a directory, a copy of foo lands in bar/foo
<clever> for example, "cp foo bar", will behave differently depending on if bar exists or not, and if its a directory or not
<clever> and for unpredictable tools like cp and mv, it shows when they are acting up
<clever> for things like ln, it reminds you which way the symlink goes, so you can spot mistakes faster
<clever> i -v all of the commands in the cp/mv/ln/rm/mkdir family out of habbit
<clever> verbose
<clever> need to mkdir -pv $(dirname ./dev/$x) first i think
<clever> i also initialy had the same hostname on both, so it was nearly imposible to tell which os was giving the login prompt, lol
<clever> it does have a pure tty mode, but i ran into trouble with text-mode before, because the guest was stealing /dev/tty1
<clever> yeah, the run-host-vm needs xorg
<clever> ah
<clever> jake_: doing another rewrite of the boot scripts, lol
<clever> the guest still refuses to quit, lol
<clever> and so, it doesnt make nodes it shouldnt have
<clever> jake_: yeah, if /dev/ is a tmpfs with nodes present, systemd doesnt try to fix it
<clever> jake_: pushd will cd, and save the previous dir on a stack
<clever> for x in null zero full random urandom tty; do mknod ./dev/$x c $(stat --format="%t %T" /dev/$x) && chmod --reference=/dev/$x ./dev/$x; done
<clever> function dupdev { mknod ./dev/$2 c $(stat --format="%t %T" /dev/$1) && chmod --reference=/dev/$1 ./dev/$2; }
<clever> function f { mount --bind /$1 $1/ ; }
<clever> jake_: i have made similiar before for --bind stuff
<clever> jake_: and if your wondering how i test, nix-build -A test-guest && ./result/bin/run-host-vm, open a terminal, doit ; cd t; ./boot
<clever> while this compiles, i'll grab some food upstairs
<clever> jake_: pushed a copy of what i have so far, still testing it
<clever> by mounting a tmpfs to the guest dev/ folder, i can bypass the nodev flag
<clever> so you cant have setuid things persist
<clever> android has a similiar thing, every fs that is writable is also nosetuid
<clever> its now a new filesystem, without that option
<clever> 9 mount -t tmpfs tmpfs dev/
<clever> it ceases to be a special device, so they cant have fun with things
<clever> that prevents an attacker from mounting a usb stick with a copy of /dev/sda on it
<clever> yep
<clever> nodev Do not interpret character or block special devices on the file system.
<clever> it prints out the flags for every fs
<clever> just run "mount"
<clever> and what mount flags is the current fs mounted with?
<clever> what does ls -l say on both?
<clever> and if you try to read from the real null?
<clever> jake_: every single one is 666 on my machine
<clever> jake_: let me try with a minimal one
<clever> jake_: with an empty /dev, something (probably systemd-udev) auto-made everything
<clever> ah
<clever> but they almost never change
<clever> reading them with bash would be ... fun, lol
<clever> so with "mknod foo c 1 3", you can create a character device, that behaves exactly as /dev/null
<clever> the name means nothing to the kernel
<clever> the character+1+3 uniquely identifies it as dev null
<clever> jake_: this is a character device with the major number 1, and minor number 3
<clever> crw-rw-rw- 1 root root 1, 3 Jun 15 17:48 /dev/null
<clever> jake_: the major/minor are also pretty stable, so that could all be hard-coded
<clever> jake_: this may also stop it from being able to reach /dev/tty1
<clever> jake_: looks like it just does stat on each, confirms its a character device, then re-creates it with mknod
<clever> jake_: ah nice, i was going to try just leaving it empty and see what gets auto-created
<clever> sphalerite: db.sqlite does have timestamps for when a path got added
<clever> jake_: this is a minimal example i have used before: https://github.com/cleverca22/fusenar/blob/master/container.nix#L13
<clever> jake_: yeah
<clever> yep
<clever> jake_: the real container is being launched via systemd-nspawn, which needs systemd on the host
<clever> maybe if we dont bindmount /dev?
<clever> jake_: i do notice that /dev in a real container is rather bare
<clever> jake_: the guest ones
<clever> jake_: yeah, i'm not sure how the nixos containers normaly block that, we need to go thru every service and disable them
<clever> jake_: so now boot and enter are inside the nixos build
<clever> jake_: yeah
<clever> jake_: the configuration.nix has doubled in size, lol
<clever> i think jake just wanted nix, but cant make /nix, so he wanted a chroot, and you might as well have an os in the root
<clever> jake_: nix-user-chroot would give you not, but not nixos
<clever> jake_: oh right, and what sphalerite mentioned might have been of use a day ago, lol
<clever> so nix will happily delete them
<clever> they arent directly depended on by anything
<clever> and they probably belong in configuration.nix, so they are correctly updated
<clever> i'll need to move the boot scripts so they dont get GC'd
<clever> jake_: 1 { config, ... }:
<clever> also, a GC still ate the boot scripts, i need to move them
<clever> jake_: ah, line 1
<clever> jake_: i have noticed that nearly everything in nix uses absolute symlinks, and its even breaking a safety within stage-1-init.sh
<clever> catern: ah, then you will want either a bind or symlink
<clever> jake_: pushed the fix
<clever> yeah, i see the solution in /home/clever/apps/nixpkgs/nixos/modules/installer/cd-dvd/system-tarball.nix
<clever> LnL: ah, i'm using make-system-tarball right now
<clever> pikajude: i was surprised that it ignored the /proc roots, so its probably LnL's idea
<clever> jake_: so the nix is going to self-destruct on any operation
<clever> jake_: oh, maybe the db import isnt being done right
<clever> when i ran nix-collect-garbage in my test env, nix deleted every last file in the store
<clever> jake_: be carefull with gc until you confirm its working right
<clever> jake_: and i think nixos-rebuild will make the gc root, but i'm not sure
<clever> jake_: yeah, you will want to copy the original configuration.nix from my github
<clever> jake_: :D
<clever> jake_: you may also need to cd out and back in after the --bind
<clever> jake_: the host might not even have cgroups enabled?
<clever> jake_: the cgroup namespace is probably optional
<clever> jake_: so you need to mount --bind /foo /foo, to make it into a mount point of itself
<clever> jake_: oh right, forgot
<clever> jake_: the directory that will become the root, must already be a mount point
<clever> catern: it doesnt have to be a symlink either, if you just get somebody to chown an empty /nix/ to your user, the curl install works
<clever> you would have to unionfs the entire /
<clever> yeah
<clever> MichaelRaskin: but in a case like jake_, the / is read-only, so you cant even make an empty /nix to --bind over
<clever> jake_: and what channel are you building this against?
<clever> jake_: can you compare the arguments in the script with unshare --help?
<clever> cstrahan: i was having issues with the guest systemd conflicting with the host systemd, and made it manual for now
<clever> cstrahan: the exec bash needs to be changed to exec /init
<clever> with a few more tweaks, i could make it staticly compile mount/chroot/nsenter, and then it would be even more isolated
<clever> but the unshare is after the chroot,
<clever> oh, nsenter does need to be on the host
<clever> cstrahan: yep
<clever> the only real requirement of the host right now, is that it has /bin/sh, mount, and chroot
<clever> it could potentialy even work on android, for example
<clever> so you can launch a nixos container on any linux distro with a compatible kernel, even if the userland tools are missing, and if it lacks systemd on the host
<clever> cstrahan: this nix expression will generate a tarball containing a nixos container, and some unshare/nsenter scripts to boot it, and gain shells
<clever> which calls nsenter with the right args
<clever> the ./enter script
<clever> id need yet another bash script in the chain of scripts on scripts
<clever> and a few of the mounts are done before the mount namespace
<clever> only issue ive had after that, is stopping the container
<clever> yeah, that should give you shells and also the boot process
<clever> jake_: as i suspected, if you dont --fork, the bash can spawn the pid1, and $! will be the real pid of 1, outside the container
<clever> jake_: done
<clever> Sonarpulse: i cant use cross-compiled arm stuff for a native arm build
<clever> Sonarpulse: yeah, cross-compile has a lot of issues like that
<clever> gchristensen: ld alone needed over 3gig of memory, which was imposible on a 32bit processor
<clever> gchristensen: that reminds me of my troubles linking firefox on 32bit ages ago
<clever> jake_: i do have nsenter working fully now, let me commit that
<clever> and its getting late here, i need to get some sleep
<clever> bennofs: what does "ping google.com" say?
<clever> nope, systemd has real uid 0
<clever> hmmm, maybe user namespacing... let me check my real container
<clever> i have boot.isContainer set on the guest
<clever> and there is an option that should have prevented that
<clever> modules/services/ttys/agetty.nix: enable = mkDefault config.boot.isContainer;
<clever> the display manager only crashes when the guest systemd gets a ctrl+c
<clever> and tried to incept me, lol
<clever> it ate the terminal, and gave a guest login prompt
<clever> sphalerite: that explains why the terminal kept going wonky when i tested without xorg
<clever> sphalerite: one big problem i can see, the container is spawning agetty on the host tty1
<clever> it didnt even care about in-use processes
<clever> sphalerite: lol, the container has no gc roots, its able to GC everything!
<clever> yet the container is still alive
<clever> sphalerite: annndd, ctrl+c on strace crashes displaymanager somehow!
<clever> yep, there are 2 dhcpcd's running
<clever> i ran both of those under nsenter
<clever> wait nvm, lol
<clever> what if i strace the fake pid 1
<clever> sphalerite: its entirely escaping the jail and quering the host
<clever> sphalerite: weird, "systemctl status dhcpcd" shows the exact same pid and logs
<clever> and getsockopt so_peercred confirms that the remote end of that unix socket is pid 1 in the current namespace
<clever> (when i just run systemctl to see basic status)
<clever> and its inside the chroot, so it should be isolated
<clever> i can see it connecting to a unix socket at /run/systemd/private
<clever> i think its crossing
<clever> just need to see whats up with systemd
<clever> sphalerite: i do have nsenter and unshare mostly working now, i can spawn shells in the same namespace that unshare has made
<clever> so you can compare those
<clever> the symlinks will have the namespace id's
<clever> jake_: via /proc/<pid>/ns/ i believe
<clever> ah
<clever> jake_: i think nsenter can handle the chroot for you, but that relies on the host having nsenter
<clever> yeah
<clever> yeah, something like that
<clever> not sure then
<clever> but fork/clone() might return the non-namespaced pid
<clever> the first child it makes will become pid 1
<clever> which is now the only thing in that namespace, with 0 processes
<clever> then it will execve the command you gave to unshare
<clever> without --fork, unshare will change the process namespace for itself, creating a namespace with 0 processes in it
<clever> and can save it
<clever> if you dont use unshare --fork, then the process creating that initial child will probably know the real pid
<clever> so you would have to save the host side pid of what will become pid 1
<clever> yeah
<clever> which just runs "su root -l" inside the container via nsenter
<clever> exec($nsenter, "-t", $leader, "-m", "-u", "-i", "-n", "-p", "--", @args);
<clever> runInContainer("/nix/store/xd2finah2jp61sax9h3d9a1xirvfggxh-shadow-4.4-su/bin/su", "root", "-l");
<clever> the nixos-container script has this function
<clever> but containers dont have access to either
<clever> jake_: normally, systemd would launch a getty process for tty1 thru tty6, and launch the X server via the display-manager service
<clever> jake_: yeah, nsenter might be better
<clever> ah, maybe it could, just bash& before you exec systemd
<clever> once you do get a shell, nixos-rebuild may work, and it will become easier to manage
<clever> jake_: you either need to use nsenter to force a shell into the same namespace, or setup sshd with the normal flags in the configuration.nix the tar is made from
<clever> jake_: it has finished booting, and all services listed in the configuration.nix are running
<clever> jake_: i think that hang is normal, thats what its supposed to do
<clever> jake_: and after stage2 does some basic setup, it exec's systemd as pid 1
<clever> jake_: for a real boot, stage1 runs stage2 via switch_root, and stage2 is what i symlinked at /init in the tar
<clever> jake_: thats where i got stuck
<clever> jake_: the pid namespace can solve that