worldofpeace_ changed the topic of #nixos-dev to: #nixos-dev NixOS Development (#nixos for questions) | NixOS stable: 20.03 ✨ https://discourse.nixos.org/t/nixos-20-03-release/6785 | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | https://r13y.com | 19.09 RMs: disasm, sphalerite; 20.03: worldofpeace, disasm | https://logs.nix.samueldr.com/nixos-dev
orivej has joined #nixos-dev
justanotheruser has quit [Ping timeout: 264 seconds]
orivej has quit [Ping timeout: 256 seconds]
justanotheruser has joined #nixos-dev
orivej has joined #nixos-dev
Scriptkiddi has quit [Quit: killed]
ajs124 has quit [Quit: killed]
das_j has quit [Quit: killed]
<{^_^}> firing: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
ajs124 has joined #nixos-dev
Scriptkiddi has joined #nixos-dev
das_j has joined #nixos-dev
lovesegfault_ has joined #nixos-dev
lovesegfault has quit [Ping timeout: 260 seconds]
justanotheruser has quit [Ping timeout: 260 seconds]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
justanotheruser has joined #nixos-dev
julm has quit [Remote host closed the connection]
julm has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.8]
lightbulbjim has joined #nixos-dev
cole-h has quit [Quit: Goodbye]
<{^_^}> firing: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
FRidh has joined #nixos-dev
alp has joined #nixos-dev
orivej has quit [Ping timeout: 260 seconds]
copumpkin has quit [Ping timeout: 272 seconds]
orivej has joined #nixos-dev
FRidh has quit [Ping timeout: 272 seconds]
lightbulbjim has quit [Quit: Connection closed for inactivity]
obadz has quit [Ping timeout: 246 seconds]
orivej has quit [Quit: No Ping reply in 180 seconds.]
obadz has joined #nixos-dev
orivej has joined #nixos-dev
obadz has quit [Ping timeout: 272 seconds]
obadz has joined #nixos-dev
alp has quit [Ping timeout: 244 seconds]
obadz has quit [Ping timeout: 256 seconds]
FRidh has joined #nixos-dev
obadz has joined #nixos-dev
orivej has quit [Ping timeout: 246 seconds]
orivej_ has joined #nixos-dev
justanotheruser has quit [Ping timeout: 272 seconds]
alp has joined #nixos-dev
Jackneill has quit [Ping timeout: 256 seconds]
<yorick> Mic92: seems like it uses the native python instead of the cross python to include/link
<Mic92> yorick: Yes. This is also how I understand our cross python setup
Jackneill has joined #nixos-dev
<manveru> any ideas what changed about `amazonImage` that it now requires KVM to build? i get `qemu-system-x86_64: CPU model 'host' requires KVM`...
<manveru> it used to work without KVM, albeit quite slow
<manveru> now i can't build AMIs on EC2 anymore :P
Jackneill has quit [Ping timeout: 264 seconds]
orivej has joined #nixos-dev
orivej_ has quit [Ping timeout: 272 seconds]
Jackneill has joined #nixos-dev
<Mic92> manveru: maybe related to: https://github.com/NixOS/nixpkgs/pull/83920
<{^_^}> #83920 (by offlinehacker, 5 weeks ago, open): runInLinuxVM, test-driver: disable vmx cpu feature for guest vm
<manveru> ah, that seems helpful, thanks
<Mic92> adisbladis: do you think it would be feasible to run some of our nixos tests with podman?
<JJJollyjim> I would love to see faster nixos test startups
<JJJollyjim> Idk how much room there is to tweak the guest kernel and stuff to make that happen in the current setup though
<gchristensen> how much do we value testing that the kernel actually works?
<JJJollyjim> I wonder if something like the Firecracker hypervisor might provide better performance while still testing the kernel
<gchristensen> yeah, good question
<Mic92> JJJollyjim: I think the biggest performance drop right now might be from the use of 9p since it's is implemented as a userspace filesystem.
<JJJollyjim> Ooo did someone say virtiofs? :3
<Mic92> JJJollyjim: cool I have not heard of that yet. Is host-native?
<Mic92> *Is this
<{^_^}> #85568 (by volth, 3 weeks ago, open): runInLinuxVM, test-driver: investigate virtiofs as 9p replacement
<JJJollyjim> Might take up the issue if nobody else is
<JJJollyjim> Host native?
<Mic92> JJJollyjim: Yes from the description the host server part is running in the kernel.
<JJJollyjim> Oh, I thought that was only the guest part
<aszlig> JJJollyjim: oh yeah, would be very good to get rid of 9p and our patches... less stuff to maintain
<{^_^}> firing: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
<Mic92> JJJollyjim: I re-read, this is indead the case. It's performance comes from zero-copy
<JJJollyjim> it's hopefully faster without dax as well, because it at least skips going through the network layer
<Mic92> JJJollyjim: is 9p actually using the network stack?
<JJJollyjim> oh i guess not with virtio
<Mic92> I thought it behaves like a unix socket
<Mic92> I don't see any reason not to enable dax.
<aszlig> Mic92: nope, it doesn't it's actually quite similar than virtiofs as far as i can see
<aszlig> similar in the way it's implemented, virtiofs uses the FUSE protocol, while 9p is using similar callbacks but it's not FUSE =)
<Mic92> aszlig: so is this some sort of shared memory queue between qemu and guest kernel?
<Mic92> For 9p I mean
<gchristensen> it'd be good to get a timely review of this one done: https://github.com/NixOS/nixpkgs/pull/87639
<{^_^}> #87639 (by mitchellh, 10 hours ago, open): vimUtils: customize accepts alternate {g}vimExecutable value
<aszlig> Mic92: unfortunately not :-/
<aszlig> it's more like a proxy between VFS operations and a virtual 9p host
<aszlig> which of course introduces a bit of overhead
<Mic92> aszlig: Yes. but the communication between qemu the the virtual 9p host should be done through shared memory, no?
<Mic92> I can see why they want FUSE. The fuse protocol is inode based and less racy than 9p.
<aszlig> Mic92: i *think* it doesn't, but let me check
__monty__ has joined #nixos-dev
<aszlig> so essentially, it's copies all over the place
<JJJollyjim> hmm
<JJJollyjim> virtiofsd is trying to create /var/run/virtiofsd on startup
<JJJollyjim> with no obvious option to change that
<JJJollyjim> which is something we can work around as long as we're confident the more complex parts will work as non-root?
<aszlig> JJJollyjim: mhm, as said, we need to patch it as well, not only because of this, but also because of the sandbox init
<aszlig> seccomp BPF is okay, but capability setup, pivot_root and namespaces are tricky
<JJJollyjim> just because the build is non-root?
<JJJollyjim> or complex nixy reasons?
<JJJollyjim> :P
<aszlig> JJJollyjim: because we're using a user namespace in nix builds
<Mic92> JJJollyjim: it is quite nice not having to run qemu as root in the first place.
<aszlig> so we already have a mapping from hostuid to nobody
<aszlig> s/host/parent namespace /
<aszlig> so if we want to make another user namespace to uid 0, this is not going to work
<aszlig> and it shouldn't since it would allow privilege escalation
<JJJollyjim> it looks like it doesn't make a namespace
<JJJollyjim> *a user namespace
<JJJollyjim> it uses CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET
<JJJollyjim> so process/mount/net namespaces
<JJJollyjim> i.e. another thing that's not gonna work as non-root :)
<aszlig> JJJollyjim: look into build.cc
<aszlig> JJJollyjim: ah, wait, you mean virtiofsd?
<Mic92> Yeah. I am surprised it re-invents systemd hardening options here
<JJJollyjim> yeah aszlig
<aszlig> JJJollyjim: ah, yeah, then you're right, but you need to be "root" (even namespaces) for that
<Mic92> JJJollyjim: let me have a look at the fuse part. This might also require root
<JJJollyjim> virtiofsd man page plainly states "This program must be run as the root user"
<JJJollyjim> i wonder what the limitations are
<JJJollyjim> Mic92: ty
<Mic92> JJJollyjim: when I was implementing cntr I also needed root
<aszlig> Mic92: the FUSE part shouldn't require root, since it only implements the FUSE protocol, not an actual FUSE fs
<Mic92> aszlig: no not because of the FUSE part
<Mic92> aszlig: I think they might want DAC_OVERRIDE in virtiofsd
<aszlig> Mic92: they do
<aszlig> see setup_capabilities()
<aszlig> Mic92: however, i don't see why it couldn't be emulated
<JJJollyjim> can the dax stuff work as non-root?
<Mic92> aszlig: We don't need DAC_OVERRIDE for our use case
* JJJollyjim uploaded an image: 2020-05-12-233339_765x421_scrot.png (65KB) < https://matrix.org/_matrix/media/r0/download/johnguant.com/qgAVstnDkCydsABgKmOXRPGW >
<JJJollyjim> ^virtio-fs without dax is marginally faster than virtio-9p, but it's not massive
<Mic92> Yeah otherwise 9p and virtio-fs are not that different.
<JJJollyjim> okay, i guess i'll leave this for now while people smarter than me find out if it's possible :)
<Mic92> JJJollyjim: I don't see why dax should not work. It's implemented in guest kernel.
<JJJollyjim> how does the virtiofsd provide access to the host kernel's page cache though?
<JJJollyjim> right it looks like the qemu process mmaps the file
<JJJollyjim> i think that makes sense
<Mic92> JJJollyjim: yes. one can achieve the same by using DIRECT_IO when opening file
<Mic92> *files
<JJJollyjim> doesn't direct_io entirely skip the page cache?
<JJJollyjim> ah!
<Mic92> JJJollyjim: right, it basically inverts sharing, the kernel space borrows pages for writing from userspace.
<JJJollyjim> oddly i can't find the original post in the mailing list archive
<JJJollyjim> if i understand that patch won't work
<JJJollyjim> but it verifies that someone has things working in non-root mode :)
alp has quit [Ping timeout: 260 seconds]
<JJJollyjim> oh except they're root inside the userns
<JJJollyjim> so maybe it still wouldn't work for us
<JJJollyjim> But I guess we think it will
<Mic92> JJJollyjim: So I read the fuse code a bit. They actually use splice() from libfuse to move page cache pages to userpace.
<JJJollyjim> I'll poke at getting it running tomorrow
<Mic92> So basically what need is skipping this sandbox buisness and forcing uid 0.
<Mic92> aszlig: Would it not make sense if we fix the uid problem in nix itself? We could map uid 0 properly in the sandbox.
<Mic92> Or does this not work if nix runs as unprivileged sandbox?
alp has joined #nixos-dev
<aszlig> Mic92: this unfortunately doesn't work, since we'd need to run the builder sandbox as root
<aszlig> Mic92: we can't simply map files to other users which we don't own
<aszlig> s/files/file uids/
<aszlig> or at least, run it as root, map the uids and then drop privileges, than could work in theory
<aszlig> it's been a while since i've dug into that, but IIRC there were a few problems with the run-as-root-drop-priviliges way
tv1 has quit [Quit: WeeChat 2.6]
tv has joined #nixos-dev
Jackneill has quit [Ping timeout: 260 seconds]
Jackneill has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
<Mic92> aszlig: I think in nix-user-chroot, I solved this problem.
<Mic92> for nix-user-chroot itself.
<aszlig> Mic92: ah, just did a quick look on it, but isn't that doing the same as nix already does?
<aszlig> it just maps a single uid from the parent/child namespace
<aszlig> so i think we'd have the same problem here as well
<aszlig> s/already/also/
<Mic92> aszlig: I think you are right
<Mic92> aszlig: However I think to remember that it was possible to map multiple uids
<yorick> Mic92: well, obviously it can't link to the host python when cross-compiling, so it fails
<yorick> I'm currently hacking cflags to make it find the cross python
<aszlig> Mic92: it is, but IIRC you need to be root for that
<aszlig> or more specifically need to have CAP_SETUID
<Mic92> aszlig: makes sense sort of.
<Mic92> I am still waiting for bindfs to happen
<aszlig> Mic92: you mean bind mounts as FUSE?
<Mic92> aszlig: There was bindfs kernel implementation that does uid shifts.
<Mic92> aszlig: right, it was called shifts
<Mic92> *shiftfs
<Mic92> Also this would not solve the problem for us, since we cannot mount it in nix.
<Mic92> bummer
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
qyliss has quit [Quit: bye]
qyliss has joined #nixos-dev
<emily> I feel like much of the tests don't need to be running kernels/VMs at all
<emily> it's valuable to test that stuff but it feels like there's a lot more userspace tests that would work fine in regular containers than there are full "VM tests" poking at kernel stuff
<emily> it would be nice not to run the kernels if only so that the test output was vaguely readable >_>
<aszlig> yeah, i think there was some work a while ago to implement this, since it needs to have some help from nix itself
<aszlig> well, you could still silence the kernel messages :-D
<aszlig> and the "help from nix"[TM] would pretty much boil down to the user namespace issue above
<aszlig> and also this would introduce a few more headaches, like for example you suddenly need to take care of networking instead of going through a user space switch
cole-h has joined #nixos-dev
<aszlig> the networking part should be manageable at some point, but how would you integrate it into nix?
<aszlig> like just some big privilege elevation escape hatch or tailored towards container runs?
<aszlig> and how do you prevent other derivations from abusing this to eg. escape the sandbox?
<aszlig> something like CSP on the web?
<aszlig> special build mode? what about hydra then?
<{^_^}> firing: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
<MichaelRaskin> There is a ton of things that need just a couple things running (service supervision not really needed) and definitely no networking and no kernel.
<gchristensen> someone might want to extend hydra to make this a configurable
<Ericson2314> Annyone wanna give me a thumbs up thumbs down on https://github.com/NixOS/nixpkgs/pull/86166 ?
<{^_^}> #86166 (by Ericson2314, 2 weeks ago, open): *-wrapper; Switch from `infixSalt` to `suffixSalt`
<Ericson2314> looking to make a pkg-config wapper after it
<aszlig> gchristensen: geesh --no-allow-... sounds so odd (and yes, I know it's because --no-* is handled in a generic way)
<aszlig> gchristensen: configurable on a per-jobset basis?
FRidh has quit [Ping timeout: 256 seconds]
FRidh has joined #nixos-dev
<cole-h> It's beautiful. nixos-unstable is unstuck. niksnut++
<{^_^}> niksnut's karma got increased to 17.00000000000000004
evanjs- has joined #nixos-dev
orivej has quit [Ping timeout: 246 seconds]
evanjs has quit [Ping timeout: 272 seconds]
<{^_^}> resolved: BuildsStuckOverTwoDays: https://status.nixos.org/prometheus/alerts
justanotheruser has joined #nixos-dev
orivej has joined #nixos-dev
<julm> floating karma, is... floating
drakonis has joined #nixos-dev
alp has quit [Ping timeout: 260 seconds]
alp has joined #nixos-dev
alp has quit [Remote host closed the connection]
alp has joined #nixos-dev
<Profpatsch> I want karma to rise exponentially, similar to power levels
<Profpatsch> By a random value between 2^(n-1) and 2^(n)
drakonis has quit [Ping timeout: 244 seconds]
orivej has quit [Read error: Connection reset by peer]
orivej has joined #nixos-dev
janneke has quit [Quit: janneke quits Mes'sing]
janneke has joined #nixos-dev
<gchristensen> adisbladis: I'm looking at ways to do that automatically (take the options from the Nix modules) and output structural json, but am tripping myself up. maybe you have some ideas with your recent spelunking
<clever> gchristensen: nixos options?
<gchristensen> NixOps specifically, but yaeh
* clever looks
alp has quit [Remote host closed the connection]
alp has joined #nixos-dev
<clever> # nix-build -o analyzer ; nix-build '<nixpkgs/nixos/release.nix>' -A options -o options ; ./analyzer/bin/option-analyzer ./options/share/doc/nixos/options.json
<clever> gchristensen: this is how the options page on the site got its data
<gchristensen> hmm yeah
<clever> gchristensen: and how my installer gui would generate it from whatever <nixpkgs> currently is, so its always up to date
<gchristensen> hmm!
<clever> 135 options = (buildFromConfig ({ ... }: { }) (config: config.system.build.manual.optionsJSON)).x86_64-linux;
<clever> nixos/lib/make-options-doc/default.nix: optionsJSON = pkgs.runCommand "options.json"
<clever> gchristensen: so just add s3-bucket.nix to the imports of your thing (maybe even -I nixos-config=?) and build that attr of nixos
<clever> gchristensen: oh, theres even an example at the top of nixos/lib/make-options-doc/default.nix
<infinisil> Profpatsch: I like that idea
orivej has quit [Quit: No Ping reply in 210 seconds.]
teto has joined #nixos-dev
orivej has joined #nixos-dev
evanjs- has quit [Quit: ZNC 1.7.5 - https://znc.in]
evanjs has joined #nixos-dev
drakonis has joined #nixos-dev
alp has quit [Remote host closed the connection]
alp has joined #nixos-dev
julm has quit [Remote host closed the connection]
julm has joined #nixos-dev
alp has quit [Ping timeout: 260 seconds]
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #nixos-dev
alp has joined #nixos-dev
orivej has quit [Ping timeout: 264 seconds]
lovesegfault_ has quit [Quit: WeeChat 2.8]
orivej has joined #nixos-dev
lovesegfault has joined #nixos-dev
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #nixos-dev
qyliss has quit [Quit: bye]
qyliss has joined #nixos-dev
orivej has quit [Ping timeout: 256 seconds]
kraem has quit [Ping timeout: 272 seconds]
orivej has joined #nixos-dev
alp has quit [Ping timeout: 260 seconds]
FRidh has quit [Quit: Konversation terminated!]
__monty__ has quit [Quit: leaving]
kraem has joined #nixos-dev
justanotheruser has quit [Ping timeout: 256 seconds]
justanotheruser has joined #nixos-dev