gchristensen changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | 18.09 release managers: vcunat and samueldr | https://logs.nix.samueldr.com/nixos-dev
drakonis has quit [Ping timeout: 252 seconds]
drakonis has joined #nixos-dev
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 250 seconds]
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 250 seconds]
drakonis_ has joined #nixos-dev
drakonis_ has quit [Ping timeout: 252 seconds]
drakonis_ has joined #nixos-dev
drakonis1 has joined #nixos-dev
drakonis_ has quit [Ping timeout: 252 seconds]
<colemickens> client: must succeed: curl --fail -g http://[fd60:cc69:b537:1:d93a:3eb:b6a2:bcdb]
<colemickens> client# curl: (3) URL using bad/illegal format or missing URL
<clever> colemickens: sounds like curl was built without v6 support?
<colemickens> yeah idk, it's from the trunk failure, thought it seemed a bit interesting, was seeing if I could figure out what change broke it :) https://nix-cache.s3.amazonaws.com/log/cg2njg8ycwqkm5i0xgvgjgyy2jysbnh0-vm-test-run-ipv6.drv
<clever> client# [ 0.000000] Command line: loglevel=7 console=ttyS0 panic=1 boot.panic_on_fail init=/nix/store/svpmpkqmzm6y6v7ngkb96csbfllc0rdf-nixos-system-client-19.03pre-git/init regInfo=/nix/store/biylx0lyrj13r48bi32jhgbf5vm23ccm-closure-info/registration console=hvc1 console=ttyS0
<clever> if you `nix-store -r` the storepath from `init=...`, it will give you that closure
drakonis_ has joined #nixos-dev
<clever> [clever@system76:~]$ nix-store -qR /nix/store/svpmpkqmzm6y6v7ngkb96csbfllc0rdf-nixos-system-client-19.03pre-git | grep curl
<clever> /nix/store/cbcxzibqf1ljyw5v902qyccqwr0r8515-curl-7.62.0
<clever> /nix/store/ygj70aid797zzcxpqcs1dqdqn1pkjfds-curl-7.59.0
<clever> colemickens: and there are 2 curls in its closure
<colemickens> interesting. thanks for sharing the nix-store commands too, I've only once used it (manually anyway) and it's nice to see some uses
<clever> 7.59 is linked to nix, 7.62 is in system-path
drakonis1 has quit [Ping timeout: 252 seconds]
<clever> ah, and curl has no binaries in .out
<clever> they are in .bin
<clever> 7.59's .bin isnt in the closure, so only 7.62 can have a curl binary
<clever> [clever@system76:~]$ /nix/store/i5pcdar16dzd5gp0dc9324vsx5yljqmz-curl-7.62.0-bin/bin/curl --version
<clever> Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy
<clever> [clever@system76:~]$ /nix/store/i5pcdar16dzd5gp0dc9324vsx5yljqmz-curl-7.62.0-bin/bin/curl --fail -g http://[fd60:cc69:b537:1:d93a:3eb:b6a2:bcdb] -v
<clever> * Closing connection -1
<clever> curl: (3) URL using bad/illegal format or missing URL
<clever> colemickens: it claims to have ipv6 support, but also fails when given the url from the testcase
<clever> but if i run my local curl, it fails with a different more expected error (host not found)
<colemickens> hm!
<clever> bisect time
<clever> nix-build -A curl, on nixpkgs master, gives me the faulty curl!
<clever> [clever@system76:~/nixpkgs]$ nix-build -A curl && ./result-bin/bin/curl --fail -g http://[fd60:cc69:b537:1:d93a:3eb:b6a2:bcdb] -v
<clever> Bisecting: 1164 revisions left to test after this (roughly 10 steps)
<colemickens> clever: how did you pick the other end of the range? did you just take the git sha from the last time the ipv6 test passed? Looks like bac76e8a9ef5981df1b133258484962b801181a3 ?
<clever> colemickens: i picked the rev from `nixos-version`
<clever> which is a random point on nixos-unstable, when i last updated my laptop
<colemickens> mk
<clever> which is known-good, from previous testing
<clever> 2 builds of curl are bad, a 3rd curl is building locally
<clever> colemickens: e9d6475e68188b2c81563cf9a96ea91a2b28623a is the first bad commit
<clever> its a curl version bump
<{^_^}> curl/curl#3220 (by hagai-auro, 2 weeks ago, closed): unexpected bad/illegal url format error. A short host + port is treated as a scheme!
<clever> colemickens: and the PR with that bump requested a backport, so 18.09 is likely also broken
<colemickens> :(
<clever> ofborg doesnt run nixos tests, and curl is also pretty deep in the dep-tree
<clever> so its hard to get good testing fast
<colemickens> surprised they haven't done a bugfix release (assuming that's the right bug, but it seems like a big coincidence if it isn't)
<colemickens> though, also surprised that that URL wasn't included in a curl unit-test. hm.
<clever> the nix sandbox may not configure ipv6, *looks*
<colemickens> I guess the number of users using the very latest curl and ipv6 must be low, go figure.
<colemickens> eh, the test pings (successfully) before curl'ing.
<clever> the nixos tests boot a full nixos in qemu, so it can have ipv6
<clever> but the curl build itself lacks qemu, so its using a network namespace on the host, configured according to the nix code
<colemickens> I'm not sure I follow what you're looking into now.
<colemickens> On the hydra test, the ipv6 ping works, so doesn't that mean the sandbox is fine?
<clever> the ipv6 ping happens inside a qemu, so the sandbox doesnt come into question
<clever> the curl derivation itself, isnt built inside qemu, so it has different rules to play by
<colemickens> hm, maybe I need to read up more about how the tests work. The lines where the curl runs and the lines where the ping look are right next to each other and have the same prefix. I don't know how I'd know they're running in different places
<colemickens> (or I'm totally off-base and missing something)
<clever> it has a `client` prefix, so its running in a qemu machine called client
<colemickens> oh you're trying to figure out why an earlier test didn't catch it?
<clever> i'm just saying that you cant test it as easily, prior to involving qemu
<clever> in the curl build itself
* colemickens nods
<colemickens> I see
<clever> also, #nixos just mentioned a very similar issue...
<colemickens> yeah, I was wondering if we should open up an issue to track at least.
<colemickens> and/or so multiple people don't (re)start diving in
<clever> too late, somebody already has a PR to fix it :P
delroth has joined #nixos-dev
<clever> [clever@system76:~/nixpkgs]$ git fetch origin pull/50905/head
<colemickens> noooooo
<colemickens> I mean, great :)
<delroth> ohai :)
<clever> [clever@system76:~/nixpkgs]$ git checkout FETCH_HEAD
<colemickens> oh it's you
<colemickens> sweet :).
<clever> [clever@system76:~/nixpkgs]$ nix-build nixos/release.nix -A tests.ipv6.x86_64-linux
<clever> the above commands will checkout whatever is in PR 50905, and then run the ipv6 test on it
<delroth> it's a curl change → needs to rebuild ~everything
<delroth> I'm currently running it locally and it's taking forever as expected
<clever> [clever@system76:~/nixpkgs]$ nix build -f nixos/release.nix tests.ipv6.x86_64-linux
<clever> this will give a better UI
<clever> cmake and others are building
<clever> ~200 packages remain
<clever> due to how fixed-output stuff works, its not a full mass-rebuild, but a lot of things still link to curl
drakonis1 has joined #nixos-dev
contrapumpkin has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
drakonis_ has quit [Ping timeout: 252 seconds]
drakonis_ has joined #nixos-dev
drakonis_ has quit [Client Quit]
drakonis1 has quit [Ping timeout: 276 seconds]
<clever> its even building llvm!
<delroth> yeah llvm and aws-sdk-cpp are the two annoying ones it seems like
<colemickens> something in some of the disk images pulls in aws stuff iirc, or it did some months ago.
<clever> nix depends on aws
<clever> for the s3:// store
drakonis has quit [Quit: WeeChat 2.2]
jtojnar has joined #nixos-dev
jtojnar has quit [Quit: jtojnar]
orivej has quit [Ping timeout: 268 seconds]
<delroth> clever: my nix-build tests.ipv6.x86_64-linux succeeded fyi with #50905
<{^_^}> https://github.com/NixOS/nixpkgs/pull/50905 (by delroth, 3 hours ago, open): curl: cherry-pick upstream patch for ipv6 url parsing
<delroth> dunno if yours is still running or what
makefu has quit [Ping timeout: 245 seconds]
<colemickens> delroth: I guess you need a new GH avatar if you stick around, eh? :)
* colemickens wants a cool nixos avatar
worldofpeace has quit [Ping timeout: 268 seconds]
FRidh has joined #nixos-dev
<srhb> delroth: Ah, I just did the same thing locally. Thanks for picking that up.
<srhb> Let's either get this in quickly to master or staging. Comments on the target? https://github.com/NixOS/nixpkgs/pull/50905
<{^_^}> #50905 (by delroth, 5 hours ago, open): curl: cherry-pick upstream patch for ipv6 url parsing
makefu has joined #nixos-dev
orivej has joined #nixos-dev
<arianvp> Has anybody tried packaging dracut for NixOS before?
__Sander__ has joined #nixos-dev
<andi-> why would you want that?
orivej has quit [Ping timeout: 240 seconds]
hedning has joined #nixos-dev
zimbatm has joined #nixos-dev
<niksnut> I wonder if Nix could be made to use systemd dynamic users to allocate build users
<niksnut> it would certainly simplify the installer if we don't have to create users
<zimbatm> would each builder be a systemd unit with socket activation?
<zimbatm> there could be some other benefits of extracting the builder as a service
<zimbatm> if the builder is re-usable between runs it could also save on the sandbox initialization time
<niksnut> turning them into units would not be practical
<niksnut> so I hope there is some d-bus mechanism to create users
orivej has joined #nixos-dev
<niksnut> basically, the UserLock class would call systemd to create a uid for us
<niksnut> however, it looks like systemd has only a limited uid range for dynamic allocations, and doesn't support allocating a uid range
<niksnut> which would be needed for full user namespace support
<niksnut> because we'd want to allocate (at least) 2^16 uids per build
<zimbatm> it looks like it allocates the user at unit startup
<niksnut> hm, too bad
<zimbatm> the logic is implemented in src/core/dynamic-user.c
<zimbatm> it looks like it uses filesystem locks to guarantee uniqueness
<niksnut> it could be done using systemd-run --property=DynamicUser=yes
<niksnut> probably too heavy though
<zimbatm> wouldn't that be a bit heavy? :D
<zimbatm> we could also boot a VM per build :p
<zimbatm> that would allow cross-compilation nicely ;)
Lisanna has joined #nixos-dev
drakonis has joined #nixos-dev
<flokli> zimbatm: that's not cross-compiling, but cheating :-P
<zimbatm> but it would make the builds more reproducible :p
<zimbatm> `derivation { vm-config = { memory = 2048; ...`
<zimbatm> it's a bit frustrating to do all the work and then be hit by memory constraints
jtojnar has joined #nixos-dev
averell has quit [Quit: .]
averell has joined #nixos-dev
<arianvp> Yes there is a dbus API for creating dynamic users
<arianvp> Afaik
<arianvp> Why is starting a unit 'too heavy' though?
<arianvp> I don't think it's that heavy to spawn units on demand with systemd-run
<arianvp> Note that systemd-run also has a dbus interface. Manager.StartTransientUnit
<clever> zimbatm: have you seen my qemu-user stuff?
<zimbatm> clever: where is it?
<zimbatm> arianvp: where should I look for the dbus API
<clever> zimbatm: that will configure binfmt-misc to run qemu on the configured arches, and also configure nix-daemon to support them
<arianvp> https://github.com/facebookincubator/pystemd is a nice way to interactively explore it
<arianvp> Which we conveniently packaged for nix during all systems go conf
<arianvp> :p
<zimbatm> clever: that's magic!
<globin> The RFC on improving the RFC process is online: https://github.com/NixOS/rfcs/pull/36
<{^_^}> rfcs#36 (by globin, 1 minute ago, open): Improving the RFC process
<clever> zimbatm: very, but it fails the occasional test within coreutils i think
<arianvp> \o/
<arianvp> Meta
<clever> zimbatm: needs a real arm mixed in, or maybe a full vm
<zimbatm> globin++
<{^_^}> globin's karma got increased to 2
<clever> but i dont think qemu-system supports smp
<zimbatm> wait. does it mean we can also run windows executables in linux
<clever> zimbatm: the examples for binfmt-misc explain how to wire PE files to wine and JAR files to java
<clever> you then just need to +x the "executables" and run them like normal
<zimbatm> boom
<arianvp> Wow cool
<zimbatm> --disable-kvm ?
<clever> qemu-system cant use kvm when the arch differs
<zimbatm> oh yeah
<clever> qemu-user can never use kvm (i believe)
<clever> i have in the past used qemu-user to emulate x86_64, on x86_64!
<clever> and that successfully reproduced bugs that occured only in non-kvm qemu-system
<clever> but without having to boot an entire OS first
<zimbatm> I don't understand how the executable gets access to the current environment
<zimbatm> is there a kernel in the VM?
<clever> qemu-user will just translate every syscall
<clever> no 2nd kernel exists
<zimbatm> oh wow
<zimbatm> and that works?
<clever> yep
<zimbatm> that's crazy
<clever> yep
<zimbatm> and thus, no p9fs bugs
<clever> ive seen p9 get inodes crossed
<clever> and a random directory appears in the contents of another
<clever> zimbatm: qemu-user also allows it to dynamically get as much ram (and swap) as it needs, rather then assigning a fixed amount at boot
<zimbatm> what is the catch?
<zimbatm> it seems too good to be true :)
<clever> some syscalls are unsupported, like some modern rng ones
<clever> and a few syscalls behave slightly differently, causing coreutils tests to fail
<clever> and its slower at single-threaded loads then a real arm
<clever> but x86 has more cores, so you can still out-do it overall
<gchristensen> /!\ RFC process updates: https://github.com/NixOS/rfcs/pull/36 /!\
<{^_^}> rfcs#36 (by globin, 24 minutes ago, open): Improving the RFC process
<Profpatsch> Meta RFC!
<simpson> "The bureaucracy is expanding to meet the needs of the expanding bureaucracy."
<Profpatsch> Waiting for RFC 37: “Merge RFC 36: Improving the RFC process”
<simpson> It *does* seem like an improvement that is likelier to accelerate RFC cycles.
<gchristensen> simpson: good :)
<arianvp> zimbatm: I linked the dbus docs above by the way. Forgot to tag you
<arianvp> In case you missed it
<gchristensen> beaurocracy isn't a bad word, as much as people would like it to be :P
<arianvp> That's a very dutch stance
<arianvp> :')
<gchristensen> arianvp: that beaurocracy isn't a bad word? or that people would like it to be?
<arianvp> That bureaucracy isn't a bad word
<gchristensen> ah :)
<Profpatsch> Meeting the needs of the many
<simpson> An inherent part of the cycle: https://en.wikipedia.org/wiki/Discordian_calendar
<gchristensen> lol
<zimbatm> thanks arianvp, I got too excited about the qemu stuff :)
__Sander__ has quit [Quit: Konversation terminated!]
<zimbatm> arianvp: I haven't found a way to access dynamic users with dbus directly
<zimbatm> the only way is to start a unit
<zimbatm> (which can be done over dbus)
<Mic92> Ok. Go cross compiling is now complete. It is now also possible to cross-compile the go compiler itself.
<gchristensen> :D
<domenkozar> would you consider debugger for Nix (step across scopes, inspect bindings) something much appreciated?
<clever> i have thought about makign something like that as well
<clever> ive also thought about a better profiler
<clever> ,profiling
<{^_^}> Use NIX_COUNT_CALLS=1 and/or NIX_SHOW_STATS=1 to profile Nix evaluation
<clever> that gets total call count for every nix function and primop, but not much much cpu time is spent in each
<clever> being able to generate something like a flamegraph would make it a lot simpler to know which thing is to blame
<domenkozar> both are not that hard to do with hnix nowadays
<domenkozar> a few days of hacking, I think
* domenkozar is thinking about xmas gifts
<Mic92> domenkozar: yes, an (inspect $expression) function that spawns the repl would be great.
<domenkozar> that almost exists in hnix
<domenkozar> I'd like to go further, instad of getting a backtrace, you get interactive debugger
<domenkozar> instead*
<Mic92> I have no idea how a debugger would work in a lazy programming language
<domenkozar> you can go into each stack
<domenkozar> and see all bindings
<clever> Mic92: i was thinking something that lets you walk the AST manually, and then eval things in a given scope
<domenkozar> it's easy in nix since they all have Show instance :D
<domenkozar> the way scopes are implement is so elegant
<clever> i was also thinking more about the c++ nix, not hnix
<domenkozar> it's just a list and it uses a readert to go provide the scope
<clever> for example, out of the box, snack takes over 3 days to eval cardano-sl-core (and it didnt even finish)
<clever> it involves derivations and IFD heavily
<Mic92> I am fine with with both
<clever> does hnix support IFD?
<domenkozar> not (yet)
<domenkozar> hnix almost got proper string context support last weekend
<domenkozar> next step is proper nix-store abstraction
<clever> then your going to have a VERY hard time getting snack to profile in hnix
<domenkozar> then ifd
<domenkozar> hnix is still much slower than cnix
orivej has quit [Ping timeout: 268 seconds]
<clever> oh, and if you profile it to optimize for the slow primops in hnix, it may still perform worse on cnix, due to its primops being different speeds
<domenkozar> I guess those were the questions more for #nixos
<domenkozar> since we're all pretty used to what goes wrong with Nix :)
<zimbatm> Mic92: I read "Gross compiling is now complete" :p
MichaelRaskin has joined #nixos-dev
<Mic92> zimbatm: sometimes it is.
FRidh has quit [Quit: Konversation terminated!]
orivej has joined #nixos-dev
worldofpeace has joined #nixos-dev
<Mic92> Ericson2314: what would be the lib/systems/examples.nix entry for macOS?
<Mic92> Ericson2314: I can already produce go binaries for windows
<Ericson2314> Mic92 haha there might not be one
<Ericson2314> Nice with Go though!
<Mic92> Yes, the question is what would I need to put there
drakonis has quit [Ping timeout: 260 seconds]
<Mic92> zimbatm: ok. direnv also compiles with nix on Windows now, your makefile however skips the extension
<Mic92> not on but for
<Ericson2314> Mic92 sorry on the phone so need ping for notification. `{ config = "x86_64-apple-darwin"; }` should be enough
<Ericson2314> You can instantiate nixpkgs non-cross and then look at `hostPlatform` for reference.
<Mic92> Ericson2314: thanks
<Ericson2314> NP! Hope that works
orivej has quit [Ping timeout: 240 seconds]
<zimbatm> Mic92: what do I need to fix in the Makefile for this to work?
worldofpeace has quit [Ping timeout: 250 seconds]
<Mic92> zimbatm: append $(go env GOEXE) to go build -o
<Mic92> GOEXE is ".exe" on Windows
<{^_^}> direnv/direnv#417 (by zimbatm, 16 seconds ago, open): make: generate direnv.exe on windows
<{^_^}> #50928 (by Mic92, 3 minutes ago, open): direnv: 2.17.0 -> 2.18.1 + make cross-compile on windows
lassulus has quit [Remote host closed the connection]
lassulus has joined #nixos-dev
phreedom has quit [Ping timeout: 256 seconds]
drakonis has joined #nixos-dev
drakonis_ has joined #nixos-dev
phreedom has joined #nixos-dev
jtojnar has quit [Quit: jtojnar]
<Mic92> To get a cross-compiler for linux