worldofpeace_ changed the topic of #nixos-dev to: #nixos-dev NixOS Development (#nixos for questions) | NixOS stable: 20.03 ✨ | | | 19.09 RMs: disasm, sphalerite; 20.03: worldofpeace, disasm |
lassulus has quit [Ping timeout: 246 seconds]
lassulus has joined #nixos-dev
alp has quit [Ping timeout: 272 seconds]
orivej has quit [Ping timeout: 264 seconds]
ris has quit []
lassulus has quit [Ping timeout: 260 seconds]
lassulus has joined #nixos-dev
ajs124 has quit [Quit: killed]
das_j has quit [Quit: killed]
Scriptkiddi has quit [Quit: killed]
ajs124 has joined #nixos-dev
Scriptkiddi has joined #nixos-dev
das_j has joined #nixos-dev
drakonis has quit [Ping timeout: 260 seconds]
<kiwiirc> with systemd-networkd integration, flakes, etc it looks like nix is solving its big shortcomings. cool that the project isn't standing still
<gchristensen> we haven't stood still much in 17 years :)
<JJJollyjim> It's pretty exciting to see everything moving
<JJJollyjim> Jesus 17 years?
<kiwiirc> well i really appreciate the honesty to admit shortcomings and with a positive attitude to just make it better
<JJJollyjim> Nix is only 4 years younger than me
<JJJollyjim> It feels so new
<kiwiirc> lots of projects turn hard ego and it just ruins stuff
<gchristensen> Nix started in March 2003
<JJJollyjim> But I guess it became a lot more well known recently
<JJJollyjim> domenkozar: fwiw, the reason I've never considered using cachix is that I live in new zealand where latency is a big deal, and there's no documentation on where stuff is hosted
<JJJollyjim> (it's not satellite internet or anything, it's just speed of light issues)
<JJJollyjim> A lot of derivations would be faster to build locally than to check a European server for a narinfo
<JJJollyjim> (rtt≈350ms)
<gchristensen> it is hosted in S3 at least
<gchristensen> but I guess that is too vague
cole-h has quit [Quit: Goodbye]
<JJJollyjim> Yeah
<JJJollyjim> s3 guarantees that it won't be Great (within NZ), but it can be Okay (Australia)
<JJJollyjim> But it could also be Europe at which point it's awful
<JJJollyjim> New Zealand's internet infrastructure is fantastic, unfortunately Australia's isn't
<JJJollyjim> So AWS charges extra for bandwidth in AU regions
<JJJollyjim> Which affects us too
<JJJollyjim> Azure is the first major cloud provider to announce a region here, but they haven't started building it
<JJJollyjim> (my former employer also has a really nice 3-region openstack)
FRidh has joined #nixos-dev
alp has joined #nixos-dev
drakonis_ has quit [Ping timeout: 256 seconds]
drakonis_ has joined #nixos-dev
alp has quit [Ping timeout: 272 seconds]
FRidh has quit [Ping timeout: 240 seconds]
FRidh has joined #nixos-dev
orivej has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 265 seconds]
orivej has joined #nixos-dev
<domenkozar[m]> Jamie: it's served by cloudflare all around the world, did you do some measurements?
alp has joined #nixos-dev
<JJJollyjim> nah hadn't tested
<JJJollyjim> presumably the backing store is still far away though?
<JJJollyjim> so if i ask about a hash for the first time, cloudflare will ask the backing store?
<JJJollyjim> (to be clear i'm not a likely high-paying customer and you shouldn't put any effort in to making things better for me :P)
__monty__ has joined #nixos-dev
Jackneill has joined #nixos-dev
orivej has quit [Ping timeout: 258 seconds]
orivej has joined #nixos-dev
<domenkozar[m]> well I want to clear up if there are any problems with its access
<domenkozar[m]> Jamie: yes, cloudflare will ask the backing store and serve that via their backbone, which won't be the same as being in the same region, but should be within reasonable ms
<JJJollyjim> 1.065s including connection establishment and such
<JJJollyjim> Trying other hashes now it's ~0.83s
<JJJollyjim> Presumably cloudflare opened a persistent connection to the backend after the first request
<JJJollyjim> Also looks like cloudflare isn't caching the 404s
<JJJollyjim> Which makes sense
<JJJollyjim> Actually repeated requests to the same 200 narinfo are still pretty slow :/
<JJJollyjim> 0.4s sometimes, 0.8s sometimes
<JJJollyjim> I guess it's not popular enough for them to actually cache?
<JJJollyjim> rtt to the cloudflare node is 30ms so 400ms is unexpectedly high
<andi-> Yeah, I have a 1.1ms RTT to the CF server and >500ms until that narinfo returns
<domenkozar[m]> Jamie: yeah all connections are HTTP2, so you pay connection price once
<domenkozar[m]> Jamie: oh narinfo is never cached, only nars are
<JJJollyjim> Ah right
<JJJollyjim> So checking for presence in the cache will always involve talking to the other side of the world :(
<domenkozar[m]> that's interesting, let me check response times on the server
<domenkozar[m]> upstream_response_time=0.004
teto has quit [Ping timeout: 265 seconds]
<domenkozar[m]> andi-: where are you located?
<andi-> domenkozar[m]: Frankfurt
<domenkozar[m]> yeah those are all affected by TTFB
<domenkozar[m]> in practice you'll be capped at that since narinfos are looked up in parallel
orivej has quit [Quit: No Ping reply in 180 seconds.]
<domenkozar[m]> I'll look into getting that down
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 260 seconds]
orivej_ has joined #nixos-dev
<domenkozar[m]> andi-: Jamie could you do another test?
<andi-> domenkozar[m]: sure, what do you want me to run?
<domenkozar[m]> (a few times)
<andi-> ~0.550 total
<andi-> there was one with 0.2
<andi-> running `ab -c 1 -n 1000` now to get some more proper results
tv has quit [Ping timeout: 260 seconds]
<domenkozar[m]> not sure if that reuses connections
<domenkozar[m]> if so I'd expect median to be ~200ms
<andi-> good point, here is the output now
<JJJollyjim> 0.85s+
<domenkozar[m]> yeah so 200ms from EU to US is sensible
<andi-> `time curl` still reports 0.5s so I'd not use that benchmark
<domenkozar[m]> well, 244ms to be precise
<domenkozar[m]> andi-: cloudflare is optimized for webpages so time to first byte will be slow
<domenkozar[m]> in practice that's not a good metric as you'll be requesting dozens if not hundreds of narinfos
<domenkozar[m]> Jamie: could you also run ab?
<JJJollyjim> is it in nixpkgs?
<domenkozar[m]> yes
<domenkozar[m]> nix-shell -p apacheHttpd --run "ab -c 1 -n 100"
<JJJollyjim> ah lol didn't think to look in the httpd package
<domenkozar[m]> there I expect median to be around 300ms, maybe 350ms
<domenkozar[m]> (according to latency reports between east coast and NZ)
<JJJollyjim> median 218
<domenkozar[m]> oh wow that's pretty good
<JJJollyjim> wildly variable though, 75th percentile is 649
<andi-> note ab doesn't actually do HTTP2
<andi-> I am just trying with h2load
<domenkozar[m]> yeah luckily that doesn't affect Nix too much
<andi-> Yeah, multiplexing multiple requests through the same H2 connection doesn't seem to make any noteable difference. Comparing 1 request per connection vs 2 right now.
tv has joined #nixos-dev
orivej_ has quit [Ping timeout: 260 seconds]
orivej has joined #nixos-dev
kiwiirc was banned on #nixos-dev by gchristensen [*!68c88135@gateway/web/cgi-irc/]
kiwiirc has left #nixos-dev [requested by gchristensen (kiwiirc)]
orivej has quit [Ping timeout: 265 seconds]
orivej has joined #nixos-dev
alp has quit [Ping timeout: 272 seconds]
rajivr has joined #nixos-dev
orivej has quit [Ping timeout: 246 seconds]
orivej has joined #nixos-dev
xwvvvvwx has quit [Quit: ZNC 1.8.0 -]
xwvvvvwx has joined #nixos-dev
__monty__ has quit [Quit: leaving]
alp has joined #nixos-dev
alp has quit [Client Quit]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Read error: Connection reset by peer]
orivej has joined #nixos-dev
<domenkozar[m]> abathur: heh, well I told him and others to use github actions
<domenkozar[m]> that's the one I maintain, but if you want something else then you'll have to put in some work :)
<abathur> heh
<domenkozar[m]> the fact that you'd rather spend time is weird to me, but :shrugs:
<abathur> you-me, or you-him? :)
<domenkozar[m]> well anyone that keeps using travis :)
drakonis_ has quit [Ping timeout: 264 seconds]
drakonis_ has joined #nixos-dev
<domenkozar[m]> abathur: do you want to be officially the maintainer for travis?
drakonis has joined #nixos-dev
<{^_^}> travis-ci/docs-travis-ci-com#2824 (by domenkozar, 7 seconds ago, open): nix: step down as maintainer
drakonis_ has quit [Ping timeout: 265 seconds]
<Profpatsch> abathur: we switched to Github actions and I can confirm that they are better than Travis in probably every aspect
<Profpatsch> easier to set up, easier to configure, better integrated into Github, and faster
<Profpatsch> where *we* is lorri
<domenkozar[m]> thanks Profpatsch
spacekookie has quit [Quit: **aggressive swooshing**]
drakonis_ has joined #nixos-dev
spacekookie has joined #nixos-dev
spacekookie has quit [Client Quit]
spacekookie has joined #nixos-dev
drakonis has quit [Ping timeout: 240 seconds]
<gchristensen> domenkozar[m]: I have a proposal for something to recommend against in your nix lang doc
<aszlig> niksnut: i'm currently trying to add nix flake support to vuizvui (, however i'm not sure how flakes should replace channels or let's say the way we're using them in vuizvui...
<aszlig> niksnut: so what we do in vuizvui is that we run tests based on various machine configurations and provide per-machine channels (built by hydra), so that whenever eg. docker is broken in nixpkgs master, machines without enabled docker will still get a channel upgrade
<aszlig> how would something like this look like when using flakes?
<aszlig> some kind of bot that randomly tries to update several flake.lock-files and provides pull requests for each machine, which are in turn built by hydra?
<domenkozar[m]> gchristensen: great, please open a PR :)
<gchristensen> aww
<gchristensen> you don't even want to discuss it...? :P
orivej has quit [Ping timeout: 258 seconds]
orivej has joined #nixos-dev
kfound has joined #nixos-dev
kfound has quit [Remote host closed the connection]
<domenkozar[m]> gchristensen: oh of course I do :)
<niksnut> aszlig: I keep forgetting that hydra still has a "channels" feature
orivej has quit [Ping timeout: 246 seconds]
<gchristensen> it is a really good feature!
<aszlig> niksnut: which is good :-)
<niksnut> I was tempted to write "channels" "feature" :p
orivej has joined #nixos-dev
<aszlig> well, otherwise implementing something like vuizvui would be pretty annoying
<gchristensen> it is a really important feature for a lot of users
<niksnut> I forgot about the semantics
<niksnut> doesn't it include the latest succeeded build from a mix of evaluations
<niksnut> ?
<gchristensen> yeah
<gchristensen> I think it is possible to say that hydra channels just stay channels
<gchristensen> and isn't an appropriate use case for flakes
<niksnut> that sort of thing is pretty incompatible with flakes
<gchristensen> yeah
<niksnut> you can't really have a flake depending on the previous state of hydra
<aszlig> niksnut: well, i know :-D
<aszlig> but let's say in a flakes-only future, how would you implement something like this?
<domenkozar[m]> gchristensen: I'm a bit jumping around since I'm packing, but do share thoughts
<niksnut> I don't understand the use case well enough, but maybe you could have multiple jobsets in the flake (e.g. with or without docker)
<gchristensen> domenkozar[m]: I think "..." is, largely, a mistake
<domenkozar[m]> hmm, but wouldn't we have to then offer an alternative for nixos modules?
<gchristensen> well, maybe not for nixos modules
<gchristensen> but so many builder functions accept `...` instead of providing a clear API
<aszlig> niksnut: well, essentially the same applies to nixpkgs as well, but since it's "upstream" it isn't such an issue, since all you need to do is update a branch ref
<aszlig> like eg. nixos-20.03
<domenkozar[m]> gchristensen: yeah it's easy to have a chain of functions with a lot of ...
<aszlig> but for vuizvui it's "that machine configuration with the latest *tested* version of nixpkgs master"
<domenkozar[m]> and it doesn't catch typos
<gchristensen> domenkozar[m]: yeah, exactly, and I think the "original sin" is mkDerivation using the main attriibute set scope to set env vars, instead of a clearer API of { environment = { ... }; }
<domenkozar[m]> gchristensen: sgtm
<gchristensen> I was helping someone who spent like 3 days to realize a function, deep in a stack of functions, doesn't actually do anything with an argument
orivej has quit [Quit: No Ping reply in 180 seconds.]
<domenkozar[m]> yeah and nixos could have extra attribute
<gchristensen> yeah
<gchristensen> exactly
orivej has joined #nixos-dev
<JJJollyjim> Aaaaa you are giving me flashbacks to the number of times I have spelt propagatedBuildInputs as propogated
<gchristensen> that too
<domenkozar[m]> you can probably make a PR to nixpkgs each month with a number of those
<pie_> i have an issue somewhere about warning users about mkderivation arguments but idk if any of them were actionable
<pie_> at least i remember something specifically related to the phases problems
<aszlig> JJJollyjim: haha, maybe write a linter that calculates the leavensthein distance and warn of such mistakes? :-D
<domenkozar[m]> we don't even have a page documenting mkDerivation :D
<domenkozar[m]> how embarrassing is that
<pie_> what id probably do is write a makeWrapper function that does the wrapping
<gchristensen> :)
<pie_> which is anoyher annoyance i have
<pie_> and then build infra for that off that
<JJJollyjim> D: ...... :D
<pie_> because guess what
<domenkozar[m]> like: here's a whole ecoystem around one function, but read the source luke
<pie_> you have to add the post and pre phases yourself when youre overriding
<pie_> havent had any issues caused by that , nope :P
<gchristensen> domenkozar[m]: look for examples, luke
<gchristensen> lol
<aszlig> domenkozar[m]: well, at least the source isn't outdated =)
<pie_> working on nixpkgs is a big pile of mistakes youve made before and folk knowledge
<domenkozar[m]> the good news we can fix that
<domenkozar[m]> is*
<pie_> right
<domenkozar[m]> yeah there's too much tacit knowledge
<gchristensen> domenkozar[m]++ I feel like that is my mantra
<{^_^}> domenkozar[m]'s karma got increased to 24
<LnL> domenkozar[m]: what do you mean? there's an entire chapter in the manual
<domenkozar[m]> it's our biggest barrier
<domenkozar[m]> it reminds of an essey: group is its own worst enemy
<LnL> not that it can't use some work
<pie_> gchristensen Inc. "the good news is we can fix that" (tm)
<domenkozar[m]> essay*
<{^_^}> #56785 (by deliciouslytyped, 1 year ago, open): Usability: warn about nonexistent phases
<{^_^}> #57323 (by deliciouslytyped, 1 year ago, open): Usability(?)/interfaces: setting a phase should not clear calls to pre- and post- hooks
<domenkozar[m]> LnL: that page is utterly confusing to newcomers
<pie_> something something four kinds of documentation
<domenkozar[m]> it goes explaining wrong things
<LnL> s/some/a lot/ :)
<domenkozar[m]> like there's a chapter on specifying dependencies
<domenkozar[m]> and two paragraphs later you're reading about natural deduction
<ajs124> pie_: these two issues feel like they're related to #72074 somehow. not exactly sure how, though.
<{^_^}> (by globin, 34 weeks ago, open): stdenv: enable __structuredAttrs
<domenkozar[m]> I'll stop complaining now :D I think it doesn't matter what we think, but what users think
<domenkozar[m]> and their feedback is consistent
<Valodim> can confirm
<niksnut> we're working on UX improvements
<pie_> ajs124: idk, crosslink it :P
<niksnut> by improving the nix command and maybe providing an alternative for the nix language that supports discoverability and documentation from the start
<domenkozar[m]> niksnut: which is great!
<domenkozar[m]> nice
<ajs124> pie_: no u
<domenkozar[m]> I'll meanwhile work on documentation
<LnL> domenkozar[m]: yeah, there's some other stuff in there but it's basically only usable as reference for (some) of the stdenv attributes
<domenkozar[m]> just to be explicit, I think everyone is doing an amazing job
<domenkozar[m]> I'm only dissatisfied with the focus
<domenkozar[m]> there's too much development on new fancy stuff instead of getting one thing right and working
<domenkozar[m]> niksnut: I think getting cli out should be the focus
<domenkozar[m]> then move on to discoverability & such
<domenkozar[m]> 2.4 is going in the other direction with experimental flags
<domenkozar[m]> (these two are a bit orthogonal, but they do impact users quite a bit)
<domenkozar[m]> I had a number of companies subscribe to cachix and then cancel as they switched away from Nix
<domenkozar[m]> "cross compilation documentation was indistinguishable from magic so we used docker"
orivej has quit [Ping timeout: 246 seconds]
<Valodim> ouch
<domenkozar[m]> that's as honest as the feedback gets :)
<Valodim> I'm really excited to see how flakes will affect the landscape
orivej has joined #nixos-dev
<domenkozar[m]> yeah flakes will be great for a lot of things
<domenkozar[m]> half of will get vastly simplified
<Valodim> I have some hopes that they solve many of the issues I find myself troubled explaining :)
<domenkozar[m]> I think we mainly need to limit the scope of what to support
<domenkozar[m]> and make those parts really good
<domenkozar[m]> speaking of which, I should finish github actions installer artifact for nix
<pie_> does anyone other than ericson understand the cross compile infra thoroughly
<pie_> does _ericson_? i kind of just assume he wrote it all
<niksnut> domenkozar[m]: btw somebody suggested turning experimental features into a warning rather than a fatal error
<niksnut> so all that --experimental-features would do is shut up the warnings
<domenkozar[m]> that's a great idea
<domenkozar[m]> it delivers the message without crippling the user
joepie91 has joined #nixos-dev
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #nixos-dev
<hexa-> I'm wondering whether the use of meta.maintainers could be improved by informing users when that list is empty for a package they're using
<gilligan> niksnut: yeah i also think that this sounds like a good suggestion
<hexa-> that might certainly be noisy for some libs, I know, but this would be a way to reach actual users of a package.
<pie_> might be worth bringing up with the people that are phrasing the stale bot, if it gets anywhee
<pie_> or well, people on the stale bot issue
<pie_> i still think a problem is we simply dont have the developer base to keep every package with a maintainer proper, but i have no explicit basis to believe that
tazjin has left #nixos-dev [#nixos-dev]
drakonis has joined #nixos-dev
drakonis1 has joined #nixos-dev
drakonis_ has quit [Ping timeout: 258 seconds]
drakonis has quit [Ping timeout: 240 seconds]
drakonis has joined #nixos-dev
__monty__ has joined #nixos-dev
drakonis1 has quit [Ping timeout: 258 seconds]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
<Ericson2314> niksnut: did you ever consider using libfetchers for fetchurl?
drakonis1 has joined #nixos-dev
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 260 seconds]
<Ericson2314> For the git stuff, I am wishing the narHash field of input was actually std::optional<FixedOutputHash>
<Ericson2314> so basically would allow something like outputHashRecursive to decide whether to use nar or git tree hashing regardless of whether one pinned a tree hash
<Ericson2314> but the change would be useful to master even if there was a libfetcher that would take advantage of flat hashing
<Ericson2314> (whereas now they all fetch trees and use recursive)
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
<abathur> domenkozar[m] Profpatsch a few thoughts re: travis-ci; 1. no on sole maintainership for now at least (I'm in "no" mode for a bit).
<abathur> 2. If it's being abandoned by everyone central to Nix, it needs a deprecation notice w/recommendations instead of breaking some day and not getting fixed. It's a bad look given the problems Nix is supposed to solve. It can leave people over a barrel when they're just trying to rush out a fix.
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-dev
<gchristensen> srhb has been +o in a handful of channels for a long time, but I'm planning on extending her +o to all the nix and nixos channels later today. also, I'm planning on making all the channels share a ban list. anyone have thoughts on this?
orivej has quit [Ping timeout: 264 seconds]
orivej_ has joined #nixos-dev
<Mic92> domenkozar[m]: I am surprised that travis does not make the ci user a trusted nix user, which makes this fail:
<Mic92> someone just send me this. I have not used this travis stuff in a while, but I thoughed it worked at some point.
rajivr has quit [Quit: Connection closed for inactivity]
justanotheruser has quit [Ping timeout: 272 seconds]
alp has joined #nixos-dev
<abathur> Mic92: I don't have a sense of what would cause that; but here's the most recent change:
<abathur> the linux install became multi-user, if that could cause it
<Mic92> abathur: thanks that would make sense
cole-h has joined #nixos-dev
<abathur> I also cut some lines around the creation of the /nix path because installers prior to 2.3 were refusing to install because /nix already existed
<abathur> (I think, again, because of something the multi-user installer does that single doesn't?)
<abathur> at least, I don't recall the single user installer objecting to /nix, but I'm not sure I've ever run it with it present
<Mic92> abathur: yes. in multi-user mode you are not a trusted nix user by default which causes this error.
<abathur> matthewbauer and asymmetric discussed that starting at and for a few comments
<Mic92> abathur: I also commented on that
justanotheruser has joined #nixos-dev
<abathur> maybe in the other PR?
<Mic92> no just now
<abathur> ah :) -- in any case, this seems to underscore point #2 earlier about adding a deprecation notice and nudging people to migrate
orivej_ has quit [Ping timeout: 260 seconds]
orivej has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
bennofs has joined #nixos-dev
bennofs_ has quit [Ping timeout: 264 seconds]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 246 seconds]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 260 seconds]
orivej has joined #nixos-dev
alp has quit [Ping timeout: 272 seconds]
orivej_ has joined #nixos-dev
orivej has quit [Ping timeout: 264 seconds]
FRidh has quit [Ping timeout: 256 seconds]
hoverbear has joined #nixos-dev
<hoverbear> Hi friends, Can I cross compile from Linux x86 to OS X x86 with pkgsCross?
<qyliss> no
<hoverbear> Ok :)
<qyliss> you can't really cross-compile to macOS, AIUI
<hoverbear> So if I wanted to do that I'd need a mac CI machine to do my dirty work?
<gchristensen> hmm I thought you could as long as you didn't need frameworks, qyliss (but I am totally not up to date on this stuff)
<hoverbear> We need securityfoundation :()
<gchristensen> ack
<hoverbear> Okay well I'll leave that for tomorrow-me =D
<qyliss> Oh that's interesting
<gchristensen> > pkgs.darwin.Security.meta.description
<{^_^}> attribute 'description' missing, at (string):317:1
<gchristensen> >
<{^_^}> "Security-osx-10.9.5"
orivej_ has quit [Ping timeout: 260 seconds]
<hoverbear> :think:
<qyliss> How does linking work? I thought Darwin had its own special linker -- can you build that to run on Linux?
<gchristensen> this might not be useful I have no idea
orivej has joined #nixos-dev
<hoverbear> Isn't it lld?
<LnL> qyliss: you can if you import some kind of xcode blob first
<hoverbear> iirc os x is basically a custom llvm tree?
<hoverbear> Whoaaa
<qyliss> It has to create mach-o executables rather than ELF ones though
<LnL> but this is all pretty unstable
<LnL> I have no idea about the details but there's even a way to build iOS stuff
<qyliss> wow
<qyliss> that's fascinating
<hoverbear> Yeah I see the ios stuff in pkgsCross
<qyliss> I'd always assumed that iOS stuff could only be build from a Mac
<gchristensen> I think that was actually a big part of john working on cross so much
<LnL> yeah
FRidh has joined #nixos-dev
<hoverbear> Ahh ok
<LnL> this is all pretty much black magic to me (I barely understand regular cross compiling)
<hoverbear> Out of curiousity if I wanted to build for armv7 musl would I want `pkgs.pkgsCross.armv7l-hf-multiplatform.pkgsCross.musl32` ?
<LnL> so unless you're _really_ motivated native is probably a more sane choice
<gchristensen> pkgsCross.armv7l-hf-multiplatform.pkgsStatic
<gchristensen> I think that is a thing
<LnL> I don't think you can mix those
<hoverbear> That's musl not gnu?
<gchristensen> oh
<hoverbear> Specifically want musl :)
alp has joined #nixos-dev
<LnL> if you want something more specific you have to define your own cross config I think
<hoverbear> Hmmm ok
<hoverbear> I think I'll leave that for future me, glibc is enough for now
<samueldr> pkgsStatic is musl
<samueldr> combining pkgsCross...pkgsCross IIRC doesn't mix well
<samueldr> but pkgsCross...pkgsStatic does
<hoverbear> Ohhh ok
<samueldr> at one point Mobile NixOS' stage-1 was built using pkgsCross.aarch64-multiplatform.pkgsStatic
<hoverbear> Ok :) Thank you I will try this, it makes a lot of sense
<hoverbear> samueldr: So should I generally be using `pkgsStatic` in place of `pkgsCross.musl*`?
<qyliss> Does pkgsCross.musl static link?
<samueldr> they don't do the same thing AFAIUI
<samueldr> pkgsCross.[...] setup the whole system
<hoverbear> Would pkgsStatic usually be sufficient for a mkDerivation?
<samueldr> while [pkgs.]pkgsStatic "only" tacks on musl+static
<samueldr> > pkgsCross.aarch64-multiplatform.pkgsCross.gnu64.hello == pkgsCross.gnu64.hello
<{^_^}> true
<samueldr> AFAIUI going through pkgsCross does not compose ever
<samueldr> now, what's with pkgsCross.musl* ? I don't know :) never used those
<FRidh> its non-static musl
<hoverbear> Ohhh wow ok
<hoverbear> Neat
<samueldr> that's what I was going state as assuming
<hoverbear> I am feeling so enlightened
<samueldr> but there is also pkgsMusl
<samueldr> > pkgsMusl.hello
<{^_^}> "<derivation /nix/store/f8b5k7l0jd4xmb1nf3l09v6aknna2d18-hello-2.10.drv>"
<FRidh> we just need to be a bit more precise sometimes with optionals whether they are for musl or musl static
<hoverbear> Is pkgsMusl != pkgsStatic?
<hoverbear> Because it's reasonable to want to static link a glibc build to everything but glibc
<samueldr> > pkgsCross.musl64.hello == pkgsMusl.hello
<{^_^}> false
<samueldr> > pkgsMusl.hello == pkgsStatic.hello
<{^_^}> false
<samueldr> AFAIUI pkgsStatic is a misnomer, and should have been pkgsMuslStatic
<hoverbear> I presumed `pkgsStatic` would be static glibc and `pkgsMusl` would be a dynamic or static musl
<hoverbear> Ok so how do I get mostly-static glibc?
<samueldr> >
<{^_^}> "<derivation /nix/store/bb2xgfzd6ls7dslmbc9p7qfni20xjpip-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.3.0.drv>"
<samueldr> good question
<samueldr> I think you should look at the definition of pkgsStatic
<samueldr> I seem to remember trying to look into it and failing to make it work, but probably because I didn't spend many brain cycles on it
<FRidh> see stage.nix and static.nix
<infinisil> I feel like there should be something like pkgs.withStdenv { static = true; musl = true; ... }
<hoverbear> Hmmm that'd be handy.
<infinisil> And the pkgsMusl/pkgsStatic/etc. would just be an alias to one of those
<samueldr> I think it's "kinda like that" but more complex in their definitions
<samueldr> (going from memory)
<FRidh> pkgsStatic is mostly just an overlay
<hoverbear> Oh a withStdenv would be awesome omg
<infinisil> Oh another related idea: Maybe aliases should not be allowed to be overridden in overlays
<hoverbear> Okay I feel like armed with the knowledge that pkgsStatic = musl builds I am much more prepared for this challenge
<samueldr> I'm mildly confused, autoPatchelfHook is failing with `readelf: command not found`
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-dev
ris has joined #nixos-dev
FRidh has quit [Quit: Konversation terminated!]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 256 seconds]
orivej_ has joined #nixos-dev
orivej_ has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-dev
gilligan has quit [Ping timeout: 260 seconds]
gilligan has joined #nixos-dev
spacekookie has quit [Quit: No Ping reply in 60 seconds.]
spacekookie has joined #nixos-dev
alp has quit [Ping timeout: 272 seconds]
orivej_ has joined #nixos-dev
orivej has quit [Ping timeout: 240 seconds]
<hoverbear> Is there a tool to turn a derivation into a tarball simply?
<hoverbear> I'm thinking like `pkgs.gimmeATarballOfIt pkgs.ripgrep`
orivej_ has quit [Ping timeout: 265 seconds]
orivej has joined #nixos-dev
julm has quit [Quit: migration shorewall -> nftables]
<gchristensen> hoverbear: the tricky bit is pkgs.ripgrep is not a complete and valid closure, so the value is a bit limited: it depends on libunistring, libidn2, and glibc. so then there is a question as to what such a builder should do with the closure
<gchristensen> hoverbear: this doesn't apply so much for your build since it is a fully static build with a closure of a single path (you might validate this with nix-store --query --requisites ./your-output-path) but that isn't the general case in Nixpkgs
<hoverbear> gchristensen So could I package like, it and all it's deps into a tarball?
<hoverbear> Like it's "closure" I guess is the right word?
<qyliss> Have you seen nix-app?
<qyliss> I think that's what it's called
<hoverbear> Hm no
<qyliss> Or maybe nix-bundle
<qyliss> Hang on
<qyliss> I can't remember the name
<hoverbear> I saw nix-bundle!
<gchristensen> I was just looking for that, yeah! nix-bunlde can do this in a principled way
<hoverbear> Hmmm once again it's a non-`nix` file based interm step that I'd have to do outside of a nix build Q_Q
julm has joined #nixos-dev
<hoverbear> Okay so given that, I need to figure out how to get my nix derivation to have service files etc....
<gchristensen> I assume it doesn't inherently require that
<qyliss> You could probably implement this sort of thing as a Nix derivation but I don't know of anybody having done that
<qyliss> What are you trying to do with this, hoverbear?
<gchristensen> hoverbear: if it were me, I would create a derivation whose out path looks like a mini FHS, and then `tar` that and call it a day. that means putting the service files in the normal places (other than the surprising prefix which is removed at the tar step)
__monty__ has quit [Quit: leaving]
<hoverbear> qyliss: I'm packaging and disitributing a binary for a dozen or so platforms
alp has joined #nixos-dev
<hoverbear> Needs to be rpm/deb/tar.gz as well as mac/windows/freebsd/nixos compatible
<hoverbear> I was kind of under the impression "turn a nix derivation into a deb/rpm/tarball" was a very solved issue
<samueldr> oh, for the record, I fixed my missing readelf by using `buildPackages.stdenv` rather than stdenv, as it was to patchelf a binary for the host and not the target
<samueldr> (hi logs reader from the future!)
<hoverbear> samueldr! I did the same thing just this morning
<qyliss> I don't think it's solved at all, unfortunately.
<hoverbear> Dangit
<qyliss> It's because of having to package up all your dependencies
<hoverbear> But nix knows all these?
<gchristensen> this is because there isn't a principled route from a closure to a clean RPM which can coexist with other RPMs
<qyliss> It would be a bad experience for a Debian user if they had to install hundreds of Nix packages
<gchristensen> you would practically need one RPM per store path
<hoverbear> Hm ok
<hoverbear> But you said if I had a static binary it's ok?
<gchristensen> (and then an additional RPM for linking /usr/bin/foo to /nix/store/hash-foo/bin/foo)
<qyliss> Well then you don't need to install lots of dependencies somehow
<gchristensen> if you have a static binary you are still providing one RPM per store path, it just so happens that you only have one store path you need to worry about -- and it doesn't refer to anything else, and so it can be relocated elsewhere
<hoverbear> I'm fine with static packages as long as I can depend on glibc
justanotheruser has quit [Ping timeout: 265 seconds]
<hoverbear> (We noted a 20% perf different between static glibc and musl)
<qyliss> depend on glibc where though? the Nix glibc in /nix/store, or the Debian (etc) glibc?
<hoverbear> The debian one
<hoverbear> Actually ideally I could depend on any glibc over 2.17 (packaged on centos7)
<hoverbear> Which is what I get when I just run `cargo build`
<hoverbear> (From centos7)
<qyliss> You could patchelf a binary built with Nix, and replace /nix/store/...-glibc-... with /usr/lib/ or whatever it is
<qyliss> But you'd have no guarantee that would be compatible with your binary.
<qyliss> (These aren't Nix-specific problems, just ones that are more apparent to us because Nix usually solves them)
<hoverbear> Can I make nix use glibc 2.17?
<hoverbear> Because I know all newer glibcs are compatible with it
tokudan has quit [Remote host closed the connection]
<qyliss> If you package it
<hoverbear> glibc 2.17 is not packaged?
<qyliss> I'd be very surprised
<qyliss> How old is it?
<qyliss> You might be able to find it packaged from an ancient Nixpkgs and import it
<hoverbear> It's about 7 year sold
<gchristensen> it isn't even listed here and it goes back to 2013 :x
<hoverbear> And it's what every single RHEL installation on the planet uses
<qyliss> We don't generally carry old versions in nixpkgs unless there's a good reason to
<hoverbear> Hm.
<gchristensen> hoverbear: out of curiosity, why do you need to link to glibc but also do static builds?
<qyliss> And I don't think there's really any reason to have a 7 year old glibc
<hoverbear> gchristensen You can't static link glibc in a asl2 package
<hoverbear> It's not allowed
<gchristensen> and musl won't work?
<qyliss> gchristensen: static builds for ease of distribution, glibc is required because it's faster than musl, glibc can't be static linked AIUI
<hoverbear> gchristensen 20% perf diff
<gchristensen> yowz
<hoverbear> So I'm free to use whatever OS provided glibc but I do need to use the OS provided one
tokudan has joined #nixos-dev
<qyliss> asl2 being the license?
<hoverbear> Yes apache
<qyliss> How come? asl2 with incompatible with the gpl2, but glibc (I assume) uses the gpl3 now, which is compatible with asl2
<hoverbear> qyliss: I mean if I can get it to work and my license auditor doesn't fire it's fine
<gchristensen> says LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
<gchristensen> so gplv2+ means you can pick gpl3 I think
<hoverbear> Hmmm
* gchristensen is not a lawyer and this is not legal advice
<qyliss> For GNU software it is surprisingly unclear to me from the glibc website what the license is
<gchristensen> I'd put money on centos getting it right
<qyliss> LGPL3 should be compatible with asl2
<hoverbear> So I know from working at the CNCF we were Apache 2.0 and our builds were only linked against glibc and that was because it was basically impossible to statically link it
<qyliss> It is technically very difficult to statically link glibc
<qyliss> Which will likely preclude the legalities
<hoverbear> Hm ok
<qyliss> I think it's basically not possible
<qyliss> But you could just provide in your tarball or whatever
<hoverbear> So can we talk about how I could use my new fancy nix derivations to build centos/debian packages and tarballs? :)
<hoverbear> qyliss: Also generally taboo
<qyliss> It sounds like, given your constraints, the thing to do would be to package your ancient glibc, use it in the stdenv of your Nix derivation, patchelf the binary to use the libc path on the target operating system, and then create your tarball
julm has quit [Quit: migration to hardened profile]
<hoverbear> Yeah it sounds like it.
<qyliss> or deb or rpm or whatever
<samueldr> could the build work against a binary blob of that glibc?
<qyliss> Yeah you could do that too
<samueldr> like, pick the centos glibc
<samueldr> patchelf it (if needed?) for the build
<hoverbear> I wonder how hard it is to package glibc...
<gchristensen> what would it be like to build against 2.27 and run against 2.17?
<qyliss> gchristensen: probably wouldn't work
<qyliss> AIUI you only get forward compatibility
<hoverbear> gchristensen: It's only forwards compat
<qyliss> hoverbear: using the CentOS binary as samueldr suggested is probably the way to go
<hoverbear> This is why we used musl for a long long time
<qyliss> You could just fetchurl the rpm and pull it out of there
<hoverbear> Oh yes that'd be nice and slick
<hoverbear> So I'd need to override the prebuild of my `buildRustPackage`?
<samueldr> if the intent was to run on nixos, that'd taint the build, but there you're treating it as kind of an "ABI cheat sheet"
<qyliss> hoverbear: I think you'd actually want to override the stdenv.
justanotheruser has joined #nixos-dev
<qyliss> Because that's (AIUI) where libc comes from
<hoverbear> Yeah :S It's a big bummer
<qyliss> Pretty much all Nix stuff is built around things being Nix all the way down.
<qyliss> Building packages for "foreign" operating systems using their libraries is sort of contrary to the model, so it's going to be a bit of a struggle. But definitely doable! And probably still nicer than not-Nix, especially once it works.
<samueldr> qyliss: I would have thought the same, but for Nixpkgs instead of Nix
<samueldr> Nix is not *that* opinionated compared to Nixpkgs, right?
<qyliss> Yeah I say "Nix stuff" sort of handwavingly
<samueldr> :)
<samueldr> good, checking if I was missing something
<hoverbear> Yeah I think the glibc problem is the only really big one
<samueldr> though there's not really a non-Nixpkgs Nix "stdenv"
<hoverbear> I can do rpm builds etc without nix if I have to
<qyliss> samueldr: well, you can override the compiler or the linker in stdenv fairly easily
<samueldr> yeah
<qyliss> I imagine you could do the same with libc?
<qyliss> Although there won't be a convenient function like overrideCC for it
julm has joined #nixos-dev
<hoverbear> I will probably need to wrench my way through this for a few hours ...
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-dev
<qyliss> hoverbear: having looked at it for a few minutes (so I could be way off), what I think you'd want to do is make a stdenv like:
<qyliss> overrideCC stdenv ( { bintools = { libc = ...; }; }
<hoverbear> Oh wow that's probably about 6 hours of digging for me
<hoverbear> Thank you
<qyliss> You'd want to put your custom glibc in there, and try to make it match the nixpkgs glibc package in terms of paths. It'll probably also require some patchelf-ing.
<hoverbear> I was still at "Where is my stdenv"
<qyliss> Interesting files: pkgs/stdenv/adapters.nix, pkgs/stdenv/linux/default.nix
<qyliss> I hope you write this up if you get it to work
<qyliss> Then you should be able to override buildRustPackage to use that stdenv
<hoverbear> I am trying to structure this so you would be able to drop this into any rust project, configure the build deps, and build for any cross target a rust package
<hoverbear> Trying to read from cargo.toml etc wher eI can etc :)
<hoverbear> And I absolutely plan to blog/doc this all once I have sorted it
<qyliss> Awesome :)
<hoverbear> But first I need to make sure the persons paying me to do this are happy with the outcome =D
<hoverbear> So would I override the stdenv somewhere around my `makeRustPlatform` call?
<hoverbear> Or would I do that like in my `pkgs` import?
<hoverbear> qyliss: Also oh hi I remember you from ... Rustfest a few years ago. :)
<hoverbear> Paris right? We got lego?
<qyliss> Oh good you remembered!
<qyliss> We did!
<hoverbear> Lol of course I do
<qyliss> I was wondering if you remebered
<{^_^}> nix#3444 (by yorickvP, 13 weeks ago, open): No GC root to EvalState.baseEnv
<hoverbear> Oh no :(
<puck> and in a very reliable way which is the worst part
<hoverbear> qyliss: Yeah I still have that lego
<puck> GC was a mistake
<hoverbear> Agreed
<qyliss> hoverbear: do you see in pkgs/development/compilers/rust/default.nix, where it takes the stdenv argument?
<hoverbear> qyliss: Do you want a picture from then? :)
<qyliss> Oh yeah if you've got one!
<hoverbear> Oh I didn't look yet Iwas scrolling back to paris lol
<qyliss> Can you email it to me? That way I'm unlikely to lose it
<hoverbear> Yes of course
<hoverbear> qyliss: It'll be a gphotos link so you can just download it if you want
<qyliss> Cool
<qyliss> So I _think_ if you do stdenv = ... where you override rust in your makeRustPlatform call, that should do it
<qyliss> But I'm not 100% on that because there's so much indirection
<qyliss> Although, hmm, that would mean building Rust with your custom libc, which might not work
<hoverbear> I think that might work...
<hoverbear> Hmmmm
<qyliss> Ideally rustPlatform would be overrideable, but it isn't
<hoverbear> Yeah I'm having mixed feelings about all the rust build helpers
<qyliss> We might want to make rustPlatform overrideable?
<qyliss> I think I'd try hacking a lib.makeOverrideable in on line 15 of pkgs/development/compilers/rust/default.nix
<hoverbear> Ohhh all of these ideas are good
<qyliss> And then I think you should be able to override the result of makeRustPlatform
<puck> niksnut: please run all tests with GC_INCREMENTAL=1
<qyliss> If that works, we can see if we can make that change in Nixpkgs
<puck> niksnut: err, GC_ENABLE_INCREMENTAL=1
<puck> basically none of the tests fail
<puck> basically none of the tests pass*
<abathur> hit a libc issue with missing symbols building tintin on macOS and this conversation made me wonder if it'll be painfully obvious to anyone what's wrong? :]
<{^_^}> #90273 (by abathur, 1 week ago, open): tintin failing to build on darwin
<hoverbear> qyliss: Thanks for all these ideas. I need to put this down for a bit though. Been at it all day xD
<puck> gods my brain is mush right now after debugging a bug in what i thought was my nix code but was instead the garbage collector kicking in
<qyliss> abathur: I had a quick look at that, and my instinct is that it'll be because Nixpkgs' Darwin stuff usually lags behind a bit
<qyliss> 10.13 is IIRC pretty recent, so we might not have those yet
<qyliss> I believe it comes down partially to upgrades being hard and partially to Apple being slow with releasing sources?
<qyliss> hoverbear: well, hope it helps ):
<qyliss> *:)
<hoverbear> qyliss: It really does =D I just need to eat nomnomnom
<qyliss> abathur: It has been a while since I used Darwin though, so my knowledge an that might be outdated
<abathur> qyliss: I think I noticed, at least from looking for security code a few weeks ago, at least a 6-month lag
<qyliss> It used to be on the order of years I think
<abathur> but I don't know how stable that is across different types of code
<qyliss> Like, I think we'd be a couple of major releases behind?
<abathur> I'm not super familiar with how they release, but 10.15.3 at least has a release up,
<abathur> but I won't pretend to understand if all of the components in that are actually up to date, or how it works, and how the developer tools releases relate
hoverbear has quit [Quit: Connection closed]
<abathur> qyliss: but your intuition seems to be dead on wrt to the ~cause; I see 10.12 in
<qyliss> Hmm, seems i'm doing well at helping people with my intuition today then :)
<qyliss> Which is very good because I felt like I hadn't accomplished anything all day
<abathur> <3 qyliss
<{^_^}> qyliss's karma got increased to 69
urkk has joined #nixos-dev
urkk has left #nixos-dev [#nixos-dev]
<qyliss> nice
<abathur> made me blush
<{^_^}> #73744 (by dredozubov, 31 weeks ago, closed): darwin.apple_sdk: 10.12 -> 10.13 update
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-dev
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #nixos-dev