dongcarl has quit [Quit: Ping timeout (120 seconds)]
dongcarl has joined #nixos-dev
<MichaelRaskin>
Wait, bot does not report Nix PRs? «Automatic UID allocation», highlight: «This allows things like systemd-nspawn and NixOS containers to run inside a Nix build.», minimal NixOS test: < 3s nix-store -r total wall-clock time.
<gchristensen>
wow!!! thanks, MichaelRaskin!
<MichaelRaskin>
I am only highlighting (and testing)
<infinisil>
MichaelRaskin: Need to fix that, but I think the karma increase makes it not show the pr
alp has quit [Ping timeout: 260 seconds]
<MichaelRaskin>
Yeah, I was wondering if this is a feature conflict
<gchristensen>
MichaelRaskin: and ++'ing niksnut :P
<MichaelRaskin>
This is a part of highlighting!
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #nixos-dev
alp has joined #nixos-dev
<infinisil>
(Fixed now, ++'ing and PR linking will work with a single message now)
<MichaelRaskin>
Good thing Github usernames and IRC nicknames are dissimilar enough that it makes no sense to ask for a ++ on PR reference to ++ the author
etu1 has joined #nixos-dev
etu has quit [Ping timeout: 246 seconds]
<FireFly>
I mean isn't there technically a kinda-canonical mapping between the two in nixpkgs? :p
<MichaelRaskin>
No, we have name-email-github
<MichaelRaskin>
And GPG
<FireFly>
oh ah, I thought IRC nickname was in there for some reason
<MichaelRaskin>
If it was true, I would ask Alyssa why in the world not to look up via maintainers-list for IRC -> email
<FireFly>
.. oh point
<FireFly>
that little tangent took far too long, the other day >.<
alp has quit [Ping timeout: 260 seconds]
alp has joined #nixos-dev
<infinisil>
I do like the idea of having irc nicknames in the maintainer list
<emily>
+1
<emily>
would also be nice to record commit bit, I don't whether it's intentional that nixpkgs-committers is private to non-maintainers (I understand there's GitHub limitations for the team member list), but it's pretty confusing for contributors to not actually know who can merge PRs and so on
orivej has quit [Ping timeout: 256 seconds]
orivej_ has joined #nixos-dev
etu1 is now known as etu
justanotheruser has joined #nixos-dev
<julm>
emily: the list of people with commit rights on nixpkgs is not public somewhere? :O
<emily>
I don't know of any other list; maybe I'm just not aware of it, but I've seen people talk about "should committers be public" before, so... just throwing in my +1 for that
<LnL>
I think all the teams are private, or is that not the case?
<emily>
non-org members can't see teams in general, it's a github restriction, is my understanding
<adisbladis>
gchristensen: I've had exactly this discussion before on another issue/pr
<gchristensen>
oh yeah?
<adisbladis>
I'll try to find it
<gchristensen>
IMO it is better to go this way first, make everything pythonPackages explicitly python2, and then it creates a natural to-do list of things to port or mark as broken
<gchristensen>
that said, I think fridh's comment there about being explicitly py2 is a good point
<adisbladis>
Indeed
<gchristensen>
I'm not sure I understand your comment there, but I think pkgs.python and pkgs.pythonPackages is a bad idea and moving them only to aliases seems like a good first step of deprecating them
<adisbladis>
gchristensen: I mean that python3Packages doesn't make any sense given that you can use python3.pkgs
<gchristensen>
ah
<gchristensen>
I don't have much opinion on that :)
<adisbladis>
I've seen cases where `python = python3;` is passed together with `pythonPackages` (which by default is python2Packages)
<adisbladis>
So I'd really like to remove python*Packages entirely
<gchristensen>
yeah
<gchristensen>
this is another case of aliasing, which should be moved in to aliases.nix at a minimum
<adisbladis>
Yes, baby steps :)
<adisbladis>
Removing python*Packages is a huge change
<adisbladis>
I wish FRIdh was here right now, I'd really like his input on this
<gchristensen>
me too, maybe it would be worth setting up a specific time to talk about it, maybe even on jitsi. it is a complicated topic, the bandwidth might be useful
<adisbladis>
+1
<emily>
hmm, this prompted me to look at nix why-depends --all /run/current-system nixpkgs#python2... should pypy(3) really depend on python-2.7.18-env? it seems to mostly be shebangs in the standard library
<emily>
(seems pypy (because of icestorm), gimp, and steam are the "only" things keeping py2 on my system at this point, at least)
<adisbladis>
emily: pypy is using python2 at build time
<emily>
yeah but it has runtime dependencies too
<gchristensen>
afaik pypy will always use python2
<adisbladis>
That's weird and shouldn't happen
<adisbladis>
rg 'python\d*Packages' | wc -l
<adisbladis>
2284
<emily>
gchristensen: again, that's build time
<adisbladis>
emily: I'd hate debugging why though :P
<adisbladis>
Pypy build times is a thing of nightmares
<gchristensen>
I guess I'm not sure I understand what happens at pypy's buildtime to make it no longer depend on python2
<emily>
gchristensen: pypy can bootstrap from itself, which is how you'd have a "fully-supported" py2 toolchain in the long run, but again, this is only about runtime dependencies
<emily>
having python2 in the bootstrap chain is one thing, I'm just trying to make python2 not exist in my system closure (eventually)
<adisbladis>
emily: Stop roping me into this
<gchristensen>
whoa! cool :D
<adisbladis>
This rabbit hole is too deep
<gchristensen>
yeah leave adisbladis to me muahaha
<emily>
haha, sorry :p
<adisbladis>
I can't resist a good rabbit hole
<adisbladis>
Btw, withPackages is broken for pypy3 :/
<emily>
it would be good if we could detach pypy from python2 entirely though. that would require bootstrapping from a binary, but pypy is going to be a maintained python 2 "forever" at least in theory
<emily>
so for the python 2 stuff we can't drop it may be valuable
<gchristensen>
there is a co that'd really, really like to support python2 inside nixpkgs for as long as we'll let them
<gchristensen>
so from that perspective, it may not be "expensive" in terms of regular contributor time/effort
<adisbladis>
Despite what PSF wants Python2 is not going away any time soon
<gchristensen>
exactly
<gchristensen>
(unfortunatley)
<adisbladis>
emily: What I _think_ is happening here is that pypy is not producing any binary called python, but only pypy3
<gchristensen>
to that end, they've mentioned an interest in maintaining something like python2.pkgs, when a python dep is updated beyond py2 compatibility, they'd like to be able to "put it back" in a py2 compatible way -- maintained separately
<adisbladis>
And patchShebangs kicks in and incorrectly patches all shebangs with the build-time python2
<adisbladis>
I don't even think patching shebangs in the pypy build makes sense?
<garbas>
gchristensen: i think the pythonPackages -> python2Package move still doesn't solve the search problem i have. you would still have double results. since python3Packages is one of 3.7 / 3.8 / etc...
<gchristensen>
ah, that is even harder because those are in a sense different packages. though maybe you could aggregate them based on the reported location?
<garbas>
maybe just moving this assigmnents to aliases.nix would solve it
<garbas>
one location can be for 2 different platforms
<gchristensen>
oh python3 vs. python37 python38 oooof that is a tricky one!
<emily>
adisbladis: aha
<adisbladis>
emily: I'm gonna try something on a really beefy box
<emily>
adisbladis: you can set it to bootstrap from the binary packages btw
<emily>
which makes it a lot less slow (but obviously changes hash)
<emily>
gchristensen: re py2 in nixpkgs -- sure, my point is that having a py2 impl that's actually maintained and doesn't depend at all on the inevitably-rotting CPython may be valuable for exactly that purpose
<emily>
in the meantime "build dependency on cpython but no runtime dependency" is a lot easier to achieve and gets most of the benefit
<gchristensen>
I get the impression we're talking past each other, and I'm sorrry for that. I hear what you mean now.
<emily>
yeah I don't think we actually disagree, sorry :)
<gchristensen>
:)
<cole-h>
:)
<emily>
I guess "python 2 stuff we can't drop" was accidentally provocative :p
<adisbladis>
If I was XXX I'd just delete python2
<gchristensen>
well and it all started with me not realizing that pypy was actually self-hosting -- I thought it still actually ran the python2 interpreter
<adisbladis>
Just for bootstrap :)
<emily>
I would admittedly rather see it all burn for the most part, but there's a whole lot of stuff in nixpkgs already, so as long as it isn't unnecessarily depended on by external components or actively exposing people to security holes it's whatever
<emily>
gchristensen: you either bring a cpython 2.x or you bring an existing pypy2 binary, basically
<gchristensen>
yeah, that is cool
<gchristensen>
it didn't even occur to me to be _possible_ to self-host it :D
<emily>
so we could bootstrap pypy2 from the pypy2 binaries (already an option) and then bootstrap pypy3 from our pypy2
<gchristensen>
I'm super impressed
<emily>
you think the pypy developers could stand waiting 8 hours for every mandelbrot?!
<gchristensen>
hah
<emily>
the bootstrap is super painful, I don't think anyone but packagers bothers with it ^^
<adisbladis>
emily: At least the compiler output is pretty
<adisbladis>
Tbh pypy is pure magic and fairy dust
<adisbladis>
Forged by wizards
justanotheruser has quit [Ping timeout: 260 seconds]
<adisbladis>
emily: Now for the long wait to see what happens with my pypy3 build
<adisbladis>
I think the fix is a simple as `dontPatchShebangs = true`
orivej has joined #nixos-dev
b42 has joined #nixos-dev
<emily>
I guess ideally we would patch the shebangs to use the installed pypy
<emily>
so you can execute the few standard library modules that act like programs as scripts (after all that's presumably what the shebangs are for in the first place)
<emily>
I wonder what even happens when you run cgi.py
<adisbladis>
emily: The only one that actually works is rot13 :P
<emily>
vital functionality!!!!
<adisbladis>
And you can already execute for example cgi with `pypy3 -m cgi`
<adisbladis>
Which makes more sense than executing something deep inside a lib directory
<adisbladis>
(And on top of that works for anything that has a __main__)
<adisbladis>
In other words: I'm too lazy to fix the shebangs when I think it's sufficient to just skip patching them
<emily>
yeah
<emily>
having FHS paths in packages just makes me twitch but I guess the postFixup would be ugly
justanotheruser has joined #nixos-dev
<adisbladis>
emily: A more serious issue with pypy currently is that a lot of packages are broken
<adisbladis>
Pip builds a wheel which it then tries to install, but bails out because apparently the platform it just produced a wheel for is not in the compatbility list
<julm>
emily: have you seen this closed PR https://github.com/NixOS/rfcs/pull/19 ? it's a more involved proposal, but it brings up the topic of merge-rights
<{^_^}>
rfcs#19 (by moretea, 2 years ago, closed): [RFC 0019] RFC for a maintainers file
dongcarl has quit [Quit: Ping timeout (120 seconds)]
dongcarl has joined #nixos-dev
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-dev
orivej has quit [Quit: No Ping reply in 180 seconds.]
teto has quit [Ping timeout: 260 seconds]
<gchristensen>
recursive nix is a thing already, right?
<srhb>
gchristensen: The PR landed in master in december, yeah.
<gchristensen>
nice
<gchristensen>
I suspect it will make the buildLayeredImages stuff much much nicer when it can be ultimately used
<emily>
guessing that's several RFCs away, some people quite strongly disapprove of (esp. general) recursive Nix...
<Profpatsch>
what can recursive nix do that generalized ifd can’t?
<gchristensen>
it can be not IFD :)
janneke has quit [Remote host closed the connection]
janneke has joined #nixos-dev
<Profpatsch>
So it’s mostly a hack working around the arbitrary restrictions of ifd?
<LnL>
IFD is all evaluation time while recursive nix it all build time?
<Profpatsch>
there’s nothing speaking against making ifd a first-class citizen in the build system, with staged builds that can be fanned out, gced and queried.
<Profpatsch>
maybe both ifd and nix are on the same level of power, but ifd feels conceptually cleaner to me
<Profpatsch>
s/nix/recursive nix/
<gchristensen>
all I can say is if recursive nix exists and IFD is still IFD in all its glory and warts, I'm going to run as fast as I can to recursive nix if it can possibly solve the same problem
<Profpatsch>
they are both about dynamically extending the dependency graph during build, but in the ifd case the package manager knows about both phases
<Profpatsch>
Also it actually composes, because ifd is wholly done on the nix expression/drv level
<emily>
Profpatsch: IFD cannot give us ccache-style behaviour in builds using arbitrary third-party build systems (say GNU make)
<emily>
at least, I don't see any way of making that work
<Profpatsch>
I thought the world agrees on recursive make being a mistake?
<emily>
I don't see what this has to do with recursive make, it's more about shimming out gcc to hash all the inputs.
<MichaelRaskin>
Wait, full recursive Nix, not a ret-cont Nix which is basically IFD in build-time?
<emily>
but careful not to move the goalposts from "generalized IFD can do everything recusive Nix can" to "everything recursive Nix can do over generalized IFD is Bad and you shouldn't want it", which are pretty different claims
<emily>
MichaelRaskin: yes, an experimental real recursive Nix impl is in new enough nix
* gchristensen
suddenly feels not smart enough for this discussion
<Profpatsch>
I think it’s more about being able to reason about the build graph, and I don’t see that happening with recursive nix.
<gchristensen>
it doesn't happen with IFD either
<MichaelRaskin>
Yes, I know it is implemented, I am surprised it is expected to end not a mess
<emily>
gchristensen: (why, not knowing what "ret-cont" is?)
<gchristensen>
emily: just feeling super out of my league (I love this feeling. part of why I stick around.)
<Profpatsch>
gchristensen: yeah, but I feel like it can be done easier via ifd, than with arbitrary recursive nix calls. Maybe I’m wrong.
<Profpatsch>
e.g. nix can detect everything up to the drvs generating more nix expressions
<MichaelRaskin>
Well, a bit of CPS transform promises us that IFD can be used in a horrific way to emulate recursive Nix, more or less
<emily>
I continue to believe that instead of a fixed (instantiate, build) two-stage process, we should unify evaluation and builds and make Nix capable of arbitrary N-stage build processes, which at least unifies IFD and recursive Nix.
<Profpatsch>
That is every part outside of an (import "${somedrv}/foo.nix") is unknown to it until it builds somedrv
<emily>
(you can still argue about whether to restrict things to "ret-cont" in that form, i.e. full recursive Nix is still an additional feature on top)
<gchristensen>
emily: one inevitible thing about that is it creates a very unpredictable experience
<Profpatsch>
But it can tell you what it needs to get there, and parallelize it
<gchristensen>
emily: "ah, only 5 more things! 1 more thing! 10,000 more things!"
<Profpatsch>
emily: yes please
<Profpatsch>
Plus tools to handle that and restrict it
<Profpatsch>
gchristensen: e.g. if you already built somedrv, you can show more of the actual build tree
<emily>
gchristensen: right. we already have this, though, with evaluation vs. build
<emily>
gchristensen: and sometimes we even punt entirely, farming out build steps to humans by making them manually run foo2nix and commit it when things change. being in denial about the existence of these stages doesn't help us manage and control them
ris has joined #nixos-dev
<Profpatsch>
emily: well, but that’s mostly because these 2nix tools are everything but pure functions
<gchristensen>
hehe
<emilazy>
see also: oh no, now we need a Nix store for instantiation (flakes' evaluation cache)
<emilazy>
(matrix.org having a bad day...)
<Profpatsch>
e.g. yarn2nix can be done in an ifd call
<Profpatsch>
provided you already have the lock file
<emily>
making this staging explicit rather than just "stuff that happens as part of evaluation" also means you could build/cache it on hydra, etc.
<Profpatsch>
gchristensen: On CI, if nix could push ifd store paths to the cache, that would be of great help already. At least with post-build hooks it doesn’t happen at the moment
<gchristensen>
weird, post-build hooks push IFD for me
<Profpatsch>
huh, interesting
<gchristensen>
(and actually that was a big motivation for adding that support to nix)
<Profpatsch>
But ifd store paths are not added to the gc tree I think
<emily>
gchristensen: btw, you can do smarter stuff than just "5 things to build - wait no, 20". like, a very basic proposal is to speculatively assume that the dependency graph will remain unchanged past a stage for at least progress reporting (and potentially "start downloading stuff from substituters with the ability to kill it", etc.)
<gchristensen>
they're not
<gchristensen>
emily: oh, true, cool
<gchristensen>
the post-build-hook is executed after every build the Nix (daemon or not) runs
<Profpatsch>
emily: I know about that, but I think people start moving in that specilative heuristic direction way too quickly.
<Profpatsch>
and often correctness goes out the window with the complexity spike it introduces
<emily>
mhm
<MichaelRaskin>
emily: weren't you just wishing for ccache-like tricks?
<Profpatsch>
I mean even the kernel could do something like “these memory pages were loaded the last time around”
<Profpatsch>
but it doesn’t. Because suddenly you have a cache
<emily>
fundamentally, statically-analysable build graphs are great and have a lot of advantages, but in many ways they're difficult to scale, and it's not like we have a fundamentally unstaged system already rather than a fairly ad-hoc evaluation one followed by instantiation
<emily>
(with flakes, there's another "flakes stage" before proper evaluation, I guess)
<Profpatsch>
Maybe that’s why we should take a step back instead of introducing additional layers and heuristics
<Profpatsch>
And get the fundamentals right (CAS, staged ifd)
<emily>
and there's lots of stuff people do where it's morally a dynamic build graph but instead you just manually keep a .nix file updated, or use horrifying fixed-output derivation hacks, or ...
<MichaelRaskin>
The nice part about node2nix expressions in-tree is that they are perfectly unreviewable
<emily>
MichaelRaskin: I think "global ccache in derivations" is a good motivator of general recursive Nix that afaik can't be done with the other solutions, but I'm not sure what you're asking exactly
<gchristensen>
lol MichaelRaskin
<Profpatsch>
I mean fully recursive nix is nice and all, but the first person I see introducing nix calls in his makefiles I’m gonna shoot on sight
<gchristensen>
wow.
<emily>
people do that already, it's time to get murdering
<Profpatsch>
bang bang
<MichaelRaskin>
emily: well, ccache-like stuff means almost truly unguessable build graph where heuristics are likely to backfiree
<Profpatsch>
(we have nix calls in our makefile, but on phony/dev targets
<emily>
MichaelRaskin: not if you do it in a sufficiently principled way (which is admittedly outside the scope of this margin, but I have sketches)
<emily>
I mean, Nix is basically a kind of ccache to start with
<MichaelRaskin>
emily: unfortunately, also requires upstream packaging to work in a principled way
<emily>
MichaelRaskin: I think you could do something pretty robust that tries to fail the build to begin with rather than just caching incorrectly, but it would probably involve hooking into libclang or something
<Profpatsch>
wow, now we’re talking
<emily>
I feel like Nix(pkgs) is, frankly, already in the position of wishing it could hook into the C compiler more than it can
<Profpatsch>
run nix in nix, so you can poison your store while you poison your cache :)
<gchristensen>
to me, part of the point is it is agnostic
<Profpatsch>
emily: I see people arguing about the “c compiler generates headers” or something all the time, but what about that cannot be made static?
<emily>
not sure what headers you're talking about
<Profpatsch>
Just make your build tool output its graph (“make 2 nix”) and convert to drv.
<Profpatsch>
lol right, make can’t output its graph
<Profpatsch>
neither can redo or probably most other build tools
<gchristensen>
it'd be fun to collaborate with michael stapelberg and distri on these things
alp has quit [Ping timeout: 246 seconds]
orivej has joined #nixos-dev
<matthewbauer>
yeah ifd gets garbage collected a lot - there's no references to it in the drv, so nix just thinks it's garbage
<{^_^}>
nix#719 (by wizeman, 4 years ago, open): Nix GC collects derivations used for IFD
<matthewbauer>
i'm hoping to get https://github.com/NixOS/nix/pull/3523 merged, so that you can at least do something like `nix-store --add-result --indirect -r $(nix path-info --eval --recursive ./default.nix)` to at least manually manage give them gcroots
<{^_^}>
nix#3523 (by matthewbauer, 4 weeks ago, open): Add --build and --eval to `nix path-info' command
alp has joined #nixos-dev
dongcarl has quit [Quit: Ping timeout (120 seconds)]