<infinisil>
Looking good to me, you get my vote in the imaginary voting system :)
<worldofpeace>
Like pick a name out of a hat :D
drakonis has quit [Quit: WeeChat 2.2]
drakonis has joined #nixos-dev
<yl[m]>
infinisil++
<{^_^}>
infinisil's karma got increased to 37
<yl[m]>
:)
drakonis1 has quit [Ping timeout: 245 seconds]
<yl[m]>
I was wondering about the top-level/all-packages is there any planned work for refactor?
<yl[m]>
it's getting intense to even edit it in vim
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 252 seconds]
sir_guy_carleton has joined #nixos-dev
orivej has quit [Ping timeout: 268 seconds]
Lisanna has quit [Quit: Lisanna]
lopsided98_ has quit [Ping timeout: 264 seconds]
lopsided98 has joined #nixos-dev
sir_guy_carleton has quit [Ping timeout: 240 seconds]
sir_guy_carleton has joined #nixos-dev
sir_guy_carleton has quit [Client Quit]
Lisanna has joined #nixos-dev
Lisanna has quit [Quit: Lisanna]
Cale has quit [Remote host closed the connection]
worldofpeace has quit [Ping timeout: 252 seconds]
__Sander__ has joined #nixos-dev
orivej has joined #nixos-dev
primeos has quit [Ping timeout: 252 seconds]
primeos has joined #nixos-dev
FRidh has joined #nixos-dev
<Profpatsch>
yl[m]: It works fine in emacs if you disable the syntax highlighting.
init_6 has joined #nixos-dev
vasarmilan has joined #nixos-dev
<arianvp>
Morning
<arianvp>
niksnut: could you perhaps review the nscd pull request? Given you were the original person to come up with the hack. I wanna make sure I'm not breaking anything
<arianvp>
Since it's something that literally touches everything that interacts with glibc it makes me a bit nervoust
<arianvp>
:p
LnL has quit [Max SendQ exceeded]
LnL has joined #nixos-dev
fadenb has quit [Remote host closed the connection]
tv has quit [Read error: Connection reset by peer]
tv has joined #nixos-dev
<yl[m]>
Profpatsch: that being true, it's still going to be an issue at scale, can you imagine how it would look like in 5 years?
<Profpatsch>
yl[m]: No idea; we’re swiftly approaching the upper limit of existing useful packages I suspect. ;)
<Profpatsch>
Maybe factor 3 of the current size?
<Profpatsch>
Editors should be capable of editing 100k+ line files, and if they aren’t they should be *made* capable.
<yl[m]>
Profpatsch: it’s going to be hard to determine what’s useful and what’s not besides it’s pretty useful to get access to a host of pkgs from one trusted source
<yl[m]>
but somehow keep them under the nixpkgs channel
<Profpatsch>
Dividing stuff by arbitrary labels never helped anyone …
<yl[m]>
I think that's up for debate :) Personally I'm a big fan of mono-repo, and I lead the move at work. But I feel that nixpkgs is a bit overwhelming at first and it took me a while to grasp
<yl[m]>
I've been thinking, what if every folder that's under `pkgs/` would contain a default.nix providing all the packages that's underneath it and shorten down the all-packages to simply import all the folders individually
<yl[m]>
at least packages would be together in some order
drakonis has quit [Quit: WeeChat 2.2]
orivej has quit [Ping timeout: 252 seconds]
__Sander__ has quit [Quit: Konversation terminated!]
<samueldr>
I think right now the most likely move to get something done about the all-packages.nix file is to make a PR, cunninham's law applies here, partially
<yl[m]>
I don't know about that. I believe the best solutions are the fruit of a long and tedious discussion (or you know wake up with one lol)
<Profpatsch>
Problem is you won’t have much fun, because you’ll get multiple merge conflicts per day.
<gchristensen>
yeah that is going to be a real unpleasent migration
<samueldr>
were I to make this, I would make it through a script specifically for this reason :)
<samueldr>
(though it would be hard)
<samueldr>
probably need some AST manipulation, something JD[...]'s rust parser could help with (thinking out loud)
<Profpatsch>
not a chance
<gchristensen>
a few at a time, maybe
jtojnar has joined #nixos-dev
FRidh has quit [Quit: Konversation terminated!]
Haskellfant has joined #nixos-dev
<infinisil>
Yeah, slow transition, but i think it's doable
cocreature has quit [Remote host closed the connection]
Haskellfant is now known as cocreature
<Profpatsch>
infinisil: The question is: to where.
<Profpatsch>
I don’t see a better scheme than we have right now. Something that is as easily greppable.
<Profpatsch>
If there is a better scheme, the advantages are way too small for the amount of work.
cocreature has quit [Ping timeout: 250 seconds]
andi- has quit [Ping timeout: 250 seconds]
cocreature has joined #nixos-dev
andi- has joined #nixos-dev
drakonis_ has joined #nixos-dev
fadenb has joined #nixos-dev
drakonis has quit [Read error: Connection reset by peer]
drakonis_ has quit [Client Quit]
drakonis has joined #nixos-dev
<infinisil>
Profpatsch: I'm thinking: Have a single directory with all packages in it, a single subdirectory or file for each package, with the filename/dirname as its attrname
<infinisil>
Maybe also use subdirectories for the first 2 letters though, so `hello` would be in `pkgs/he/hello`
<infinisil>
In a flat structure however, queries for attribute would still be hella fast (it's just a directory listing)
<infinisil>
This also leads the way to us not having to edit multiple files for adding a package
<infinisil>
A bit of a question is how the calling of those files should happen, because in `all-packages.nix` we have all those `callPackage ./file { ... }` which don't always have no arguments
<samueldr>
don't forget multiple calls to the same package file, with differing arguments
<Profpatsch>
infinisil: Oh god.
<Profpatsch>
Please no.
<infinisil>
Profpatsch: What's so bad about that?
<Profpatsch>
So we replace a file with 100k lines with a directory with 100k files? And then we manuall implement a prefix scheme “optimization”?
<infinisil>
Profpatsch: No, there are about 9000 top level attributes, all these get turned into files/directories
<infinisil>
Well most
drakonis_ has joined #nixos-dev
<infinisil>
yl[m]: The problem with these categories is that there is no specific one you can assign to every package. And if users need to know in which category a package is in order to get their attr path, that doesn't seem very good
<infinisil>
With such a flat structure, we can get away from these arbitrary categories, and use a tag based system instead
<infinisil>
E.g. for weechat: `meta.tags = [ "networking" "console" "IRC" ]`
drakonis has quit [Ping timeout: 264 seconds]
<infinisil>
Profpatsch: I don't see any major downside to this, assuming we can figure out the file calling thing
drakonis has joined #nixos-dev
<domenkozar>
I think it would be cool to just have one folder structure
<domenkozar>
so pkgs/mypackage/default.nix
<domenkozar>
but pkgs/haskell/*
<domenkozar>
the problem is that some packages have the same name, but have a different purpose
drakonis_ has quit [Ping timeout: 240 seconds]
<Profpatsch>
infinisil: You don’t see any downside in 9k files?
<Profpatsch>
*additional files
<infinisil>
domenkozar: That makes me think: How about separating standalone packages, and package sets. We could have a standard way to create package sets and put them into nixpkgs/pkgsets/<pkgset>/
<infinisil>
Profpatsch: No? Is there?
drakonis_ has joined #nixos-dev
<Profpatsch>
Every good aspect of one all-packages.nix is gone.
<Profpatsch>
You add one more file for every single invocation.
<Profpatsch>
All for dubious cause (file is too big for your editor/somewhere in the future it is going to be)
<infinisil>
Profpatsch: Evaluating nixpkgs currently also involves evaluating a whole bunch of files?
phreedom has joined #nixos-dev
<infinisil>
I'm pretty sure evaluation time will stay the same or get better (because all-packages.nix doesn't have to be parsed)
orivej has joined #nixos-dev
<Profpatsch>
It has nothing to do with evaluation time.
drakonis has quit [Ping timeout: 264 seconds]
<infinisil>
Profpatsch: No, also: no more merge conflicts for adding packages, no more arbitrary categories, and it's generally really hard to know where to place stuff in all-packages.nix, it's a mess, this fixes it
<Profpatsch>
So the arbitrary sorting is what bothers you?
<Profpatsch>
I still don’t know what pressing reason there is for restructuring everything.
<infinisil>
Profpatsch: Yes that's a problem too. But can you tell me the actual downside to this idea now? You've never said what's so bad about so many files
<Profpatsch>
Apart from “uuuh, the file is too big and badly sorted”
<Profpatsch>
infinisil: Well, simple.
<Profpatsch>
First: It’s going to be many hours of work to restructure it.
<Profpatsch>
Second: There is no pressing reason to invest many hours of maintainer time.
<infinisil>
That's not a problem with the idea and you still didn't explain what's so bad about the many files
<infinisil>
Actually, we wouldn't even get many new files
<Profpatsch>
infinisil: Well, you listed the problem that you have no idea how to call packages.
<infinisil>
Because we have a file for every package already
<Profpatsch>
Now you’re just replacing a semi-magical thing (callPackage) with a completely magic invocation.
<Profpatsch>
Explicitely listing all attributes is a *good thing*.
<infinisil>
I didn't mention anything explicit
<Profpatsch>
First, you can grep for it.
<infinisil>
We could maybe even remove the callPackage magic
drakonis has joined #nixos-dev
<Profpatsch>
infinisil: I can’t tell you the downsides of your method, because I don’t understand it in full.
<infinisil>
Profpatsch: We switch from `grep <foo> all-packages.nix` to `ls | grep <foo>`
drakonis_ has quit [Ping timeout: 245 seconds]
<Profpatsch>
I can just tell you that I think there is no reason to spend many hours restructuring that has more than constant gains.
<infinisil>
"constant gains"?
<Profpatsch>
Stuff like “where do I put my package invocation” and “how do I pass the right attributes”
<Profpatsch>
If you have a good idea & migration plan, you are of course free to create an PoC and an RFC.
<Profpatsch>
All I can tell you is that I think it’s not worth the effort. Maybe a PoC changes that opinion.
<yl[m]>
infinisil: I did not suggest putting it under applications, I was just making linter happy. My overall idea is to split the top-level into the categories we have today (as in applications, development etc..) but in a transparent way to the user (you still access it via nixpkgs.<attr>)
<infinisil>
Okay so the only real reason I got is that it's a scary change, yes it is. Automating the bulk of it would be viable though
<yl[m]>
it is a scary change
drakonis_ has joined #nixos-dev
<infinisil>
However any idea that tries to fix all-packages will be scary, I don't think there's a way around it
<yl[m]>
but I think it would help us scale the nixpkgs while keeping it the same format (mono-repo)
<yl[m]>
and would allow us to divide it further if there's a need later
<yl[m]>
it can be done with the idea of no rebuilds whatsoever
<yl[m]>
anyway gtg for a meeting but I would love to discuss this further
drakonis1 has joined #nixos-dev
drakonis has quit [Ping timeout: 252 seconds]
drakonis_ has quit [Ping timeout: 252 seconds]
drakonis1 has quit [Read error: Connection reset by peer]
JosW has joined #nixos-dev
drakonis1 has joined #nixos-dev
phreedom has quit [Ping timeout: 256 seconds]
drakonis has joined #nixos-dev
drakonis1 has quit [Read error: Connection reset by peer]
phreedom has joined #nixos-dev
<infinisil>
yl[m]: Glad somebody agrees :P. But yeah Profpatsch, I know what you mean and I agree, the gain from doing this big refactor isn't very big. But I think it will make for a better future, and the sooner we fix such issues, the less people have to be bothered with it, and the smaller is the refactor gonna be.
<samueldr>
while the argument is dubiously good, it still exists: github can't cope with the huge file, and as long as it's hosted there, it would be nice to allow linking to lines directly
<gchristensen>
let's just get browsers to handle that big of a file better, then get github to show bigger files
<LnL>
what about git blame pkgs/top-level/all-packages.nix
<samueldr>
easy: it's everyone's fault
worldofpeace has joined #nixos-dev
<samueldr>
in reality: that's a point
<samueldr>
also, even though vim handles it fine here on my workstation, on a puny allwinner CPU, it's not the best experience
drakonis_ has joined #nixos-dev
<LnL>
yeah it works, but ...
<LnL>
if people would stop putting overrides there and use default arguments I wouldn't need it that much anymore :p
drakonis has quit [Ping timeout: 244 seconds]
<LnL>
for packages I like the idea of triton, just split up everything alphabetically
<LnL>
I was referring more to the intermediate file with all "a" callPackage definitions
<gchristensen>
yeah, I know :) what about python packages though?
<gchristensen>
(I also like Triton's method, but am also unsure about a migration strategy)
<LnL>
large packages sets like python are already defined outside of all-packages, but depending on the size we could do the same
<infinisil>
If we have some nice way to organize packages as attributes in files, it might be pretty easy and natural to use it everywhere.
<infinisil>
I never liked having to both define the package file and the attribute
<infinisil>
I mean, python-packages.nix is also quite big.. 5000 lines
<LnL>
the alternative would be to go the haskell direction, nobody cares there because humans don't touch it
<infinisil>
Considering most of it is just `<name> = callPackage ../development/python-modules/<name> { };` it sure feels a bit useless
<infinisil>
LnL: I've thought about this before: How about create an ultimate nix packaging app, that just takes a source, and spits out a nix expression to get a build. Would need lots and lots of special cases. It could try to build it and respond smartly to the error it gets, reiterating over and over again.
<infinisil>
s/create/creating
<infinisil>
Sounds cool and useful, but would be hella hard
<LnL>
I'm talking about just all-packages.nix
drakonis has joined #nixos-dev
<infinisil>
LnL: Oh, you just mean to auto-generate all-packages.nix by it going through all folders in nixpkgs/pkgs or so? That sounds pretty neat for a start
<gchristensen>
infinisil: how about a nix which can build A and B in parallel, even if B depends upon A: by watching stat() open() opendir() etc. calls and blocking until they exist :=)
<infinisil>
gchristensen: Yeah, had this idea too before, that would be hella cool!
<gchristensen>
(not really possible, but it would be a fun terrible-ideas hackathon project)
<infinisil>
And applying it to everything could lead to binaries that would dynamically fetch dependencies!
<LnL>
half of the exceptions in all-packages could be defined somewhere else and the other half shouldn't be there in the first place
<gchristensen>
oh dear
drakonis_ has quit [Ping timeout: 244 seconds]
<gchristensen>
infinisil: a few international flights ago I wrote a little thing to crawl my ~ for shell.nix and default.nix files, and it fetched the build-time/run-time closure of all of them -- I considered making it run every channel change -- but it got a bit wacky.
<LnL>
infinisil: easy, clone nixpkgs and copy the expression for the package you want from there ;)
<infinisil>
gchristensen: Neato
<infinisil>
LnL: Wait, what are you referring to?
<LnL>
the ultimate nix packaging app
<LnL>
kidding aside, I wonder what percentage of packages you'd be able to build by passing file not found errors to nix-locate
<clever>
LnL: how many things will break because busybox includes everything? :P
<LnL>
darnit, grep -v busybox
<clever>
ive seen at least 2 users break nixos hard, by installing busybox in systemPackages
coconnor has quit [Remote host closed the connection]
<yl[m]>
I'm back. I like the idea of getting rid of top-level in favor of alphabetically sorted list of folders. The obvious question is how to deal with overrides?
<yl[m]>
would the override within the package itself make sense?
<yl[m]>
take `buildGoPackage` for instance. Some packages (such as perkeep that I maintain) requires a specific version of Go, so the override to which version of go it needs totally makes sense to live within the nix expression itself
<yl[m]>
this could also be argued that it would make it easier for people to extract the expressions they need from nixpkgs to whatever project they are using without having to eval nixpkgs all together
<infinisil>
yl[m]: Oh, I've wanted this before, how about we stop using different attributes for different versions and instead use a subattribute instead. So instead of having `foo` and `foo_1`, we'd have `foo` and `foo.version "1.0"` or so
<yl[m]>
anyway back to the subject of commit access, thx infinisil for your vote of confidence and I would still like to get those rights. While we figure out the overall process of on-boarding volunteers, who should I talk to for my behalf? I don't really want to burden edolstra with this as I'm sure he has a better use of his time.
<yl[m]>
infinisil: that's interesting
<yl[m]>
I like much better than a global attribute that would pollute the global namespace anyway.
<yl[m]>
terraform does this with the full versus withPlugins and I think it works quite well
<yl[m]>
that's quite a big refactor though and it probably should be taken step by step. probably best not to move packages around, just remove the attributes out of all-packages and move the overrides inside the expressions as a phase 1
<yl[m]>
to ease the folder migration and make way for the passthru attributes to access any given version
<infinisil>
yl[m]: garbas and domenkozar also seem to have the capability to give commit access, maybe ask them
<infinisil>
yl[m]: Yeah, would be cool to explore this way of doing it in a PR/issue
<infinisil>
s/cool/interesting
<yl[m]>
I would start by ripping out all packages with no override, should be quickly doable with a script
<infinisil>
Possibly
<gchristensen>
the really hellish part is (1) existing PRs (2) backports
<yl[m]>
for the existing PRs, it would also be doable to fix any conflicts with a script (assuming they have kept the 'Allow edits by maintainer' checkbox)
<yl[m]>
but yea, backports is going to be a bit challenging especially for the derivations with overrides
<yl[m]>
but the question we need to ask ourselves. This is a technical debt that we have to pay sooner or later, the question is when are we going to pay it?
<yl[m]>
the longer we wait, the worse it's going to get
<gchristensen>
at 10s of thousands of pkgs I'm not sure that holds much urgency
<yl[m]>
gchristensen: with the amount of work planned, probably not. But it's a task that someone like me could handle without affecting the overall output of the core team, do you not agree?
<gchristensen>
I don't know, I think it is likely to be a majorly disruptive change, and require collaboration and coordination with a lot of sort of sub-groups of the ecosystem
<gchristensen>
(and to be clear, I'm not saying it is bad or undesirable)
<yl[m]>
it's definitely going to a challenge to get through this
<gchristensen>
ekleog: any chance you can debug / patch ofborg's failure to build nixos tests since the nixos test change?
<LnL>
do you have an idea what broke it?
<LnL>
looks similar to the fetchGit error from before