worldofpeace changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | NixOS 20.09 Nightingale ✨ | | | 20.09 RMs: worldofpeace, jonringer |
supersandro2000 has quit [Disconnected by services]
supersandro2000 has joined #nixos-dev
<{^_^}> firing: RootPartitionLowInodes:
rajivr has joined #nixos-dev
jonringer has quit [Remote host closed the connection]
supersandro2000 has quit [Quit: The Lounge -]
supersandro2000 has joined #nixos-dev
teto has quit [Ping timeout: 256 seconds]
orivej has quit [Ping timeout: 240 seconds]
<{^_^}> firing: RootPartitionLowInodes:
Cale has quit [Remote host closed the connection]
pmy has quit [Ping timeout: 265 seconds]
pmy has joined #nixos-dev
AlwaysLivid has quit [Ping timeout: 240 seconds]
justan0theruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 272 seconds]
kalbasit has joined #nixos-dev
kalbasit has quit [Remote host closed the connection]
pmy has quit [Ping timeout: 265 seconds]
pmy has joined #nixos-dev
cole-h has quit [Ping timeout: 264 seconds]
pmy has quit [Ping timeout: 265 seconds]
supersandro2000 has quit [Ping timeout: 264 seconds]
pmy has joined #nixos-dev
supersandro2000 has joined #nixos-dev
justanotheruser has joined #nixos-dev
justan0theruser has quit [Ping timeout: 260 seconds]
srk has quit [Ping timeout: 240 seconds]
tilpner has quit [Remote host closed the connection]
tilpner has joined #nixos-dev
pmy has quit [Ping timeout: 265 seconds]
pmy has joined #nixos-dev
<siraben> supersandro2000: do you think that this means that we should use lib over stdenv.lib?
pmy has quit [Ping timeout: 265 seconds]
pmy has joined #nixos-dev
evils has quit [Quit: leaving]
evils has joined #nixos-dev
srk has joined #nixos-dev
<{^_^}> firing: RootPartitionLowInodes:
<supersandro2000> I see that file the first time
<supersandro2000> great that this literally flew under my radar
<supersandro2000> possible many radars
<supersandro2000> siraben: is there a rendered version of this?
orivej has joined #nixos-dev
<siraben> supersandro2000: it's too new of a commit to be rendered
<siraben> you have to pull the changes in your clone of nixpkgs then render the docs
<supersandro2000> I mean like any version
<supersandro2000> I am not fidiling around with docbook now
<supersandro2000> so easy to find
<siraben> right
<siraben> Honestly I'm not even clear on the distinction between stdenv.lib and lib myself
<siraben> I view lib as utility functions like lib.optionals, lib.concatStringsSep
<siraben> etc
<qyliss> there is no distinction
<qyliss> one is an alias for the other
justanotheruser has quit [Ping timeout: 272 seconds]
<siraben> pinging Profpatsch committed the PR
<siraben> #108834
<{^_^}> (by alyssais, 1 day ago, merged): doc: add "prefer lib over stdenv.lib" convention
<Profpatsch> siraben: lib is a pure library of nix functions that only depends on builtins
<Profpatsch> It’s exposed via pkgs.lib
<Profpatsch> I don’t know why pkgs.stdenv.lib even exists, possibly an artifact from the very early days of nixpkgs
<Profpatsch> lib could even be hosted separately from nixpkgs and it would still work
<V> I pretty much only write out stdenv.lib explicitly when it's the only place I'm using lib in that file (e.g. meta with-clause), and it makes no logical sense to do so as it's not part of stdenv and does not vary between stdenv
<siraben> So we should seek to stop using `stdenv.lib` in packages?
<V> I do not know why stdenv contains it
<Profpatsch> yes
<Profpatsch> you can already do { lib }: in your package header
<Profpatsch> And it just works
<siraben> Right
<siraben> supersandro2000: hm, maybe you want to adjust your reviews accordingly
<siraben> IIRC you recommend stdenv.lib over lib
<Profpatsch> I consider it a minor problem, but certainly we shouldn’t tell people to use stdenv.lib
<siraben> If I did a PR to remove all occurences of stdenv.lib, would that be good?
<siraben> in Nixpkgs
<qyliss> siraben: I'd wait a bit in case anybody has serious
<Profpatsch> There’s also pkgs-lib, which infinisil recently introduced because it it a library but it depends on a few things in nixpkgs.
<qyliss> in case serious objections come up with formalising it in documentation
<Profpatsch> So we opted for splitting it off rather than making lib depend on nixpkgs
<siraben> qyliss: I see
<siraben> Profpatsch: where's pkgs-lib?
<qyliss> but at some point in the future, maybe a couple of months down the line, I think that would be nice
<Profpatsch> qyliss: If we remove any trace of it in nixpkgs it amounts to the same thing no?
<qyliss> siraben: pkgs/pkgs-lib
<qyliss> Profpatsch: I don't understand
<Profpatsch> qyliss: I mean we don’t need to doc the preference if we don’t have any remaining examples
<qyliss> well that's true
<Profpatsch> Because nobody would ever think to use stdenv.lib in that case
<qyliss> but I think that has to be step 2
<Profpatsch> we can even add a trace warning
<Profpatsch> yeah
<Profpatsch> But tbh, why bother
<Profpatsch> It’s not like it’s a semantic problem
<siraben> It's confusing I think to think lib is under stdenv
<qyliss> Profpatsch: one less thing for newcomers to get confused about?
<Profpatsch> true
<siraben> And we can do treewide PRs
<Profpatsch> Oh man, I *love* monorepos :)
<V> lib is just <nixpkgs/lib>. stdenv has nooothing to do with it
<Profpatsch> I hope that doesn’t go away SMILEY
<qyliss> I think we'd have to keep it around, for external code
<V> :)))))))
<qyliss> could add a trace that it's deprecated
<qyliss> but we're getting ahead of ourselves
<qyliss> let's let the doc change settle down first
<siraben> So maybe it should be a more formal discussion heh
<siraben> yeah
* siraben puts sed away
<V> we're definitely getting ahead of ourselves, I've been trying to find where it's set for the last 10 minutes
<Profpatsch> Hm, I don’t see much reason for discussing the doc change
<Profpatsch> It’s a trivial simplification after all.
<Profpatsch> And the sed can come if somebody finds the time
<qyliss> and definitely not for a couple of months
<Profpatsch> I mean the projects lives from people doing small simplifications like these, since it removes accidental complexity and makes new stuff possible
<siraben> Right, one less thing to worry about.
<siraben> Hm it's not entirely non-trivial to make this treewide change since it may involve some AST-sensitive changes
<siraben> I've had to do manual treewide PRs a few times because of lack of meta-programming
<siraben> not entirely trivial*
<qyliss> you don't have to do it all at once
<qyliss> if you can get just the easy cases that's a good start
<siraben> Heck I've seen `let lib = stdenv.lib in `
<qyliss> wtf
<V> there's only 46132 instances of it
<V> ez
<Profpatsch> A trick I’ve found is that if you use emacs and helm you can do a projectwide search with helm, then use helm-edit which opens a buffer with all matches, then use replacements on that buffer, when you are happy hit C-c C-c and it applies everything
<siraben> `ag -l "lib = stdenv\.lib" | wc -l` gives 12
<Profpatsch> Which is a lot more interactive than blind sedding
<siraben> Profpatsch: no way, really? I haven't heard about helm-edit before
<V> we're talking all stdenv\.lib instances
<Profpatsch> siraben: ikr
<V> but literally half of the references are in hackage-packages.nix
<siraben> V: yeah that's just the crazy ones where they let-bind lib
<Profpatsch> V: ah, but then we need to change the tooling that generates that file
<V> see, clearly we should make hackage-packages into a flake, as it will then stop cluttering up the rest of nixpkgs with stuff like that
<qyliss> don't even suggest that sarcastically :P
<Profpatsch> grml
<siraben> grml?
<Profpatsch> I’d rather have the code of the generation tool in nixpkgs itself :P
<V> grml.
* V contemplates making a haskell joke
<siraben> monorepos make git-bisect so OP
* V decides against it
<Profpatsch> siraben: German way to signify a combined “grrr” and *snort*
<siraben> Profpatsch: ah
<siraben> 11804 files where there is at least one occurence of `stdenv.lib`
<V> siraben: I wouldn't seriously suggest making nixpkgs even more flaky
<siraben> How would this be automated? Obviously replacing stdenv.lib with lib is the easy part, then add `lib` to the function inputs?
<V> nah
<siraben> V: What's a better way?
<V> there's specific usage patterns; you run through each of the appropriate replacements in reverse order of their generality
<siraben> Right.
<supersandro2000> siraben: if you pay attention I stopped caring about lib stdenv in the last weeks
<siraben> supersandro2000: oh, that's good to hear
<supersandro2000> every other person has a different opinion about it and I just have better things to do with my time to be honest
<supersandro2000> like fixing nokogiri 1.11.1
<V> supersandro2000: it's not a matter of opinion, lib isn't part of stdenv
<Profpatsch> But also not worth a fight over *shrug*
<V> if I stuck, say, an alias to youtube-dl in stdenv, it wouldn't make it the canonical way to refer to it :p
<siraben> The whole point of course is to make there be one true way to use lib
<qyliss> V: we could do { youtube-dl, ... }: and then use youtube-dl.stdenv.mkDerivation in every package :P
<V> qyliss: you're not allowed to do that, that only applies to changes made 10 years ago
<qyliss> > youtube-dl.stdenv
<{^_^}> "<derivation /nix/store/5g153v8pyp35swf8k1nq1zcbnk4yw3zb-stdenv-linux.drv>"
<supersandro2000> V: I couldn't care less what is supposed to be part of what because it is just there and convenient
<qyliss> sounds like a route to logical and readable code!
<supersandro2000> doing that with ytdl is pointless and would make the ytdl package recursive
<supersandro2000> that does not apply to stdenv.lib
<V> supersandro2000: adding lib to the package params and using it elsewhere is literally less characters than using stdenv.lib
<qyliss> I'm not sure this is worth arguing over when supersandro2000 has already said they don't care
<supersandro2000> if there is a benefit of splitting them no one told me which it is
<V> 4 for "lib, ", 3 for the use of lib. that's 3 whole characters more :o
<qyliss> I think we should probably all disengage, because there's nothing productive that can come from this conversation.
<V> as for me, I'm still trying to find where lib is set in stdenv
<qyliss> yeah I tried and failed
<siraben> V: huh, is it that hard? wow
<qyliss> siraben: join the hunt! :D
<V> >you are in a maze of twisty passages, all alike
<supersandro2000> qyliss: maybe someone could explain me what the problem is
<V> ?
<supersandro2000> or like why you shouldn't do
<V> I just did
<supersandro2000> lib is not part of stdenv
<V> it's been explained by multiple people I think three times over the course of this conversation
<qyliss> the problem is that lib is not part of stdenv, and so using "stdenv.lib" is confusing
<V> yes
<V> stdenv just *happens* to contain an alias to lib
<qyliss> because it misleads people as to what stdenv is
<supersandro2000> yeah, I have literally no clue 😂
<V> actually
<siraben> Is it not in /Users/siraben/Git/forks/nixpkgs/pkgs/stdenv/generic/default.nix?
<siraben> Oops well wherever your nixpkgs is :P
<qyliss> siraben: not on my computer :P
<qyliss> but yeah looks like you're right
<siraben> At the top it says, let lib = import ../../../lib; in lib.makeOverridable (
<V> that's just a let
<qyliss> V: scroll down
<V> I found that like 50 years ago
<samueldr> though there _is_ an inherit
<qyliss> # For convenience, bring in the library functions in lib/ so
<qyliss> # packages don't have to do that themselves.
<siraben> inherit lib;
<V> no, I just figured out the sensible solution
<V> > builtins.unsafeGetAttrPos "lib" stdenv
<{^_^}> { column = 14; file = "/var/lib/nixbot/nixpkgs/master/repo/pkgs/stdenv/generic/default.nix"; line = 157; }
<V> There.
<supersandro2000> so why is there an alias you are not supposed to use?
<qyliss> supersandro2000: because we haven't removed it yet
<siraben> Amazing
<V> If I actually had functioning memory I'd have just done that 15 minutes ago
<qyliss> V: oh that's a cool trick
<V> time to build myself a time machine so I don't look like such an idiot right now
<qyliss> when was callPackage invented?
<samueldr> supersandro2000: with the Nix ecosystem, many answers are answered by something like "it was like that way back when"
<qyliss> yeah this dates from 200
<qyliss> *2007
<samueldr> I would have believed 200!
<qyliss> lmao
<qyliss> I'm sure it was a good idea at the time
<siraben> Commit fd268b4852d39c18e604c584dd49a611dc795a9b
<qyliss> but it isn't now
<siraben> Date: Mon Aug 2 13:57:57 2010 +0000, "Add callPackage etc."
<V> siraben: that sounds believable
<supersandro2000> so it is for deprecation?
<samueldr> oh, and follow-up to what I said, generally it is also because "no one changed it or questioned it since then"
<V> supersandro2000: don't worry, this is normal
<V> as is the bikeshedding when someone tries to remove it
<V> actually, maybe do worry
<V> a lot
<supersandro2000> so... I just ignore it?
<V> what else can one do
<siraben> Delete this line and see 10K+ packages break o.O
<samueldr> from the commit, it looks like it was introduced before `lib` was available in any other form, so since callPackage is ~3 years younger, that makes sense
<supersandro2000> tree wide change
<supersandro2000> if you could automate the fixing
<V> it's a matter of slowly trying to clean things up
<V> supersandro2000: can I introduce you to guix
<V> * V has been kicked from #nixos-dev
<qyliss> supersandro2000: yeah, this is step one of trying to clean it up
<qyliss> much easier to start with a documentation change than a 30000 line diff
<siraben> guix using scheme sounds so nice, do they routinely write lisp programs to do metaprogramming?
<siraben> treewide PRs equivalents
<supersandro2000> I feel like I want some CI nitting about this
<samueldr> isn't step 1 finding _actually why_ things are the way they are, or is it off-by-one and step 0?
<supersandro2000> siraben: if nix used lisp I wouldn't be here
<siraben> supersandro2000: not a fan of lisp? hehe
<supersandro2000> I have no clue about functional programming but lisp is just .... hard
<siraben> supersandro2000: yeah, I think what I would do to remove stdenv.lib is first identify where lib is being imported in packages and safely do a sed to replace stdenv.lib with lib
<V> step 1 is someone innocently asking why something is like it is. step 2 is locating the commit/line where it was introduced in a large commit containing a bunch of unrelated things in 2007 because of the personal preference of the three people who used it back then. step 3 is having a huge argument about why it should be left like that
<siraben> though, some packages break their imports by newlines >.<
<supersandro2000> siraben: we could do that maybe in batches for 100 or 1000 packages
<siraben> supersandro2000: right
<qyliss> supersandro2000: CI bringing it up would be great IMO!
<samueldr> V: step 4 is arguing about the step numbering and ordering
<qyliss> oh good does that mean step 3 is over now :P
<supersandro2000> do whatever you want. The Factory must grow
<V> samueldr: there's a pair of steps for ??? and profit somewhere in here
<siraben> Eelco in 2007: "yeah it'll be convenient"
<V> siraben: basically
<samueldr> V: yea, 3 ;)
<supersandro2000> if someone asks I just say it is planned for deprecation
<V> samueldr: okay, but the real question is whether it should be numbered from 0
<qyliss> supersandro2000: you can link to the manual, which now explains it
<siraben> It'd be nice to stop the flow of new stdenv.lib immediately
<qyliss> we can start by slowing it downs
<qyliss> *down
<supersandro2000> qyliss: it doesn't
<siraben> supersandro2000: that's the stable manual
<siraben> when unstable advances, the unstable manual will reflect the changes
<qyliss> I'm so glad we finally have unstable manuals
<qyliss> who made that happen?
<supersandro2000> I had a hard time finding it...
<qyliss> I'd like to tell them how grateful I am
<siraben> the only reason i noticed this was because i was bored and saw the commit message of c06b2b3d671da4847f950e7e9041920b40eca0bf
<qyliss> supersandro2000: 09:26 <siraben> when unstable advances, the unstable manual will reflect the changes
pmy has quit [Quit: WeeChat 3.0]
<V> so like, we have treewide cleanup operations every few months. how much time would it take to write a whitespace-preserving parser for performing operations like this, compared to how much time we waste on trying to manually do the equivalent for every one of these (and the inevitable breakages from human error)?
<qyliss> so it doesn't say it yet, but it should in a few hours
<supersandro2000> if it wouldn't be in commondoc you could actually read it
pmy has joined #nixos-dev
<V> hey, we'd be able to read it immediately if CI happened before merge, like in any other project
pmy has quit [Client Quit]
<siraben> V: i agree, having done some treewide PRs myself it can be painful
<V> (which is, again, something that will take a bunch of time and energy to do anything about)
<siraben> for 200 files I just went ahead and did it manually
<siraben> but if it concerns thousands, well automation is needed
<V> siraben: the most painful part about treewide PRs is how little attention they get compared to the effort involved
<siraben> V: hm, so far I've had success in getting them attended to and merged
<V> whereas trying to change a weird default chosen in 2007 gets immense scrutiny and bikeshedding
<V> my hypothesis is that people just scroll through the first n pages of similar-looking diff output and go "mm, looks good"
<siraben> This can be split up by doing sections of nixpkgs, so it's manageable
<qyliss> siraben: are you the person who's been doing a bunch of treewide PRs recently?
<siraben> qyliss: cmake, whitespace cleanup and (unmerged) single quoted strings?
<siraben> that's me :P
<qyliss> thanks for doing all that!
<qyliss> I know it's hard, thankless work
<siraben> no problem!
<siraben> At the very least for now, I will look through all the packages I maintain and remove stdenv.lib
<qyliss> you're not a committer, right? do you have one working with you who'll be able to merge when it's ready?
<qyliss> (the single quoted strings PR)
<siraben> I'm not a committer. I'll just ping Ericson2314 again for #108099
<{^_^}> (by siraben, 1 week ago, open): treewide: fix double quoted strings
<qyliss> well I'm also happy to take a final look when you think it's ready for merge :)
<siraben> Thanks, will keep that in mind.
<V> interested in more thoughts on building a tool for performing mass rewrites like this (à la gorename, etc)
<siraben> I would support such efforts. It would let us do more ambitious treewide PRs and take less time to do so.
<V> while nix isn't quite as simple to do this with as a lisp, it's still a fairly simple syntax
<qyliss> could you use hnix for that?
<V> I can't use haskell, so no
<siraben> Someone mentioned hnix, in a treewide issue
<siraben> I'm familiar with Haskell
<Profpatsch> supersandro2000: don’t worry, there’s a movement for moving docs to markdown & the tools that rust uses
<qyliss> that's happening right now isn't it?
<Profpatsch> There’s an rfc which was merged which codifies the descision to move from docbook to markdown with extensions
<V> unfortunately haskell is a theoretical language, so people who solve practical problems can't actually use it :p
<Profpatsch> Tooling not yet decided afaik
<siraben> qyliss: yeah, IIRC almost all of the language frameworks docs have been migrated
<siraben> and I migrated the cross docs
<Profpatsch> V: hnix doesn’t have a whitespace-preserving pretty printer
<qyliss> the language framework docs were already markdown aiui
<qyliss> what does nixpkgs-fmt use/
<siraben> Profpatsch: aww
<V> wellll that makes hnix effectively useless for this then
<Profpatsch> I don’t even know how you even write something like that
<Profpatsch> It’s a friggin hard problem
<V> it's... not that hard?
<siraben> nixpkgs-fmt can also produce a large diff no?
<siraben> e.g. in the imports
<Profpatsch> V: really?
<qyliss> yeah but it leaves whitespace and stuff alone aiui
<Profpatsch> I haven’t wrapped my head around it
<qyliss> I've never used it but that's my impression from reviewing PRs formatted with it.
<siraben> nixpkgs-fmt + some context sensitive rewrites is probably he way to go
<qyliss> so it must have a whitespace-preserving parser that could be reused
<qyliss> I'm not saying literally use nixpkgs-fmt for it, but it must include code that's capable of this
<Profpatsch> It probably just does some hacks :)
<siraben> yeah in long enough imports it splits them line by line
<V> Profpatsch: consider how many non-AST elements there are
<V> just whitespace and comments
<Profpatsch> V: but like, when you insert something you want to also reflow the new code
<Profpatsch> without having to reindent the whole file
<V> whitespace can be broken down into newlines and indentation, comments into linewise and spanwise
<siraben> every time I make a treewide PR I also leave the code snippets used, it's always been some sed/awk voodoo
<siraben> with the cmake one fortunately there were only like 230 occurrences so doing it manually sufficed
<V> Profpatsch: define "reindent"
<V> I don't consider reformatting anything an acceptable solution here fwiw
<V> (it would certainly help if everything was formatted consistently beforehand, though...)
<Profpatsch> V: so what you could do is just find the problematic spans and then do simple string-replacement
<Profpatsch> e.g. you want to parse and have rules like “if a symbol called `stdenv` is imported in the file header, any `stdenv.lib` that refers to the same symbol should be replaced by the string `lib`”
<Profpatsch> And another rule that says “if that happens, there should be a `lib` import in the function header”
<qyliss> the latter is going to be hard
<Profpatsch> why? pattern match on the AST
<siraben> without things being formatted automatically to begin with, ugh seems hard
tilpner_ has joined #nixos-dev
<Profpatsch> if the file starts with a function header, check whether it contains `lib`, and if not insert it with a regex that amounts to s/^{/{ lib, /
<qyliss> usual style is { stdenv, lib, <rest> }
<siraben> quick can someone use machine learning for this :P
<qyliss> so it would be easy enough to just insert it after stdenv I guess
<Profpatsch> well, you can have some special rules like that if you are feeling fancey
<Profpatsch> but totally optional
<qyliss> well it wouldn't need to be a special rule
<Profpatsch> at this point, it might be easier to just dump the spans and go through by hand
<qyliss> because ~everything that uses stdenv.lib must import stdenv
<Profpatsch> What I’m saying is that you don’t really need a whitespace-preserving parser, just a span-preserving parser
<Profpatsch> hnix does that for example
tilpner has quit [Ping timeout: 264 seconds]
tilpner_ is now known as tilpner
<qyliss> tbh I'm becoming less convinced you even need a parser for this
<Profpatsch> yeah, helm-edit is probably also a viable solution
<siraben> look at how Ericson2314 did it
<siraben> another thing to bikeshed, remove pkgconfig with pkg-config treewide
<Profpatsch> I mean, better tools also lead to better refactors
<V> <siraben> quick can someone use machine learning for this :P ← now you have an undecidable number of problems
<Profpatsch> So it might be worth investigating if a rewriter like that could work
<V> better stick with regex, then you only have two :p
<siraben> sounds like an interesting theoretical problem too, because it would generalize to other languages
<Profpatsch> Is perl-packages.nix autogenerated?
<siraben> Also, stemming the flow of new stdenv.lib occurrences by putting a warning in ofborg and nix-hammering is an easy first step
<qyliss> Profpatsch: don't think so
<supersandro2000> Profpatsch: yeah I know. it is mostly done for the language docs
<V> <qyliss> usual style is { stdenv, lib, <rest> } ← what about cases where someone inserted something between the two? what about when lib is on a different line (but still in the parameter set? what about a function that takes another function that itself takes a stdenv or lib? what about where there's both stdenv and lib in the params, but the file contains stdenv.lib regardless?
<V> there's sooooo many places where just random find and replace can go wrong
<V> no matter how confident you are that you've done the change correctly this time, you *have* to check over the entire thing to look for mistakes
<Profpatsch> V: not really, you just instantiate and check whether there’s instantiation errors
<qyliss> any of the cases you've brought up would be identified by OfBorg
<V> (I do this kind of thing fairly regularly and still come across subtle mistakes that are only obvious afterwards)
<V> qyliss: what if lib is already in the scope and refers to something else
<Profpatsch> V: you mean it is aliased?
<Profpatsch> That’s why my rule above included a check for whether it’s the same symbol
<Profpatsch> nix can already do static checks like this
<qyliss> if somebody assigns something other than lib to "lib" I think that's just asking for it tbh
<Profpatsch> It’s not too hard to write a rule for if you have the AST parsed
<V> qyliss: like in nixos? :)
<Profpatsch> that’s the same lib
<V> there's another lib dir
<V> there's also paths containing the literal lib
<Profpatsch> It’s duck-typed
<Profpatsch> it might have a few other fields
<Profpatsch> but the basic ones are there
<V> there's also newScope
<V> and extends, etc etc
<V> so many implicit assumptions you don't know you're making until after something goes wrong
<siraben> What's the difference between pytest and pytestCheckHook?
<qyliss> I'd guess the latter runs the tests automatically in checkPhase
pmy has joined #nixos-dev
<siraben> yet another area of nixpkgs that needs cleanup and consistency, the python packages
<siraben> there's many packages that import the python package set (e.g. python3Packages) directly instead of the specific python packages it needs
<{^_^}> #108927 (by Profpatsch, 30 seconds ago, open): perl-packages: stdenv.lib -> lib
<Profpatsch> This one is a quick win
<qyliss> I'm usually afraid to touch perl-packages.nix lol
<Profpatsch> well, quick if we merge it quickly
<Profpatsch> That’s a culture problem
<Profpatsch> tree-wide refactors should take priority usually
<Profpatsch> Otherwise they become stale quickly and not feasible
<qyliss> yeah that's true
<Profpatsch> But I see one problem of doing it by hand instead of writing a script now, in case of merge conflicts I have to do it over again
<Profpatsch> So refactors like this that touch a lot of code *need* to be automated or they become infeasible
<qyliss> more reason to split them up
<Profpatsch> yeah
<siraben> Sounds like we need one of those "burndown" charts companies like to use to track progress, heh
<siraben> burndown of occurences of stdenv.lib
srk has quit [Ping timeout: 240 seconds]
srk has joined #nixos-dev
<Profpatsch> First source of stdenv.lib uses done
<V> you're making this sound like an infection
<V> stdenv.lib patient zero
<Profpatsch> V: it is a blight and we need to purge it
<Profpatsch> the lands shall be ferrrrrtile again
AlwaysLivid has joined #nixos-dev
xwvvvvwx has quit [Read error: Connection reset by peer]
xwvvvvwx has joined #nixos-dev
<Profpatsch> qyliss: siraben rest of toplevel
<{^_^}> #108934 (by Profpatsch, 22 seconds ago, open): toplevel: stdenv.lib -> lib
<qyliss> wow
<Profpatsch> just more or less simple replace
<Profpatsch> has to add a few libs to imports
<Profpatsch> there’s a lot of ugh withs everywhere
AlwaysLivid has quit [Remote host closed the connection]
<siraben> great, will review
<lukegb> Wait what's wrong with stdenv.lib :p
<qyliss> lukegb: read Profpatsch's commit message, or the recent documentation change, or the backlog
<Profpatsch> this one is a tiny change, but it should lead to a 1000 lines change once somebody reruns the generator
<{^_^}> #108935 (by Profpatsch, 32 seconds ago, open): x11/xorg: stdenv.lib -> lib in meta generator
<lukegb> Oh, I scrolled back but not far enough to get to that discussion, just the beginning of the automated refactoring tool discussion
<Profpatsch> well, I’m running it but it might take a while yet
<Profpatsch> a treewide meta = with stdenv.lib { is still missing
<qyliss> lovesegfault: update on node2nix -- node2nix doesn't seem to generate a file that Nixpkgs needs, so the update is blocked for now.
<qyliss> I'm trying to figure out why it's stopped doing that and I'll do a node2nix issue or PR when I have more info.
<siraben> Heh, #108934 is getting some activity
<{^_^}> (by Profpatsch, 17 minutes ago, open): toplevel: stdenv.lib -> lib
<Profpatsch> huh, hnix is actually building
xwvvvvwx has quit [Quit: ZNC 1.8.2 -]
<V> >pkgs.stdenv.lib.optionalString pkgs.stdenv.isDarwin
<V> that's... quite the call
xwvvvvwx has joined #nixos-dev
<V> needs more youtube-dl /s
<lukegb> supersandro2000: does your nixpkgs-review thing for not building tensorflow cause it to error when it comes up?
<lukegb> Or does it just not try to build it at all
<lukegb> (thinking about my envoy init CL - envoy also takes a while to build on not-a-64-core machine)
<lukegb> s/CL/PR/
<lovesegfault> qyliss: Oh, interesting
<{^_^}> svanderburg/node2nix#222 (by lovesegfault, 2 days ago, open): New Release
<qyliss> yeah I saw
<Profpatsch> beware, I’m wielding hnix now
<Profpatsch> It actually uses a fixpoint, so mapping over subsets of the AST is trivial, what a relief
<supersandro2000> lukegb: could it be. It should filter it out of the build.nix
qyliss has quit [Quit: bye]
qyliss has joined #nixos-dev
<siraben> I opened issue #108938 to track it
<{^_^}> (by siraben, 12 seconds ago, open): Deprecate use of stdenv.lib across Nixpkgs
orivej has quit [Ping timeout: 240 seconds]
qyliss has quit [Quit: bye]
AlwaysLivid has joined #nixos-dev
qyliss has joined #nixos-dev
tilpner has quit [Remote host closed the connection]
tilpner has joined #nixos-dev
qyliss has quit [Quit: bye]
qyliss has joined #nixos-dev
orivej has joined #nixos-dev
<{^_^}> firing: RootPartitionLowInodes:
__monty__ has joined #nixos-dev
<{^_^}> svanderburg/node2nix#223 (by alyssais, 49 seconds ago, open): node-env.nix is no longer generated
<lovesegfault> qyliss++
<{^_^}> qyliss's karma got increased to 111, it's a crit!
qyliss has quit [Quit: bye]
qyliss has joined #nixos-dev
immae has joined #nixos-dev
immae[m] has left #nixos-dev ["WeeChat 2.9"]
nyanotech has joined #nixos-dev
orivej has quit [Ping timeout: 260 seconds]
teto has joined #nixos-dev
AlwaysLivid has quit [Remote host closed the connection]
cole-h has joined #nixos-dev
cole-h has quit [Quit: Goodbye]
cole-h has joined #nixos-dev
orivej has joined #nixos-dev
teto has quit [Ping timeout: 246 seconds]
catern has quit [Ping timeout: 264 seconds]
costrouc has joined #nixos-dev
<{^_^}> firing: RootPartitionLowInodes:
<lukegb> If I'm fixing some packages because nixpkgs-review is complaining that they're broken (but they're broken at HEAD), should I put that in a separate PR or include it in my existing one
<qyliss> seperate
* lukegb sad trombones
bridge[evilred] has joined #nixos-dev
srk has quit [Read error: Connection reset by peer]
justanotheruser has joined #nixos-dev
srk has joined #nixos-dev
v0|d has joined #nixos-dev
catern has joined #nixos-dev
sphalerite has quit [Quit: WeeChat 2.6]
sphalerite has joined #nixos-dev
rajivr has quit [Quit: Connection closed for inactivity]
zarel has quit [Ping timeout: 246 seconds]
teto has joined #nixos-dev
<{^_^}> firing: RootPartitionLowInodes:
mkaito has joined #nixos-dev
<{^_^}> #108979 (by Profpatsch, 11 seconds ago, open): doc: stdenv.lib -> lib
<Profpatsch> and the absurd one cc siraben qyliss
<{^_^}> #108978 (by Profpatsch, 26 minutes ago, open): treewide: with stdenv.lib; in meta -> with lib;
<andi-> Profpatsch: are we able to enforce this via some sort of CI step?
<Profpatsch> andi-: I think the trick is to reduce examples in the codebase to 0
<Profpatsch> Most of these are because the docs had them
<Profpatsch> And the rest because of copy-pasting
<andi-> ok
<abathur> plus the ones baked into the pipeline in current PRs :) I saw noticed the issue and already changed the one I have open
__monty__ has quit [Quit: leaving]
Emantor has quit [Quit: ZNC -]
risson has quit [Excess Flood]
risson has joined #nixos-dev
<cole-h> 101-500 rebuilds is fine to go to master, correct? Or should it go to staging? ref: #108958
<{^_^}> (by Atemu, 7 hours ago, open): Update zen kernels
Emantor has joined #nixos-dev
<infinisil> cole-h: master imo
<cole-h> Cool, thanks. I was leaning towards master mostly because it was 2 packages that contributed to it. The _zen kernel alone was 11-100 (which I was going to merge before they created that all-inclusive zen PR).
* cole-h runs nixpkgs-review first
<samueldr> cole-h: all of it is churn from linuxackages_* from those linux packages updated
<samueldr> and many, maybe most, of the linuxPackages are trivial builds
<adisbladis> samueldr: I couldn't help myself and ordered the A5 Pro CC anyway
<samueldr> adisbladis: move to -chat? :)
<adisbladis> Oh, I thought this were -chat :)
<adisbladis> Sorry
<samueldr> n/p
<cole-h> samueldr: Fair. Thanks.
<samueldr> cole-h: one thing to keep in mind is whether it's expensive rebuilds or not, when looking at what seems like big numbers
<samueldr> (and then learning what is and is not expensive)
<cole-h> Yeah, how should I figure out what's expensive and what's not? Does Hydra list how long it takes to build a certain package?
<samueldr> you can get that information... I don't think there's anything ready-made to get it out in a useful manner for contributors
<samueldr> I guess that could be helpful
<infinisil> Recently I've been noticing that there's a common pattern in Nix I haven't heard anybody talk about: Static vs dynamic configuration
<infinisil> Basically, static config contains the values in the config files *directly*, while dynamic config only points to a *file* where the config can be read from. Some observations:
<infinisil> Static config can't be changed at runtime without restarting/rereading the config file, while dynamic config can
<infinisil> With Nix, static config values are usually in the /nix/store, and as such world-readable. This means only dynamic config should be used for secrets
<infinisil> It is often desirable to be able to selectively choose which values of a config file are static and which are dynamic. Static config for persistence, but dynamic config if quick iteration is needed
<infinisil> I think it would be really interesting to have a Nix/NixOS-native concept of static vs dynamic config
<infinisil> Here's a motivating example:
<{^_^}> #60339 (by thoferon, 1 year ago, open): wireguard: Add peers.*.publicKeyFile as an alternative to publicKey
<infinisil> What if *all* options implicitly had the capability to be declared either statically or dynamically
<cole-h> qyliss++ Thanks for putting the "prefer lib over stdenv.lib" into words (and a proper guideline)
<{^_^}> qyliss's karma got increased to 112
<infinisil> And we had a framework to support that easily, e.g. by having a `(pkgs.formats.json {}).generate`-like concept which can compose config files at runtime consisting partially of static values, partially of dynamic ones
<ekleog> This would sound awesome, though it also sounds like something that would require rewriting basically all modules as the configuration now needs to be written at runtime and no longer at compile-time
<infinisil> ekleog: I'm thinking that the options declarations could choose whether they support static, dynamic, or both types
<infinisil> E.g. `mkOption = { supportsStatic ? true, supportsDynamic ? true }`
<infinisil> Um rather `mkOption = { supportsStatic ? true, supportsDynamic ? false, type, ... }`
<infinisil> And users might declare values as either `some.value = mkStatic "value"`, `some.value = mkDynamic "/some/file"`, or `some.value = "value"`, which is the same as the former
<infinisil> And module authors could check whether an option is static/dynamic (if their option supports both) with e.g. `{Static,Dynamic}`
<infinisil> This might have to be a value-level thing though, because with a single `settings` option, you want to be able to declare any subpart of it as static/dynamic
<ekleog> Sounds like a cool design then, still requires rewriting modules for them to take advantage of it but then that's something that can't really be worked around anyway :)
<infinisil> So maybe `config.some.value ? _dynamicValue` instead, where if that's false it's a static value
<infinisil> Yeah :)
<infinisil> I'm not entirely convinced yet, but it's an increasingly common pattern I see, especially with sensitive data often ending up in configuration files
<ekleog> as for mixing static and dynamic values, yeah, that'd be possible iff it's possible to define a merge strategy… or maybe decide to crash if the same value is defined in both static and dynamic configs?
<ekleog> actually no that doesn't make sense, because sets themselves are values
<infinisil> No need for merging, you can't set a value as both static and dynamic
<infinisil> Well you do need to somehow combine the static and dynamic configs in the end, but that shouldn't give any conflicts
<ekleog> Right, I meant wrt. your “this might have to be a value-level thing” to define static/dynamic on a whole option tree, but after further thought I'm with you that probably doesn't work anyway
<infinisil> Yeah
<infinisil> Also, this might only work for strings, because file contents are strings
<infinisil> Although you could implement some transformers inbetween which e.g. parse an int from the fiel
<ekleog> well, files could be JSON or similar things
<ekleog> (though that'd mean more work for eg. “have this file be the private key")
<infinisil> Oh yeah
<infinisil> So maybe each option has an associated runtime parser, which transforms files into the structure that option expects
<ekleog> Sounds reasonable, in which case supportsDynamic => dynamicKind != null
<infinisil> dynamicKind?
<infinisil> I'm thinking that if a value supports dynamic, and has a type != string, it needs to provide a runtime parser that can transform a string into type
<ekleog> a name I just made up for the associated runtime parser
<infinisil> Ah :D
<ekleog> like supportsDynamic = true; dynamicKind = kinds.json;
<infinisil> Oh actually that should probably by handled by the `lib.types.*` themselves
<ekleog> which… maybe could be override-able in `mkDynamic`? though I'm not actually sure that'd bring anything so it may be pointless over-configurability
<infinisil> Although never ind
<infinisil> mind
<infinisil> I guess the most basic version could just support `str`'s for simplicity
<ekleog> actually upon further thought, making it possible to override in `mkDynamic` would allow eg. fetching the configuration from a remote source and automatically merging it in, so it may not be pointless (though whether it holds its weight is TBD)
<infinisil> Ohh
<infinisil> Yeah so we can have values directly, values in files, but we can further abstract that to an arbitrary command that outputs the value
<infinisil> Then static values are `echo "the value"`, files are `cat /the/file`, and arbitrary commands are `curl`
<ekleog> oooh yeah that can be one further layer of abstraction!
<infinisil> Interesting..
<ekleog> unifying both use cases
<infinisil> So maybe it should be something like `mkOption { kind = "value" / "file" / "command"; }`
<infinisil> Um, I mean, `supportsValue`, `supportsFile` and `supportsCommand`
<infinisil> And `mkValue` (implicit), `mkFile`, `mkCommand`
<ekleog> hmmmmm actually `kind = "value" / "file" / "command"` made sense to me? and then `mkValue` used on `kind = "command"` would lead to `mkCommand "echo ${value}"`
<ekleog> and `mkValue` used on `kind = "file"` would throw
<infinisil> Yeah but options need to support multiple kinds, it can't just be a single one
<ekleog> Hmm…? I'm thinking option implementations needs to support only one kind, so long as eg. `mkValue` automatically casts from a value to a command: for the user it's as though the option supported both kinds
<infinisil> So that a single option can be defined as either `mkValue` (if it supportsValue), `mkFile` (if it supports etc.
<infinisil> Ohh I see what you mean
<ekleog> like, `mkValue foo` would be `{value = foo; command = "echo ${foo}"; }`
<ekleog> `mkFile foo` would be `{ file = foo; command = "cat ${foo}"; }`, and `mkCommand foo` would be `{command = foo;}`
<infinisil> Yeah that sounds good!
<infinisil> Although problem
<ekleog> (actually maybe `mkValue foo` could even be `{ value = foo; file = writeText "stuff" foo; command = "echo ${foo}"; }`
<infinisil> These values would then only be available at runtime
<infinisil> But module evaluation often uses values at eval time
<infinisil> We'd probably want a normal value definition to be just that, `some.value = "foo"` really just defines it as foo, so that other modules can depend on that
<infinisil> But if it's `mkFile` or `mkCommand`, `some.value = mkFile "/foo"` turns into `some.value = throw "Only known at runtime"` for other modules
<ekleog> Hmm… is that not what I was suggesting with `mkValue`? and then `kind = "value"` would mean “use [option].value” etc.
<infinisil> I'm a bit lost now
<infinisil> But yeah the whole depending-on-values-at-eval-time thing becomes a bit problematic with this
<infinisil> Because you can't rely on *any* option being known at eval time anymore, even if it has `types.str` or so
<infinisil> And I believe that's kind of a strong assumption we rely on currently
<ekleog> What I mean is, `let mkValue = val: { value = val; file = writeText "value" val; command = "echo '${val}'"; }; mkCommand = cmd: { value = throw "Only known later on"; file = throw "Only known later on"; command = cmd; }` etc. would IMO solve that issue
<ekleog> But yes, it means that all options that are not `kind = "value"` can no longer be used at eval time,
<ekleog> (but that's cosubstantial with the idea of having dynamic values)
<infinisil> Ohh I see, yeah your mkValue and co. implementations look great
<infinisil> Yeah, and I guess you can get very good errors in such cases
<infinisil> And it's not like it breaks any existing workflow, it only allows more use case
<infinisil> s
<ekleog> right, that idea sounds great :D
<infinisil> ekleog: One issue left is that for files/commands it needs to be defined in which context they're accessed/ran
<infinisil> For files it matters which user accesses them, so they need appropriate permissions
<infinisil> For commands there's the PWD, the user who runs it, whether it runs in a sandbox, etc.
<infinisil> This could just be left to the module itself though
teto has quit [Ping timeout: 264 seconds]
<ekleog> well, all that's outside of the realm of what the module system can do by itself, I think? because all that will be defined by the module implementations themselves
<ekleog> (if only because runtime options mean that there needs to be a program started, and not all modules have one)
<infinisil> Ohh, idea: To implement this, we provide a unix socket at `/run/nixos-dynamic.sock`, where each module can request the values it needs over an API
<infinisil> On the server side we can then implement generically how these files/commands should be accessed/ran
<infinisil> So this wouldn't have to be decided by the module itself
<infinisil> Hmm is this actually useful though