<worldofpeace>
That would be useful for us I think
<samueldr>
the page you linked, I thought it was about the whole security thing, didn't look too closely at the exact link you linked (sorry)
<worldofpeace>
has there ever been nixos specific security issues that could've benefit from confidentiality?
<samueldr>
>> Publish the advisory to alert your community of the vulnerability
<samueldr>
it's more about the signal it gives
<samueldr>
>> GitHub will review each published advisory and may use the advisory to send security alerts to affected repositories. For more information, see "About security alerts for vulnerable dependencies."
<gchristensen>
I'm on my way to bed, but: having a confidential way to discuss vulns ticks off a requirement for the embargoed security list. (if github satisfies that, maybe not)
<samueldr>
maybe we should get ahold of a github person and ask "does it make sense for nixos to use advisories to say '$software vuln (CVE-xyz) fixed in abcde"?
<samueldr>
like even if it's done post-embargo, as a centralized location
<samueldr>
acknowledging what's been done
<worldofpeace>
Yeah perhaps we should ask github if that feature is for that or could be for that. They did mention it's in "beta" so it probably could still be mutable if the community provides feedback for what they need out of a feature.
<worldofpeace>
samueldr: you can draft an advisory to discuss in private
clever has quit [Ping timeout: 244 seconds]
Synthetica has quit [Quit: Connection closed for inactivity]
clever has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.4]
gchristensen has quit [Quit: Connection closed for inactivity]
justanotheruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 246 seconds]
justanotheruser has joined #nixos-dev
noonien has quit [Read error: Connection reset by peer]
kalbasit has quit [Ping timeout: 276 seconds]
alunduil has quit [Ping timeout: 252 seconds]
sdier has quit [Ping timeout: 276 seconds]
c00w has quit [Ping timeout: 252 seconds]
teehemkay has quit [Ping timeout: 252 seconds]
c00w has joined #nixos-dev
sdier has joined #nixos-dev
teehemkay has joined #nixos-dev
kalbasit has joined #nixos-dev
alunduil has joined #nixos-dev
noonien has joined #nixos-dev
nix-build has joined #nixos-dev
{^_^} is now known as Guest7513
Guest7513 has quit [Ping timeout: 252 seconds]
noonien has quit [Ping timeout: 250 seconds]
noonien has joined #nixos-dev
johnny101 has quit [Remote host closed the connection]
sdier has quit [Ping timeout: 252 seconds]
johnny101 has joined #nixos-dev
sdier has joined #nixos-dev
cbarrett has quit [Ping timeout: 250 seconds]
teehemkay has quit [Ping timeout: 252 seconds]
nix-build has quit [Ping timeout: 252 seconds]
cbarrett has joined #nixos-dev
teehemkay has joined #nixos-dev
orivej has quit [Ping timeout: 244 seconds]
pie_ has quit [Ping timeout: 252 seconds]
justanotheruser has quit [Ping timeout: 272 seconds]
justanotheruser has joined #nixos-dev
pie_ has joined #nixos-dev
tilpner has quit [Quit: WeeChat 2.4]
tilpner has joined #nixos-dev
gchristensen has joined #nixos-dev
init_6 has joined #nixos-dev
justanotheruser has quit [Ping timeout: 258 seconds]
{^_^} has joined #nixos-dev
justanotheruser has joined #nixos-dev
psyanticy has joined #nixos-dev
coconnor has quit [Ping timeout: 272 seconds]
init_6 has quit [Ping timeout: 248 seconds]
grahamc has joined #nixos-dev
gchristensen has quit []
grahamc is now known as gchristensen
Jackneilll has quit [Ping timeout: 272 seconds]
Jackneill has joined #nixos-dev
orivej has joined #nixos-dev
pie_ has quit [Ping timeout: 252 seconds]
coconnor has joined #nixos-dev
drakonis has joined #nixos-dev
drakonis1 has quit [Ping timeout: 246 seconds]
drakonis_ has joined #nixos-dev
drakonis1 has joined #nixos-dev
drakonis has quit [Ping timeout: 248 seconds]
drakonis_ has quit [Ping timeout: 245 seconds]
drakonis has joined #nixos-dev
drakonis1 has quit [Read error: Connection reset by peer]
drakonis has quit [Ping timeout: 259 seconds]
MichaelRaskin has joined #nixos-dev
psyanticy has quit [Quit: Connection closed for inactivity]
drakonis has joined #nixos-dev
hedning_ has quit [Quit: hedning_]
<infinisil>
I wish nixpkgs wasn't so split up for specific topics
<infinisil>
E.g. for python there's pkgs/top-level/python-packages.nix, pkgs/development/interpreters/python, and pkgs/development/python-modules
<infinisil>
For Haskell the same
<infinisil>
And for a lot of other languages too in fact
<ekleog>
have all packages from pkgs/top-level/python-packages.nix been moved to pkgs/development/python-modules something like between 18.03 and 18.09 or similar?
<ekleog>
haven't*
<ekleog>
(personal opinion, though: there is no classification that will satisfy everyone [should a python library for scientific computation be under sci-comp or python-modules?], so we would ideally just have everything as a flat list [potentially namespaced by prefix if that's too many files in a single directory for tools to handle])
<infinisil>
ekleog: Yeah exactly, it's what I've been thinking about for a while
<infinisil>
Getting rid of this directory-categories thing and move to a tag-based system instead
<infinisil>
flat hierarchy + meta.tags
<infinisil>
And then we can group python together under e.g. pkgs/python, standard packages go under pkgs, haskell stuff under pkgs/haskell
<infinisil>
hello could live under pkgs/he/hello/default.nix (or pkgs/he/hello.nix directly)
<simpson>
infinisil: A consideration: What makes a package "standard"? It sounds like it's a language-based consideration?
<infinisil>
simpson: Yeah that just hit me too.
<infinisil>
We do have "stdenv", which kinda says C/C++ packages are standard
<infinisil>
But there's nothing special on C/C++ really
<gchristensen>
obviously ed is a standard package
<ekleog>
simpson: things that don't need to run in `python.withPackages`, for me
<LnL>
yeah, the categories in general are a bit vague but some structure is probably necessary
<ekleog>
things you can directly nix-shell into
<LnL>
like we do with attribute sets
<simpson>
It sounds like you're thinking about how package managers demarcate territory, but it's important to point out that the typical C library has *no* package management nor any notion of a package, and considering them to be "standard" in terms of their *lack of* build system is not very forward-looking IMO.
<infinisil>
LnL: Did you even need that structure though?
<LnL>
yes, conflicting names
<infinisil>
The only thing this structure does for me is make it difficult to find where packages are defined
<ekleog>
LnL: attribute sets bring more confusion than benefits, to me (except when it's for “this can be used only within .withPackages”)
<samueldr>
is borg-backup a python thing, just because it ended up written in python? while in reality it's most likely an application thing, a leaf package
<simpson>
ekleog: As an example, shouldn't Go be a "standard" language? Its build tools are self-contained, self-hosting, and emit native code.
<infinisil>
Also by having a flat hierarchy (+ auto-calling the files), we can get rid of the confusion of attribute name vs file name that's so common among beginners
<ekleog>
simpson: if go and all its libs can be used outside a withPackages like any regular package, they should be top-level for me (now, that may raise issue for automatically-maintained package sets)
<ekleog>
“can I nix-shell it?” should be the question to know whether it's namespaced, to me
<ekleog>
(python packages can… cannot… well… it works but it doesn't… be nix-shell'd)
<infinisil>
LnL: What do you mean by conflicting names? attribute sets can very much like the file system only contain one value for a key
<ekleog>
(really I'd love something that just beeps at you and error when you nix-shell a python package, it causes so much harm when people do it without noticing)
<samueldr>
hm?
<simpson>
ekleog: What are your examples, though? The Python applications I'm thinking of are correctly set up to work in nix-shell; say, magic-wormhole. Python libraries aren't meant for such standalone use, and while it sucks that we can't nix-shell them, the answer is really to stop expecting so much from Python's lackluster ecosystem.
<simpson>
A Python "package" is a pile of Python source code with opinions on filesystem layout.
<ekleog>
simpson: python applications should not be namespaced, python modules should, IMO
<simpson>
ekleog: If Python's model allowed it, then maybe. But, e.g. how would PyPy work? Or packages which advertise both Python 2 and Python 3 compatibility?
<ekleog>
(and things that are both an application and a module should be split because it makes no sense to couple the two)
<LnL>
infinisil: sure, but the "native" names in the python/haskell/etc registries overlap
<ekleog>
simpson: having un-namespaced python applications doesn't prevent having also namespaced python applications for people who want to pick their specific python version for some reason
<simpson>
LnL: That's a good point. e.g. "redis" is one I encountered this week.
<infinisil>
LnL: All language-specific packages will still be under their own package set though, python packages under python.* or so
<infinisil>
As it is today
<ekleog>
actually, one might consider “libraries under package set, applications global”, which would maybe be even better
<infinisil>
ekleog: I like that thought
<ekleog>
(so adding `cPackages`, `cppPackages`, as needed)
<LnL>
infinisil: so wouldn't it make sense to structure those in the repo in a similar way
<samueldr>
or `nativeLibrairies` for `.so/.dylib` ?
<simpson>
ekleog: We may need an RFC for this. Heck, I need to write an RFC for Python in particular, to figure out CPython 2.7 deprecation.
<ekleog>
simpson: we'll definitely need an RFC for this
<infinisil>
LnL: Yeah that's what I mean, have python under pkgs/python/*, in the pkgs.python.* attribute set, haskell in pkgs/haskell/* under pkgs.haskell.*
<ekleog>
I won't have enough time to give it in the foreseeable future though
<LnL>
infinisil: ah, I was thinking about the same thing with "some" structure :)
<infinisil>
LnL: You mean to still keep some categories?
<LnL>
I'd consider python a category in this case
<ekleog>
infinisil: might rename into pkgs.pythonLibs.* to make things explicit? and pkgs.nativeLibs for things that are not in an upstream package repository (and are native packages)
<infinisil>
LnL: Well I guess that's what it would look like then yeah
<infinisil>
And python as a category mostly makes sense too
<ekleog>
I would definitely love something like it, though it would definitely insta-bitrot all PRs and all nix files that rely on nixpkgs :D
<simpson>
infinisil: The thing is that people shouldn't have to care whether a package, say youtube-dl or get-flash-videos, is written in a particular language, in terms of looking stuff up in a package listing. The only reason that we care about where packages are from, as packagers, is that we need to know which third-party package manager repository to use, whether it be npm, PyPI, CPAN, etc.
<ekleog>
but we need to tackle that dragon *someday*, and better do it as early as possible
* ekleog
predicts strong opposition
<LnL>
and perhaps specific stdenv related things specific to nixpkgs like the cc-wappers etc. are also a bit of a special case
<infinisil>
ekleog: Hmm, I'm not sure I get the native vs non-native distinction
<ekleog>
infinisil: well, nativeLibs is just the “library that's not python, haskell, etc.”
<simpson>
infinisil: So maybe we should ensure that this plan doesn't say "python as a category" and instead considers PyPI individually.
<infinisil>
simpson: Agreed, so the thing ekleog earlier said sounds like a good solution: "actually, one might consider “libraries under package set, applications global”, which would maybe be even better"
<infinisil>
(as we do today already, mostly)
<simpson>
infinisil: Sure. But now, consider: If we fetch from GitHub, then aren't those fetches, which we've agreed are language-independent, now all clumped together? And what clumps them together is the *common API* of fetchFromGitHub.
<ekleog>
I don't understand
<simpson>
So now we start to consider talking about fine-grained Python builders, and having to tackle the uncomfortable fact that such a deep integration means maintaining our own Python filesytem layout. (But of course we already *do* such a thing, so this is merely a matter of deepening those ties.)
<infinisil>
simpson: Ah yeah good thing you're bringing this up
<infinisil>
A lot of things don't fit into python/haskell/whatever, but are nixpkgs utilities like fetchers and meta-builders
<infinisil>
And those would be in their own pkgs/meta or so
<infinisil>
Separate from libraries or applications
<infinisil>
Or pkgs/builders
<infinisil>
Or so
<ekleog>
simpson: I don't think anyone suggested automatically namespacing libraries fetched from github in githubLibs or anything similar
<simpson>
`buildPhase = pythonBuild [ dontEasyInstall useSphinx ];` or `buildPhase = haskellBuild [ (scrapeCabal "package.cabal") ];` or something along those lines.
<infinisil>
ekleog: I'm pretty sure he means the fetcher definitions themselves, like where should pkgs/build-support/fetchers go
<infinisil>
simpson: +1 to that!
<simpson>
ekleog: I'm not suggesting such a thing either; I'm trying to explain the outlines of what it means to stop treating applications as being from some particular language-tribe and instead treating them as a composite of language-oriented build actions.
<simpson>
To pick an unimportant example off the top of my head, the package Z3 contains C, C++, and Python, and its outputs include libraries with C linkage, Python extension modules, and other stuff. Is Z3 a Python package, a C package, a native or standard package? By insisting that we treat the package as being from a certain language, we complicate it beyond what it is.
<ekleog>
actually I was just now thinking of tensorflow, which has a similar issue
<ekleog>
but what we are discussing is about the API we offer, which can (and maybe should?) be orthogonal to the way we implement it
<ekleog>
ie. Z3 could be implemented as “one derivation with multiple outputs for python, C and C++ bindings”, and then we have an alias for each of the multiple outputs in each of the package sets
<ekleog>
(and given nix has deduplication we don't need to have Z3 in the API fix-point)
<infinisil>
Interesting..
<ekleog>
now, the question of “where in the file hierarchy do we put Z3” is complex, and I'd guess “just stuff it into pkgs/weirdstuff” would be a reasonable answer
<gchristensen>
seems overcomplicated to patch Z3's use case over a sort of impedance mismatch
<infinisil>
ekleog: It seems that it would almost make most sense to just dump *everything* into a single directory
<ekleog>
(with `weirdstuff` never being referenced from the API fixpoint)
<simpson>
Yes. But now we have a combinatorial question: Given that Z3 can build against a Python 2 *or* a Python 3, (or CPython vs PyPy), *and* given that the Python build requires other Python modules (maybe not in Z3's case but I think in TF's case?), how do we allow for the composition of these "package sets" as if they were typical derivations?
<ekleog>
gchristensen: this doesn't necessarily need an additional patch to Z3 -- I think we are already having a multiple-output derivation, just because we need one to be able to put it in pythonPackages
<gchristensen>
not a code patch, but a conceptual patch
<ekleog>
infinisil: this would complicate handling of automatic package set imports, I fear
<simpson>
gchristensen: The way I see it is that this is a bit of an Inverse Conway's situation. If Z3 or TF split their work into multiple repositories, then there'd be no problem; as an example, the Prometheus client libraries for C++, Java, Python, etc. are all split up, and no problems in nixpkgs.
<simpson>
If upstream expects to have a single unified build phase, and nixpkgs expects build actions to be language-specific, then we'd better have a way to compose build actions from different languages into a single derivation.
<ekleog>
simpson: at $job we did the TF derivation by having an internal `python.withPackages [dep1 dep2]` (hopefully we'll be able to open-source it when we're sure it works well and it can replace nixpkgs' downloading the python wheel again)
<ekleog>
and then using this python to do the build
<infinisil>
simpson: ekleog: How about just not declaring Z3 in a single place, but instead have 3 totally separate definitions (with potential interdependencies)
<infinisil>
E.g. the python part taking the source from the C part, because it depends on it
<infinisil>
But otherwise be a separate build
<ekleog>
infinisil: the issue is (for TF at least) it takes lots of hours to build, so you don't want to be building it 3 times, and one multiple-output derivation is much nicer
<ekleog>
also, the build process is complex, but very much similar
<infinisil>
I see
<simpson>
infinisil: In the very specific case of Z3, ISTR from thoughtpolice that the C and Python components need to be interlocking in a way that couldn't be done with two derivations. However, in general, I agree: If the *only* problem is that many packages live in one repo, then just split.
<ekleog>
(like, adding 1 line is enough to build an additional target)
<simpson>
infinisil: This is why I mentioned Conway's Law (the shape of a group of people affects the shape of the code): Working in a monorepo *causes* the build steps to become intermingled over time.
<ekleog>
simpson: I'm wondering whether you're thinking of also revamping the way we handle multi-language builds, and if so then I think we don't need to do that at the same time as a “move all the stuff to their right places” step -- it's an orthogonal step, I think
<simpson>
...A neat result, logically from fitting all of the above together, is that some platforms might be very *very* good candidates for Nix flakes, if they have a minimal core that can rebuild themselves. Splitting stuff like the core of GCC, Clang?, Go, OCaml?, etc. from nixpkgs and then having nixpkgs import and depend upon those flakes?
<infinisil>
simpson: Yeah I've thought of this too, would be very nice to split that apart and have really just package definitions in nixpkgs
<ekleog>
(that we likely should also take, but we should just think again hard time on how to do it properly by also thinking of interaction with things like bazel, clang, etc.)
<simpson>
ekleog: Like my poster of Confucius would say if it could speak, we must first know what the right names for everything are before we can know their right places.
<infinisil>
Which makes even most sense, considering the name "nixpkgs"!
<ekleog>
simpson: well… are you suggesting we cut derivations in a way different to the one exposed by the current “reposition everything [here and there]” plan proposes?
<simpson>
There's also some hot-take ideas, like changing the policy that we must make a new path and default.nix for every single tiny library that's only used by one application. If somebody wants to add a package called, I don't know, `frobenius`, then `libfrobenius` probably is not going to be shareable and probably will update at the same time!
<ekleog>
^ in practice it's already what we do
<ekleog>
(see eg. nheko)
<simpson>
ekleog: Oh! I didn't know that there was a plan being executed right now. I thought that folks were spitballing for an RFC to reshuffle the shape of nixpkgs. Sorry, is there some required reading I should do? Please link me.
<infinisil>
Name clashes.. :/
jtojnar has joined #nixos-dev
<ekleog>
simpson: well, we're totally spitballing, sorry for the wrong wording. what I meant by “plan” was “the state of discussion with applications global-scope and libraries in pythonLibs.*, haskellLibs.*, etc.”
<ekleog>
ie. one derivation (or output) per either application or library that we want to expose
<simpson>
ekleog: Oh, then yeah, I am aggressively trying to explain *why* language-based hierarchies are wrong.
<samueldr>
software should be hierarhchised by the country they come from
<gchristensen>
there you go
<ekleog>
well, the point of pythonLibs is to mean “this can be used only in python.withPackages”
<ekleog>
samueldr++
<{^_^}>
samueldr's karma got increased to 88
<ekleog>
of haskellLibs would likely be “we're automatically importing the whole hackage set so just don't name conflict pretty please”
<ekleog>
(also, I'm not sure how one uses a haskell lib)
<ekleog>
basically, namespacing by the way a package can be used doesn't appear wrong to me
<ekleog>
if a package can be used only by calling it from haskell, it being namespaced as a haskell lib doesn't appear wrong, because the downstream user will need some haskell-specific build system anyway
<ekleog>
(rust being callable from C will be more fun, as it should land stuff in nativeLibs… but then not all crates are actually callable from C, and imports will likely name conflict, so I'd suggest a rustLibs from which aliases are taken to nativeLibs for libs that expose a C-like interface)
<simpson>
ekleog: Mm. I would suggest maybe not calling such things "packages" if they can't be used as such within Nix. I wonder whether we could limit that API somehow.
<ekleog>
well, I totally can do nix-shell -E "python.withPackages (ps: with ps; [numpy])" and then start my python
ma27 has quit [Quit: WeeChat 2.4]
<ekleog>
it's not an application, but it's a library
<ekleog>
(I'd love it if we someday included a way for the python.withPackages to be automatic, but that's a separate question :°)
<simpson>
Ah, sure. I think that this is more name overlap; Nix packages aren't Python packages. I can agree that not every Python package should have a name from the top of nixpkgs; indeed, perhaps *no* Python package should get that, and `python.withPackages` would be the primary entrypoint.
<simpson>
The tradeoff, of course, is that now you have an even-more-Python-flavored subtree.
<ekleog>
right, having no python library getting that would likely be even better indeed, as it should be used only in withPackages -- but python-written applications definitely should get a top-level name IMO
<ekleog>
(with the default python that nixpkgs decides of)
<infinisil>
Another problem we haven't touched yet: different versions, aliases to different versions, and related problems
<ekleog>
(I'm trying to use the “package” word only for nix packages, “library” for language-(specific|agnostic for C libs) libraries, and “applications” for “things an end-user is expected to want”)
<ekleog>
with that discussed I'll be getting out for today, thank you for this discussion and hope you can find some ideas for versions -- and that some day this discussion will give birth to an RFC :)
<infinisil>
See ya o/
qyliss^work has quit [Ping timeout: 252 seconds]
ma27 has joined #nixos-dev
qyliss has quit [Ping timeout: 258 seconds]
qyliss has joined #nixos-dev
qyliss^work has joined #nixos-dev
jtojnar has quit [Ping timeout: 268 seconds]
pie_ has joined #nixos-dev
drakonis has quit [Ping timeout: 248 seconds]
copumpkin has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
drakonis has joined #nixos-dev
janneke has quit [Ping timeout: 259 seconds]
janneke has joined #nixos-dev
init_6 has joined #nixos-dev
justan0theruser has joined #nixos-dev
justanotheruser has quit [Ping timeout: 268 seconds]