<{^_^}>
mvn2nix-maven-plugin#5 (by trentonstrong, open): project-info.json missing "authenticated" flag sometimes if dependency has already been fetched into local maven repository
<ekleog>
let's go the gephy way instead and just build twice
garbas has joined #nixos-dev
vcunat has quit [Quit: Leaving.]
vcunat has joined #nixos-dev
sir_guy_carleton has joined #nixos-dev
phreedom has quit [Ping timeout: 250 seconds]
phreedom has joined #nixos-dev
__Sander__ has joined #nixos-dev
phreedom has quit [Ping timeout: 250 seconds]
phreedom has joined #nixos-dev
orivej has quit [Ping timeout: 264 seconds]
pie_ has quit [Ping timeout: 260 seconds]
pie_ has joined #nixos-dev
Sonarpulse has joined #nixos-dev
<thoughtpolice>
Uhhhhhh... Okay, let's say I have a problem, of having a really really _really_ big closure that I need to ingest into the Nix store, but even with all the latest memory improvements, etc, no dice for fitting it in. Is there a way to like... Extract a closure into the store manually and register it with the database?
<thoughtpolice>
(Assuming 'closure' means the results of `nix-store --export /nix/store...` ofc)
<thoughtpolice>
And by "fitting it in" I mean actually importing it. Actually, with some patches, `nix-store --import` doesn't fail due to running out of memory, but it appears a bug causes it to die while talking to the daemon (maybe integer overflow?)
<thoughtpolice>
Sonarpulse: right, I know that one, but that's for individual files, right? I don't want to add a file, I want to import the entire closure.
<thoughtpolice>
So I'd have to like manually register all the GC refs and everything, right
<thoughtpolice>
To be clear this closure is something like 30GB so it isn't exactly small.
<thoughtpolice>
Maybe I should just try to find the bug in Nix instead that causes it to die but, ughhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
<Sonarpulse>
i would have thought the limitations were per output
<Sonarpulse>
ugh
<Sonarpulse>
Nix why...
<thoughtpolice>
It's literally just been one thing after another like this dude, for like a week now. :| Needless to say Hydra and I are very upset at each other and aren't on speaking terms, either.
<thoughtpolice>
So maybe debugging C++ will actually make me happier, who knows :P
<Sonarpulse>
the underlying issues with the per file ones have been fixed in PRs
<Sonarpulse>
maybe applying one of those would help even though this is different
<thoughtpolice>
Good news: the peak memory usage went from like 20GB in approximately ~0 seconds to something like 5GB over a few minutes. Aaaand then it dies.
<thoughtpolice>
Like I said I speculate this is just an outright bug somewhere, otherwise it *probably* would have worked.
__Sander__ has quit [Quit: Konversation terminated!]
<genesis>
http://nixpaste.lbr.uno/zQ0hCyiT?nix i get error: anonymous function at /home/genesis/devel/nixpkgs/pkgs/tools/networking/photon/default.nix:1:1 called without required argument 'urllib3', at /home/genesis/devel/nixpkgs/lib/customisation.nix:69:12
<genesis>
any idea ? i'm getting mad on that :)
<FRidh>
genesis: urllib3 and requests are part of python.pkgs
<FRidh>
don't add them as function parameters
<genesis>
oki
jtojnar has quit [Remote host closed the connection]
biopandemic has joined #nixos-dev
genesis has quit [Remote host closed the connection]
Sonarpulse has quit [Ping timeout: 260 seconds]
genesis has joined #nixos-dev
<pierron>
FRidh: any thoughts on pkgs.overrideWithScope?
<FRidh>
pierron: thanks for the PR. I had a look at it, but I need to play around with it, actually use it.
<FRidh>
some things may need to be "hidden" for users, when they are overriding themselves
<FRidh>
interpreter35Packages = let withSet = self // { interpreter = interpreter35; } // self.interpreter35Packages; in
<FRidh>
here for example, we should provide I think a function that takes a set with overrides
jtojnar has joined #nixos-dev
<pierron>
FRidh: I agree, that a shortcut function might be necessary but I do not think this should be the injectOverlay that Sonarpulse suggested
<FRidh>
Overriding packages in these package sets become a lot easier this way. For sure people are going to like your change more than the current situation.
<Sonarpulse>
pierron: I'm skeptical of putting more stuff in the custimization.nix family of things
<Sonarpulse>
injectOverlay works great until the duplicating case
<pierron>
FRidh: Also users would not see the uses of overrideWithScope, this is more likely to be in Nixpkgs, or in a few cases, when they want to fork a list of packages.
<pierron>
Sonarpulse: no, injectOverlay does not work well for the second case.
<pierron>
Sonarpulse: where foo is changed.
<pierron>
Sonarpulse: From an end user perspective this implies knowing that you have these particular sets.
<FRidh>
pierron: currently overriding the python package set is very verbose, this would be a significant improvement
<pierron>
Sonarpulse: and they have to be manipulated with the injectOverlay functions.
<pierron>
Sonarpulse: failing to use the injectOverlay function would lead to confusion to end users.
<Sonarpulse>
pierron: I'd assume every time we have a nested attribute we'd want a new scope anyways
<pierron>
Sonarpulse: so why having a function at all?
<pierron>
Sonarpulse: which is what the ideal case would be.
<Sonarpulse>
err the function would always be used to override a non-top-level thing?
<Sonarpulse>
minus the "__overlay__" hack, which is only needed for duplicating case, it's a monoid homorphism, which to a haskeller like me gives it some innate value
<pierron>
if you always have a function, then there is no need for having a function (as it would be in the ideal case)
<pierron>
So, we should go to the solution which causes the less rewrites as possible?
<Sonarpulse>
that would change the `super //` to something recursive
vcunat has quit [Quit: Leaving.]
<pierron>
That's the next step of SOS.
<pierron>
that we would have to discuss and settled on with niksnut one day.
<Sonarpulse>
then is overrideScope even needed?
<Sonarpulse>
and what do you do about the duplicate case?
<pierron>
Yes, because callPackage is not mandatorry yet.
<Sonarpulse>
but you can always do no scope on the inside
<pierron>
As soon as it is, we can get rid of overrideWithScope
<Sonarpulse>
and let the recursive override splicing fix it
<Sonarpulse>
callPackage doesn't need to go for that
<Sonarpulse>
just need the newScope calls on the inside
<pierron>
having callPackage be mandatorry is a way to ensure that every derivation expects to have a scope input for the default set of packages.
<Sonarpulse>
pierron: really, I need to think more about the about "1 big fixed point" in general
the has joined #nixos-dev
the has joined #nixos-dev
the has quit [Changing host]
<Sonarpulse>
whether I agree with that goal
<Sonarpulse>
then I'll be more useful to you talking about how
<pierron>
Sonarpulse: A single fix-point is mandatorry for being able to peal it off for making security updates as described in https://github.com/NixOS/nixpkgs/pull/10851
<Sonarpulse>
pierron: oh but I'm very sketched about how you use that for finding the runtime deps
<Sonarpulse>
I'm currently writing my strictDeps RFC which can allow use to enforce build time vs run time
<pierron>
Sonarpulse: not if callPackage is mandatorry ;)
<Sonarpulse>
I'll be sure to include your use-case :)
<Sonarpulse>
I also don't like "super.callPackage"
<Sonarpulse>
I wish for self to be the default
<Sonarpulse>
I find that much more natural
<pierron>
Sonarpulse: This we can change, but this would no longer be consistent with the rest of recommended uses of functions, and the fact that we need to "count" the number of hops through self for the security update.
<Sonarpulse>
"and the fact that we need to "count" the number of hops through self for the security update." yeah that's what I want to replace
<pierron>
Sonarpulse: honestly, I thinking of getting rid of self.callPackage and only leave it in super.
* Sonarpulse
squeels
<pierron>
Sonarpulse: I do not think you can remove counting the number of hops, because you need the packages to be evaluated to diff them.
<FRidh>
pierron: in your example, "# Duplicate a package set with a custom interpreter.", assuming there is no reference from interpreter to the package set, then why would this be different from "# Change a package dependency. Like any ordinary package, as opposed as today."
<pierron>
FRidh: because we give it a different "namespace"
<pierron>
FRidh: it would not be different if it were in interpreter27Packages.
<pierron>
FRidh: being in interpreter27ProfilePackages, we have to overriweWithScope all the packages once more, to take their future inputs from this new sub-set.
<FRidh>
pierron: I see, ok. So as soon as we reach outside the subset, we nede that
<FRidh>
ah, no, it's the namespace.
<Sonarpulse>
pierron: think instead of "self" we have "runtime" and "buildtime"
<FRidh>
but say we override interpreter27Packages, so use the current namespace, then it's not needed
<Sonarpulse>
this is roughly what my cross stuff does, and has done for quite some time
<Sonarpulse>
we just need to up the enforcement
orivej has quit [Ping timeout: 248 seconds]
<pierron>
Sonarpulse: While I agree on the intent, and I had the same idea of splitting self in 2 sets. (as mentionned in FOSDEM 2016) I do not think you can wave away the fact of having a single hop through the fix-point.
<Sonarpulse>
pierron: let normal = f normal formal; security updates = f normal alt;
<Sonarpulse>
something like that
<pierron>
Sonarpulse: h = patch (g (fix f)) (g h);
<pierron>
Sonarpulse: We do not want to compile any new packages at all, having multiple hops through the fix-point causes extra recompilations.
<Sonarpulse>
have you seen the abomination I've written in pkgs/stdenv/booter.nix btw haha?
<pierron>
Sonarpulse: yes, and this can move in all-packages too, with pkgs.overrideWithScope.
<Sonarpulse>
it's true
<pierron>
Sonarpulse: remember, I reviewd it. ;)
<Sonarpulse>
hehe :)
<Sonarpulse>
the buildPackages and pkgs thing is orthogonal to 1 vs many fixpoints
* pierron
back in 5 minutes/
<FRidh>
pierron: in a subpackage set, a `self` reference is needed that to refer to other packages in that set, as well as a `pkgs` reference for the main package set. We can't have just one because of common attribute names. withSet corresponds to one of them. If I add another lambda, how would that act with overrideWithScope?
* pierron
back
<FRidh>
or have say { python = python27; inherit pkgs; } ?
<pierron>
FRidh: the subpackage set has all the dependencies it needs from `withSet` (to be named `self`)
<pierron>
FRidh: inherit pkgs shound be banned.
<FRidh>
pierron: yes, but we have e.g. pkgs.pkgconfig, and pythonPackages.pkgconfig, which are two different package. Sometimes we need to explicitly point to pkgs.pkgconfig
<FRidh>
from within pythonPackages
<pierron>
FRidh: Then you probably need the equivalend of a `parentSelf` or to name the pythonPackagepkgconfig differently.
<FRidh>
naming packages differently is not really an option in my opinion as we're going to have this quite often
<FRidh>
yes, parentSelf or pkgs (I don't care about the name)
<FRidh>
but how to pass it to the set?
<FRidh>
considering the mapAttrs
<pierron>
FRidh: if you do not rename it, then this would be a problem for you as you will have to specify it as argument of each callPackage, and everytime you ""clone"" a subset.
<FRidh>
pierron: currently, the callPackage inside pythonPackages defaults to choosing python packages. If one needs pkgs.pkgconfig, then that needs to be explicitly passed in. That's fine.
<FRidh>
python packages take precedence
<pierron>
FRidh: you will have to use a second overlay (stage) to override every pkgconfig attribute to be the parent attribute :/
<pierron>
FRidh: you will still required `recurseIntoAttrs`
orivej has joined #nixos-dev
<FRidh>
pierron: yes, but not for now
<FRidh>
it's about the pkgs
<FRidh>
what to do with it
<pierron>
FRidh: pkgs is the proper `self` in Nixpkgs.
<pierron>
FRidh: `pkgs` is the name of `self` in all-packages.nix
<pierron>
FRidh: the `self` argument of all-packages.nix is like rec.
<pierron>
FRidh: let withSet = pkgs // { python = python27; } // pkgs.python27Packages; in
<pierron>
FRidh: one thing possible would be to give `python` as argument of `pythonPackagesSet`, which would avoid the weird { python = python27; }
<FRidh>
How do we distinguish between pkgs/self of the main set, and of the subset? let withSet = pkgs // { python = python27; } // pkgs.python27Packages; in
<pierron>
`self` has to be explicit.
<FRidh>
mapAttrs expects an attribute set, if we have python as parameter we'll have a lambda / one thing possible would be to give `python` as argument of `pythonPackagesSet`, which would avoid the weird { python = python27; }
<pierron>
otherwise we have a big `with pkgs` at the top-level which makes me sad.
<pierron>
let withSet = pkgs // pkgs.python27Packages; in mapAttrs (n: v: v.overrideWithScope withSet) (pythonPackagesfun python27 withSet);
<pierron>
FRidh: ^
<pierron>
Note that (n: v: v.overrideWithScope) expects everything to be using callPackage to provide default arguments, which is not the case of pythonPackages.
<pierron>
FRidh: ^, it might be easier to try it with simpler package set first.
<FRidh>
pierron: I don't think that really matters. I just don't get how I can get a reference to both the main package set, and the sub package set, using this approach. As I mentioned before, sometimes we need to explicitly pick a value from the main package set instead of the python package set.
<pierron>
FRidh: you cannot, because overrideWithScope will erase it.
<pierron>
FRidh: as the input has the same name.
<pierron>
FRidh: you either have to rename pkgconfig to parent-pkgconfig, or rename pkgconfig to py-pkgconfig
<pierron>
FRidh: note, that you can have the pythonPackagesSet contain allpkgs-pkgconfig = super.pkgconfig;
<FRidh>
pierron: Ok, I see. I don't think that's a good enough solution. If we were to use prefixes I think we should apply it to all items in the set. But then having to write say pythonPackages.python-numpy is a bit silly I think.
<pierron>
FRidh: what about the alias of pkgconfig?
<pierron>
pythonPackages.top-pkgconfig = … alias the top-level pkgconfig …; ?
<FRidh>
could we use mapAttrs to make all pkg attributes available like this?
<FRidh>
I suppose we would end of in infinite recursion
<pierron>
FRidh: yes, but that would be much more painful to handle in pythonPackages (I presume)
<FRidh>
we have a bunch of these cases, especially now more bindings are made available via pythonPackages
<pierron>
FRidh: let withSet = lib.mapAttrs' (n: v: nameValuePair ("_" + name) v) // pkgs.python27Packages; in
<FRidh>
pierron: what would be against it by using this to not merge the set of { allpkgs-pkgconfig = .. } // self but instead have self.pkgs
<pierron>
FRidh: the problem is that callPackages needs to know every individual package.
<pierron>
FRidh: such as libxml / pkgconfig
<pierron>
FRidh: otherwise you will have a single pkgs arguments, which would make it more painful for the future.
<pierron>
FRidh: I think having a common prefix for parent packages makes sense, such as prepending "_" each time we go deeper.
<pierron>
or appending a ' at the end, which would be less confused with special types.
goibhniu has quit [Ping timeout: 260 seconds]
<FRidh>
pierron: somehow I have an issue with overrideWithScope. I'll try again later, and will have to think a bit about what you said about the prefix.
FRidh has quit [Quit: Konversation terminated!]
<pierron>
FRidh: or with a suffix, such as an apostrophe
the has quit [Remote host closed the connection]
orivej has quit [Ping timeout: 268 seconds]
the has joined #nixos-dev
the has joined #nixos-dev
the has quit [Changing host]
pie_ has joined #nixos-dev
<gchristensen>
just wrote my second ever C++ patch, feels weird, man
goibhniu has joined #nixos-dev
phreedom has quit [Ping timeout: 250 seconds]
phreedom has joined #nixos-dev
orivej has joined #nixos-dev
sir_guy_carleton has quit [Quit: WeeChat 2.0]
lassulus has quit [Ping timeout: 244 seconds]
genesis has quit [Remote host closed the connection]
the has quit [Ping timeout: 244 seconds]
genesis has joined #nixos-dev
<andi->
gchristensen: yay! Why does it feel weird? To me it feels more natural then most languages.. Guest that depends on experience
<gchristensen>
because I don't know C++ and it makes me nervous
lassulus has joined #nixos-dev
<samueldr>
pretty sure I share a similar sentiment: the footgun is well-made and safe enough, but I don't necessarily know the most efficient way to handle it :/
jtojnar has quit [Read error: Connection reset by peer]
jtojnar has joined #nixos-dev
jtojnar has quit [Read error: Connection reset by peer]