<samueldr>
(the traces I think should be innocuous for eval, see the last line Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS)
justanotheruser has quit [Quit: WeeChat 2.4]
justanotheruser has joined #nixos-dev
<samueldr>
this is more about learning how it works than actually doing the thing, about nativeBuildInputs and splicing
<samueldr>
here it is said that the splicing that makes `nativeBuildInputs` do the trick only works with `callPackage
<samueldr>
I tried using `callPackage ./validated-config.nix { ssh = cfgc.package; } ` (elided) in hope it would splice correctly, but it doesn't
<samueldr>
could it be (although probably the wrong terminology) that once set as an option some magic sauce is lost on the derivation so it won't try to splice it but instead use the value as is?
<samueldr>
hmm, scratch half of that thought, just tried `pkgs.callPackage { openssh = pkgs.openssh; }` and the magic sauce is lost, I think I can intuit how
drakonis1 has joined #nixos-dev
init_6 has joined #nixos-dev
<samueldr>
I guess there is, but it doesn't jump to me, but is there a reason that the derivations with special sauce included aren't directly used instead of being stashed quietly in __splicedPackages?
<samueldr>
(and, I guess it would not work equally in some cases?)
<samueldr>
yeah, figured that out (scratch half that thought)
<matthewbauer>
The way to fix the issue is making `pkgs = splicedPackagesWithXorg`, i think
<samueldr>
if a user ends up making its configuration in a weird way, and uses `let myOpenssh = callPackage ./my-openssh.nix {}` and sets the options ssh.package = myOpenssh; would that not fail at that point since it hasn't got nativeDrv/crossDrv?
<samueldr>
the output of callPackage doesn't have the splicing information in, right?
<matthewbauer>
samueldr: yeah that would still be a problem. any help on that would be appreciated! it all kind of revolves around whole package sets and we don't have a way to deal with external packages
<samueldr>
one day maybe I'll have the skills and know how for that much of a deep plunge :)
<matthewbauer>
callPackage ./my-openssh.nix probably could work with some modifications though
<samueldr>
yeah, I assume "warranty void when not using stdenv.mkDerivation" or something like that
<matthewbauer>
yeah you are always going to need callPackage, too, to get the right function args
<samueldr>
there's so much in there to unpack that I could very well not see something obvious stopping all that, so I won't say "but why isn't it ____?" :)
<matthewbauer>
but I want to work on this for 19.09 to make this experience better
<samueldr>
♥
<samueldr>
though, when I used pkgs.__splicedPackages.openssh (as a test) without callPackage it was able to figure out the right nativeBuildInput
<samueldr>
so I guess if *some* information was directly made available from mkDerivation, something might be doable?
init_6 has quit []
init_6 has joined #nixos-dev
init_6 has quit [Client Quit]
orivej has joined #nixos-dev
init_6 has joined #nixos-dev
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 258 seconds]
init_6 has quit [Read error: Connection reset by peer]
init_6 has joined #nixos-dev
init_6 has quit [Ping timeout: 248 seconds]
drakonis1 has quit [Quit: WeeChat 2.5]
lopsided98 has quit [*.net *.split]
Cale has quit [*.net *.split]
aristid has quit [*.net *.split]
ma27 has quit [*.net *.split]
orivej has quit [*.net *.split]
callahad has quit [*.net *.split]
drakonis_ has quit [*.net *.split]
justanotheruser has quit [*.net *.split]
MichaelRaskin has quit [*.net *.split]
fadenb has quit [*.net *.split]
chrisaw has quit [*.net *.split]
pie_ has quit [*.net *.split]
nwspk has quit [*.net *.split]
ivan has quit [*.net *.split]
tv has quit [*.net *.split]
_rvl has quit [*.net *.split]
hl has quit [*.net *.split]
copumpkin has quit [*.net *.split]
obadz has quit [*.net *.split]
_e has quit [*.net *.split]
andi- has quit [*.net *.split]
infinisil has quit [*.net *.split]
LnL has quit [*.net *.split]
phreedom_ has quit [*.net *.split]
etu has quit [*.net *.split]
tilpner has quit [*.net *.split]
octe has quit [*.net *.split]
qyliss^work has quit [*.net *.split]
flokli has quit [*.net *.split]
v0|d has quit [*.net *.split]
layus has joined #nixos-dev
LnL has joined #nixos-dev
copumpkin has joined #nixos-dev
asymmetric has joined #nixos-dev
sorear has joined #nixos-dev
vdemeester has joined #nixos-dev
noonien has joined #nixos-dev
_rvl has joined #nixos-dev
alienpirate5 has joined #nixos-dev
disasm has joined #nixos-dev
rsa has joined #nixos-dev
aristid has joined #nixos-dev
lopsided98 has joined #nixos-dev
ma27 has joined #nixos-dev
Cale has joined #nixos-dev
{^_^} has quit [Ping timeout: 252 seconds]
alp has joined #nixos-dev
FRidh has joined #nixos-dev
{^_^} has joined #nixos-dev
<gchristensen>
reposting for a wider audience: right now, ofborg returns a "meh" reply if a build fails. what if, when x86_64-linux builds fail it is a red X, timeouts become gray meh's, platform-not-supported is a gray-meh
<niksnut>
sounds good
<gchristensen>
I wish we could make ofborg's status check mandatory for merging, without blocking pushes
<FRidh>
gchristensen: seems fine. Does it happen regularly that PR's are merged despite failing builds? Also, would it mean every package has to be build before it can get merged?
<FRidh>
gchristensen: by the way, just merged staging-next again, but unfortunately no darwin packages for it.
<gchristensen>
yes, PRs are merged where builds fail, because it still shows up as a big green tick, despite gray boxes
<gchristensen>
I think yes it would mean every package either has to build for x86_64-linux for a merge, or be marked as not supported for that architecture
NinjaTrappeur has quit [Ping timeout: 252 seconds]
NinjaTrappeur has joined #nixos-dev
<MichaelRaskin>
Reposting from nixos-borg: what happens when someone checks a rev-dep, and it fails, and for unrelated reasons?
<gchristensen>
good question
<MichaelRaskin>
I guess the answer to it might affect which further questions are meaningful
<FRidh>
I think we should have a meta/passthru attribute for listing reverse dependencies we want tools like borg to check. I also want an attribute for describing whether major/minor updates of a derivation are permitted or not.
<MichaelRaskin>
FRidh: maybe just add them to the tests?
<MichaelRaskin>
My question is still relevant for manual requests, given the GitHub GUI
<FRidh>
MichaelRaskin: you mean the nixpkgs tests?
<MichaelRaskin>
Well, there is a passthru entry for tests (like related NixOS tests)
<FRidh>
wasn't aware that was a thing. Yes, could use that for some cases
<FRidh>
and let ofborg also do the tests aside from building the derivation?
<MichaelRaskin>
I don't remember if it is the plan or deployed reality
<gchristensen>
planned
<gchristensen>
I think it can already do it, but by request
<MichaelRaskin>
How can we make builds on Linux be fully safe for ofBorg (so that all PRs can get tested)?
<gchristensen>
perhaps it is time to just do that
<MichaelRaskin>
Well, there is that fixed-output issue
<gchristensen>
I don't want to subject people running builders to arbitrary PRs
<gchristensen>
but I could run the builders, and feel okay with that
<MichaelRaskin>
I wonder if there is a sane way to limit how much a fixed-output derivation is allowed to download
<MichaelRaskin>
Also, I guess a way to make sure FO network access is controlled without doing everything through Nix builtins could be along the line of «it is safe to give network access for derivations fully generated by the whitelisted functions»
<gchristensen>
right
<clever>
what if you just parse the derivation, and then dont run it
<MichaelRaskin>
Probably needs some support from Nix
<clever>
if it looks like a fetchurl call, use nix-prefetch-url
<clever>
if it looks like a fetchzip, use nix-prefetch-url --unpack
<clever>
and escape the heck out of the URL :P
<clever>
that will cover 90% of the fetching, is simpler then detecting if somebody override 1 attr to mess with you, and should be safe
<MichaelRaskin>
But my fetchsvn!
<clever>
fetchgit and fetchsvn, can be mapped to the same nix-prefetch-{git,svn} scripts
<MichaelRaskin>
If you want that, you could just re-call the fetcher function
<clever>
MichaelRaskin: i was thinking of something that could go over the `nix-store -qR $(nix-instantiate -A something)` and then do all the FO first
<clever>
MichaelRaskin: and then when the nix build is ran, set a flag to disallow FO, or just run it on a machine without network
<LnL>
that's an interesting idea
<MichaelRaskin>
Allow FO but do not remove network isolation is what you want
<MichaelRaskin>
And not all the FO first, but all the recognised FO
<clever>
it also gives rise to a new test you could run
<MichaelRaskin>
Because fonts
<clever>
always download the FO urls, even if the cache has a copy
<clever>
ah yeah, postFetch on fetchzip
<MichaelRaskin>
OK, now that
<MichaelRaskin>
is indeed annoying
<LnL>
postFetch is problematic but not that prevalent I think
<MichaelRaskin>
In the sense of arbitrary code that still has network access
<MichaelRaskin>
I think it is used for cleaning up generated zips
<MichaelRaskin>
Should not be too frequent, indeed
<MichaelRaskin>
At least in the raw form
<LnL>
yeah rewriting stuff before hashing is the only case I can think of
<MichaelRaskin>
Might be used by something like fetchpypi
<LnL>
doesn't that just handle the url convention?
<FRidh>
yes, just url
<MichaelRaskin>
I am neever sure which of the upstreams rebuild the archives on the fly (from time to time)
<MichaelRaskin>
And so I don't remember which of the fetchers unpack and clean up
<LnL>
generic cases could also be whitelisted
<niksnut>
we should just ban fixed-output derivations (except builtins like <nix/fetchurl.nix>) in nixpkgs and add a sandbox mode that enforces this (https://github.com/NixOS/nix/issues/2270)
<clever>
niksnut: reading that reminded me, builtins.fetchgit doesnt support submodules
<clever>
its very difficult to fetch submodules from a private git repo because of that
<MichaelRaskin>
I think it is good to have fetchers for VCSes in Nixpkgs (some upstreams just never release), and I doubt there is a reason to push all of them into Nix
<gchristensen>
100%
<clever>
MichaelRaskin: i was recently working on a gclient2nix, which would use a mix of fetchgit, fetchzip, and custom fetch functions
<clever>
and it would have to recursively traverse some python-like files in repos, to find all of the deps
<clever>
all because i was refusing to do an impure build of electron :P
<niksnut>
it's clear in retrospect that FO derivations were a mistake because they allow hiding arbitrary amounts of impurity (see fetchcargo)
<niksnut>
and the only reason they exist is that Nix didn't have a builtin mechanism to fetch tarballs
<clever>
i was designing gclient2nix to be less of a fetchcargo mess
<clever>
but it did still need some custom fetch functions
<clever>
the entire source of electron (including .git folders) was ~30gig
<clever>
and i didnt want nix to have to re-download that (hours) every time the smallest piece changed
<clever>
so i was making each repo its own fetch function
<gchristensen>
I think they're necessary, and if we want to prohibit arbitrary impurity we can easily take a route of requiring special reviews when PRing changes to F-O PRs
<gchristensen>
FO functions*
<clever>
i think gclient only kept the .git so you could `pull` and not have to redownload too much, but thats impure!
<LnL>
talking about fetchgit, do you think it would make sense to add an optional sha256 argument to the builtin to enable substitutes?
<clever>
LnL: i think that would be nice, then you have less reason to use pkgs.fetchgit
<LnL>
I have a branch for it but unlike the eval time variant I'm not sure how to add the metadata
<clever>
LnL: then, when your about to fork out the child worker, decide if its a builtin, and fork out a child-worker that just calls a c++ func, rather then execve()'ing
<LnL>
oh, that one would run in the daemon, ie. not redentials
<clever>
LnL: this happens deep within the "run builder in derivation" code, so it may even wind up in the sandbox
<clever>
i think builtins:buildenv does actually get sandboxed
<clever>
and because this is now a normal derivation, it doesnt serialize at eval time
<clever>
and it solves the ugly problem that you cant `nix-build fetch-nixpkgs.nix`
<clever>
because it returned a string with context, not a drv!!
<LnL>
sure builtins:fetchgit wasn't what I was talking about, but it could reuse the same code
<clever>
oh, but one problem i can forsee now
<clever>
builtins.fetchgit happens at eval time, in the nix-instantiate process (not the daemon)
<clever>
so it can take advantage of a users ~/.ssh/id_rsa and ssh-agent
<clever>
but builtins:fetchgit would be fully inside the sandbox, and then your back to the pkgs.fetchgitPrivate problems (with the restriction that it will never be remote)
<andi->
samueldr: you just restarted it and it went through this time?
<samueldr>
haven't verified yet if it went through, but I did restart it
<andi->
went through 8min ago
<samueldr>
those MAX* errors are not consistent when on the cusp
<samueldr>
and we've been flirting with the cusp on and off lately
<andi->
I've been hacking on some changes to how nix allocates memory (mostly around the parser).. Nothing final yet but in my FFI binding test it reduced the memory consumption by 30%... But only a few percent for nixpkgs :/
<samueldr>
not sure what's the state of this all, if it was only an experiment
<andi->
ahh, I was told there was a brnahc but must have missed that one
Guanin has joined #nixos-dev
<samueldr>
niksnut would know, though since it removes usage of boehm gc, it would act differently with the current cusp-y issue
<samueldr>
I hope something lands for memory use before 19.09 because if both stable and unstable are finnicky at getting eval'd on hydra it's not going to be so nice :/
<andi->
I have been mostly focusing on the stuff that is never GC'ed since most of our eval programs are having a very short lifetime... (at least that is what I figured).
<samueldr>
I'm not the best one to bounce stuff about all that, it's all a big black box to me :)
<andi->
the branch looks promising but I am just at the 2nd commit :D
Guanin has quit [Remote host closed the connection]
drakonis has joined #nixos-dev
<tilpner>
Are only security fixes backported to 19.03?
<tilpner>
I was usually on unstable ny now, so never looked into backporting something
<tilpner>
I wouldn't expect e.g. zfs 0.8 to be backported, but how about Synapse 1.0 (and accompanying module change)?
<FRidh>
assuming semver we could (should?) backport patch releases
<samueldr>
service-backed software, e.g. chat, streaming clients, are generally fine to backport
<samueldr>
most of the time nothing depends on them
<samueldr>
and if not for security purposes, often services stop working right with older revisions
<tilpner>
samueldr: So I'll just cherry-pick the relevant commits and send a PR against release-19.03?
<samueldr>
tilpner: sounds good to me
<tilpner>
Hmm, 19.03 pythonPackages isn't enough for that
<tilpner>
Or not
orivej has quit [Ping timeout: 244 seconds]
<manveru>
are hashes gonna change to this format?: sha256-rwTuPrS3RV61qxfpirhrDa2LhCCtOuYFMTZEpMb0lnU=
<manveru>
or is that a side-effect of using nixFlakes
<FRidh>
so after 19.09 adding support for the `hash` parameter to the fetchers in nixpkgs. We could do it now as well, but then have to make sure it's not yet used in nixpkgs.
<manveru>
how is decided which versions of nix to support in nixpkgs?
<samueldr>
IIRC for the move to 2.0 features it was two releases after 2.0 was released, but not sure there is anything written
<samueldr>
search for the PRs introducing placeholder into nixpkgs for precedents
drakonis_ has joined #nixos-dev
copumpkin has joined #nixos-dev
contrapumpkin has quit [Ping timeout: 244 seconds]
copumpkin has quit [Remote host closed the connection]