<Profpatsch>
Even though "${null}" aborts with an error.
<dtz>
if no default value it has to exist as an attr in the calling scope
<dtz>
which at top-level nixpkgs everything is there, just not all evaluatable, but that's probably where the idiom comes from
<Profpatsch>
Sure, but that’s fine?
<dtz>
it needs to be defined even if not used
<dtz>
nix-repl> callPackage ({ a }: 1) {}
<Profpatsch>
Well, you can still set it to null manually.
<Profpatsch>
Or even better, to a stub derivation.
<dtz>
anonymous function at (string):1:14 called without required argument 'a', at /home/will/nixpkgs/lib/customisation.nix:74:12
<dtz>
ah, yes, that'd probably work if you were okay with caller setting to null explicitly
<Profpatsch>
sure, but that’s what callPackage already does, no?
<Profpatsch>
What I want to say is that we have the whole abstraction already working completely w/o null
<Profpatsch>
I’d even argue for removing null from the language if possible.
<gchristensen>
null--
<simpson>
I don't think that Nix is the right hill to fight that battle on.
<Profpatsch>
Of course then we’d have to rewrite the module system implementation, which relies on null pretty heavily.
<gchristensen>
simpson: I agree
<Profpatsch>
But at least inside packages we don’t need null.
<MichaelRaskin>
And think of a better way to remove actually-optional dependencies…
<Profpatsch>
Removing that confusion from the already pretty big list of confusions.
<simpson>
There should be an explicit pre-derivation post-module staging form for packages, and that form should allow some sort of configuration via explicit mention of which packages are optional and which are required.
<simpson>
Because `? null` is a *convention* rather than a *signature* in a lot of ways, especially inside the package, as Profpatsch says.
<simpson>
Looking at the API from the other side, I think I might want `pkgs.someArchiver.withSupportFor { lzma = true; xz = false; }` rather than an override on the full derivation.
<Profpatsch>
LnL: Not really, because you’d use callPackageNew ./foo.nix { fooSupport = true };
<simpson>
callFeaturePackage ./foo.nix { baz = true; } # foo devs say it should be on by default, I guess
<Profpatsch>
And something like callPackageNewWith ./foo.nix (self: super: { inherit (self) otherDepednency; }) { fooSupport = true; }
<Profpatsch>
simpson: You want a small overlay, like in ghcWithPackages
<Profpatsch>
So (self: super: { ghc = self.ghc810; })
<simpson>
Profpatsch: Yeah, I guess. I was just picking GHC as something that you might want to pass in. I meant to indicate that you'd have the same extra {} as with a normal callPackage.
<simpson>
But yeah, totally.
<Profpatsch>
Maybe super wouldn’t be needed.
<Profpatsch>
All packages should also not use a giant let, but a { }
<Profpatsch>
So that everything must go through the fixpoint.
<Profpatsch>
That’s quite the daunting refactor …
<Profpatsch>
Merge conflicts from hell, yes.
Lisanna has quit [Quit: Lisanna]
sphalerite_ has joined #nixos-dev
<sphalerite_>
Hi folks. I missed the discussion about the squashfuse debacle that's been going on so far. I've mentioned my preferred solution in #nixos
<simpson>
sphalerite_: Don't worry, we had immediately generalized from squashfuse to the nixpkgs-wide problem. I think that we're hoping for niksnut's input.
<sphalerite_>
Right. So IMHO the best way is to not have any customisation hooks in the package expression at all, since that's what we have override and overrideAttrs for.
<Profpatsch>
sphalerite_: If you have a fooSupport flag, you can do more than just enable/disable a dependency.
<Profpatsch>
Setting packages to null is pretty crude because it depends on the fact that the nulls passed to a derivation builder are translated to empty strings.
<Profpatsch>
Compared to an explicit lib.optional in setting the buildInputs.
<Profpatsch>
Also, you can do multiple things, like changing the builder script or passing additional stuff to another field.
<Profpatsch>
So basically having a support flag is one layer of abstraction the package maintainer gives their users.
<Profpatsch>
compared to that, package symbols are pretty arbitrary.
<Profpatsch>
And setting some to null is even more arbitrary and heavily depends on how the package is implemented and even on the behaviour of the Makefile, which could change from version to version.
<Profpatsch>
Without proper tests and feature abstraction you descend into regression nightmare-land.
<gchristensen>
ofborg can now automatically label PRs as having new packages or cleanup, if packages are deleted
taktoa has joined #nixos-dev
<dtz>
mmmmm
<dtz>
pretty soon ofborg will auto-review PR's
<dtz>
and then auto-merge
<dtz>
and then start generating them
<dtz>
muahaha
<dtz>
:P
<gchristensen>
now that is using your noggin, dtz
orivej has quit [Ping timeout: 268 seconds]
FRidh has joined #nixos-dev
ma273 has joined #nixos-dev
<MichaelRaskin>
I basically don't believe in maintainers actually exploring the full situation with optional dependencies, and there are many cases where just not passing a dependency is enough, so I think overriding with null often gives users a cheap way to quickly check if the package cares about the dependency at all.
vcunat has joined #nixos-dev
terrorjack has quit []
terrorjack has joined #nixos-dev
ma273 has quit [Quit: WeeChat 2.0]
ma27 has joined #nixos-dev
orivej has joined #nixos-dev
vcunat has quit [Ping timeout: 240 seconds]
michaelpj_ has joined #nixos-dev
ma27 has quit [Ping timeout: 240 seconds]
vcunat has joined #nixos-dev
orivej has quit [Ping timeout: 248 seconds]
<Dezgeg>
is there a reason Nix lists are arrays of Value* instead an array of Value directly?
vcunat has quit [Ping timeout: 240 seconds]
vcunat has joined #nixos-dev
ma27 has joined #nixos-dev
<MichaelRaskin>
Dezgeg: I think values are often reused, and Value is three times larger than a pointer.
vcunat has quit [Ping timeout: 240 seconds]
<Dezgeg>
do you have some pointers where this happens?
FRidh has quit [Remote host closed the connection]
vcunat has joined #nixos-dev
<MichaelRaskin>
concatLists in eval.cc
<Dezgeg>
right, makes sense. maybe it'd be worth to embed strings/lists data in the same allocation, past the Value object
<Dezgeg>
or do they get stack-allocated as well?
michaelpj_ has quit [Ping timeout: 268 seconds]
pie_ has joined #nixos-dev
ma27 has quit [Ping timeout: 276 seconds]
ma27 has joined #nixos-dev
vcunat1 has joined #nixos-dev
vcunat has quit [Ping timeout: 256 seconds]
vcunat1 has quit [Quit: Leaving.]
vcunat has joined #nixos-dev
<simpson>
What's the policy on unstable runtimes? I know that Rust and Nix have unstable versions, and I want to add an unstable version of Monte.
<MichaelRaskin>
I thought you are the person who could declare a development branch release of Monte (separate from stable releases)?
orivej has joined #nixos-dev
<simpson>
Yeah, I could, but I also want to bring in an unstable version because it'd have some useful meta tools that I want to use as buildInputs elsewhere.
<simpson>
I guess I could build all the tools, make a stable release, and PR *that*.
<MichaelRaskin>
I didn't say it should be a stable release.
<MichaelRaskin>
We have development-branch and stable-branch packaged at once for a lot of things
<simpson>
Ah, okay. Are there policies on that?
<MichaelRaskin>
Well, one is expected to package stable if it works at all, then one can package development branch if there is anyone willing to use it, then you can package something without a tag if there is a good reason, for a very loose definition of a good reason (but you should be willing to say it is a good reason if asked)
<MichaelRaskin>
I think packaging something from an untagged revision means either there is a problem in process of being fixed (dunno, a noticeable bug is fixed, but the fix is not released) or that the packager thinks upstream could have already done a pre-release.
<MichaelRaskin>
Or maybe upstream just doesn't do releases at all.
<MichaelRaskin>
If you are the upstream and you actually do have releases — make a prerelease, or a preview, or partial development release, something named.
<simpson>
Okay, cool.
<simpson>
What about IFD? Monte's got its own Nix expression. Do I copy it into nixpkgs?
<MichaelRaskin>
You can also make a really bleeding-edge overlay and host it under your own control and link it from the prerelease longDescription.
<MichaelRaskin>
I think it is safer to copy.
<MichaelRaskin>
Eventually Borg will get the evaluation sandboxed, so IFD won't work. Do you want Monte PRs in Nixpkgs testable by Borg?
<simpson>
Sure. But I also want to know what happens to my upstream development cycle.
<MichaelRaskin>
I think PRs require some explicit actions from you (and even if you eventually ask for commit rights, still pushing to Nixpkgs is an explicit action), so you could just script copying the default.nix over
<MichaelRaskin>
In any case, even with IFD you need a reference to a fixed default.nix state.
JosW has joined #nixos-dev
<simpson>
That's true. And I can force nixpkgs to do stuff like trust our binary bootstrap, rather than having all the code which does the self-hosting and self-rebuilding.
<simpson>
So maybe what's actually copied over is pretty small.
<MichaelRaskin>
By the way, if you want to have an overlay, that can actually live inside you repository and just import the same default.nix
<simpson>
Hmmm. Maybe we should look at overlays, then. That's a cool idea.
<MichaelRaskin>
Mozilla has an overlay with Firefox nightly official builds
Lisanna has joined #nixos-dev
<gchristensen>
oh jeeze, it looks like I need to setup monitoring on PRs not being evaluated :/
<MichaelRaskin>
Unless Borg can _merge_, we have a capacity bottleneck re: PRs!
<gchristensen>
right. (1) make ofborg really smart (2) send more PRs
<gchristensen>
how do we make ofborg make it even easier to choose to merge a PR?
<MichaelRaskin>
The problem is that you still need to review the code itself…
<MichaelRaskin>
I guess if we could have well-sandboxed tests on macOS, that would give more confidence in some corner cases
<vcunat>
automatically proposing updates seems OK, but reviewing them is a human work so far
<gchristensen>
I would give up good macocs sandboxing in trade for a stack of macs arriving at my door, where I can just erase them every week and not care about their security
<MichaelRaskin>
I would say that «being a straightforward version-bump» + success of all (existence is required) tests dependent on the package in question could be enough for auto-merge.
<gchristensen>
where a straight-forward version bump is -2/+2 (name/version and sha) that seem pretty workable
<MichaelRaskin>
Meaningful changes should receive comments, if only for Nixpkgs not to look as an attractive target for backdoored patches.
<gchristensen>
copumpkin: do you still have that stack of macs? are you using them? :)
<MichaelRaskin>
Yes, the change should be inside the hash string, inside the version string and inside the name string.
<vcunat>
maybe we could have some extra-unstable branch where bots would live, and review it periodically and merge to master
<MichaelRaskin>
To staging!
<vcunat>
or there
<vcunat>
we can now even automatically tell how big a rebuild it causes
<gchristensen>
probably better not staging, not run on hydra.nixos.org, and a branch which can be force-pushed
<MichaelRaskin>
Why allow force-pushing?
<copumpkin>
gchristensen: only bought one so far but could get another couple. Could probably repurpose this one for Hydra needs soon
<gchristensen>
if something nasty gets in to that branch, it'd be nice to just get rid of it
<gchristensen>
copumpkin: I was thinking for ofborg, not hydra :)
<copumpkin>
Oh that could work too
<copumpkin>
I was thinking of just sticking hydra worker in one VM and ofborg in another
<gchristensen>
oh, cool
<MichaelRaskin>
I would forbid force-pushing in the bot sandbox; if you need to clean it, maybe temporarily unlock and force-push
<copumpkin>
I need to wipe this one because I used it as a dumping ground for really shady software
<MichaelRaskin>
Because if bots can force-push…
<gchristensen>
I'd like to be able to open up access so all nixpkgs contributors can run macos builds, copumpkin, which is currently disabled because of teh sandboxing issues
<MichaelRaskin>
We can learn more about software reliability than we really wanted to ever know
<gchristensen>
so it'd be ideal if it were a system that could, say, get wiped daily/weekly
<copumpkin>
I see
<copumpkin>
Yeah with a VM that’d be pretty easy
<copumpkin>
I’ll see if I can pick up another machine or two in the next week and clean up this one so we can start experimenting
<gchristensen>
it is doing enough things and providing enough info that I think a proper UI is about in order :/
<aminechikhaoui>
gchristensen: can't help but notice that this is very close to a decentralized hydra architecture, you're so close ;)
ma27 has quit [Ping timeout: 248 seconds]
ma27 has joined #nixos-dev
<MichaelRaskin>
Except with no mechanism for proper validation of build results…
LangeOortjes has quit [*.net *.split]
tv has quit [*.net *.split]
Moredread has quit [*.net *.split]
Profpatsch has quit [*.net *.split]
GlennS has quit [*.net *.split]
{^_^} has quit [*.net *.split]
ghuntley has quit [*.net *.split]
alunduil has quit [*.net *.split]
teh[m] has quit [*.net *.split]
cbarrett has quit [*.net *.split]
ocharles has quit [*.net *.split]
manveru has quit [*.net *.split]
nocent has quit [*.net *.split]
adisbladis[m] has quit [*.net *.split]
hedning[m] has quit [*.net *.split]
primeos[m] has quit [*.net *.split]
srhb has quit [*.net *.split]
yorick has quit [*.net *.split]
clever has quit [*.net *.split]
ChanServ has quit [*.net *.split]
<aminechikhaoui>
MichaelRaskin: well one could start with a set of trusted builders
ocharles has joined #nixos-dev
ChanServ has joined #nixos-dev
manveru has joined #nixos-dev
primeos[m] has joined #nixos-dev
hedning[m] has joined #nixos-dev
yorick has joined #nixos-dev
adisbladis[m] has joined #nixos-dev
nocent has joined #nixos-dev
clever has joined #nixos-dev
srhb has joined #nixos-dev
kgz has quit [Ping timeout: 276 seconds]
<gchristensen>
20 days ago ofborg couldn't show more than 10 lines of logs either :)
ma27 is now known as Guest97638
stites[m] has quit [Ping timeout: 248 seconds]
<MichaelRaskin>
That's true; but multidimensional trust control will also increase the necessary capacity.
{^_^} has joined #nixos-dev
ghuntley has joined #nixos-dev
LangeOortjes has joined #nixos-dev
teh[m] has joined #nixos-dev
cbarrett has joined #nixos-dev
Moredread has joined #nixos-dev
GlennS has joined #nixos-dev
Profpatsch has joined #nixos-dev
alunduil has joined #nixos-dev
tv has joined #nixos-dev
<aminechikhaoui>
I think of this more as an operator of a private hydra deployment + s3 binary cache (similar to hydra.nixos.org) where I see a lot of overhead of back and forth copying of inputs/build results between the frontend and builders
regnat[m] has quit [Ping timeout: 240 seconds]
hl has quit [Ping timeout: 240 seconds]
florianjacob has quit [Ping timeout: 240 seconds]
peterhoeg has quit [Ping timeout: 240 seconds]
<MichaelRaskin>
(multidimensional — because «wishes good», «knows well» and «gives a damn» are three independent variables, and a single point of failure cannot be trusted without full confidence in all three)
dtz has quit [Ping timeout: 256 seconds]
rycee has quit [Ping timeout: 256 seconds]
olejorgenb[m] has quit [Ping timeout: 240 seconds]
pstn has quit [Ping timeout: 252 seconds]
<gchristensen>
that is true
copumpkin has quit [Ping timeout: 255 seconds]
grahamc has quit [Ping timeout: 255 seconds]
teh[m] has quit [Ping timeout: 255 seconds]
<aminechikhaoui>
so didn't think of the security aspect very well :D
nocent has quit [Ping timeout: 256 seconds]
adisbladis[m] has quit [Ping timeout: 256 seconds]
hedning[m] has quit [Ping timeout: 256 seconds]
primeos[m] has quit [Ping timeout: 256 seconds]
sphalerite has quit [Ping timeout: 260 seconds]
moredread[m] has quit [Ping timeout: 260 seconds]
<gchristensen>
but also I think aminechikhaoui isn't necessarily thinking about distributed in terms of distributed trust, but distributed in that the main hydra machine does a ton of work which could be distributed and ofborg does distribute it
<MichaelRaskin>
And when you need replication, you suddenly start really caring about bit-perfect builds, and capacity.
<ekleog>
... wait, did I miss a change? in my mind ofborg was calling nix-build on a custom machine, in what way is it a distributed-trust infrastructure?
<MichaelRaskin>
Well, if the complaint is about inefficiency of not shipping binaries around correctly, borg is currently less efficient than Hydra
<gchristensen>
ofborg is not a hydra competitor in any way
<MichaelRaskin>
ekleog: if three independently managed builds by different community members in good standing give the same result…
LangeOortjes has quit [Ping timeout: 248 seconds]
<gchristensen>
ofborg is a PR testing tool, becoming a hydra competitor would be a lot of work away
<ekleog>
MichaelRaskin: but… isn't that orthogonal to what ofborg does?
<MichaelRaskin>
ekleog: you want to say it doesn't collect the result hashes?
<MichaelRaskin>
That should be ten lines of code or something
<gchristensen>
it also doesn't collect the result at all :P
<aminechikhaoui>
right, it's just that when I think of a decentralized hydra, I see a lot of common building blocks between it and ofborg
<ekleog>
afaik ofborg “just” builds PRs to check they build correctly and (potentially) pass tests, in order to only merge commits that pass tests
LangeOortjes has joined #nixos-dev
<MichaelRaskin>
gchristensen: in a way, if there is a convincing way to certify the result without trusting reliability of your central server, distribution is another story
<ekleog>
then, distributed-without-trust builds would be great, but it would require bit-perfect reproducibility, as you say
<MichaelRaskin>
Some work into it has happenned in Nixpkgs, actually.
<MichaelRaskin>
It might be that bogr is already causing builds that would be enough to verify it for some packages.
<MichaelRaskin>
But the per-path checksums are not collected. Yet.
<ekleog>
(distribution could be handled by a notary-style approach, I guess, where you have a replicated table of input hash -> output hash on all computers that did compute it and computers that pgp-web-of-trust-like trust other builders, and then people decide whose hash-server to trust)
<ekleog>
oh, so what you mean is that ofborg's builds could serve as an independent validation of hydra's results?
<MichaelRaskin>
More a validation of reproducibility efforts
<MichaelRaskin>
Then, possibly, as a cross-validation between builders.
<MichaelRaskin>
It is a bad idea to trust a single builder («giving a damn» is hard to estimate across internet), consistent opinion of multiple people whom you kind of trust is more convenient
Lisanna has joined #nixos-dev
Guest97638 has quit [Ping timeout: 256 seconds]
<ekleog>
hmm... like a “check-this-is-reproducible” option (I seem to remember of one but can't find it again) on hydra, but using the builds already performed by ofborg as a shortcut? and then moving some trust into ofborg's own building?
<MichaelRaskin>
Yes
<ekleog>
I guess one issue is that ofborg doesn't build on the same scale as hydra, and could check only a few builds (afaik)
<gchristensen>
ofborg's fundamental infrastructure can scale to very high numbers of concurrent builders (10s of millions concurrently on a single cluster)
<gchristensen>
by which I mean, the problem there is hardware
<ekleog>
hmm... but this would mean duplicating the build effort across all builders, right? (which is actually maybe something requested for when checking reproducibility, and then all the builders would do something like pgp-sign their results before sending them to the borg master)
<gchristensen>
Nix 2 supports packages be signed by multiple keys, each builder could sign their results with their own keys and upload to a content-addressed store, when enough builders sign the same thing it could be signed by a master key
<ekleog>
wow that looks great
<gchristensen>
that could be pretty dang cool
<LnL>
that's not really now, is it?
<ekleog>
(for the part about the master key I first thought about local nix performing a check to verify enough “trusted” builders did sign the package, rather than a master key, but anyway it's the same infrastructure :))
<gchristensen>
LnL: you mean not real now?
<gchristensen>
I think Nix 2 can require packages be signed my N keys
<gchristensen>
(already)
<LnL>
new*
<gchristensen>
oh it might not be new
vcunat has quit [Ping timeout: 256 seconds]
<ekleog>
oh wow. future looks bright :)
<MichaelRaskin>
Pure numeric threshold is not really enough, but oh well.
<gchristensen>
MichaelRaskin: you must specify which keys, of course
<ekleog>
let's say that's a good first step :)
<gchristensen>
you could specify 5 keys and say everything must be signed by any 3
<MichaelRaskin>
gchristensen: you might have two AWS builders from different people who have different keys, and I would trust them but still want a non-AWS verification
<MichaelRaskin>
Or the other way round: you should have different keys for your AWS builder and your builder at office, but for many people it is still a single «you»
kgz has joined #nixos-dev
<gchristensen>
sure
<gchristensen>
that is a complicated thing to model
<ekleog>
hmm... maybe just passing the keys that signed as arguments to a boolean-returning function?
<ekleog>
this way people can go for as simple or as complex as they want
<gchristensen>
ekleog: the future is bright! I don't have immediate plans to implement any of this... but the future is bright ;)
vcunat has joined #nixos-dev
vcunat1 has joined #nixos-dev
vcunat has quit [Ping timeout: 256 seconds]
JosW has quit [Quit: Konversation terminated!]
orivej has quit [Ping timeout: 256 seconds]
vcunat1 has quit [Quit: Leaving.]
florianjacob has joined #nixos-dev
<Profpatsch>
gchristensen: You’re a BEAST
<Profpatsch>
I wouldn’t even know how to find out which attributes changed.