<marek>
etu: I mean, if it's redistributable, we should be allowed to carry it in the binary cache
<etu>
Sure, that makes sense
<marek>
oh no gchrholiday is away
<marek>
I wonder who could know the answer to this :)
<marek>
zimbatm maybe?
<andi->
marek: is it really redistributable?
<andi->
Doesn't that have the same limitations as the nvidia blobs that we aren't allowed to ship? On a completly different not I think we shouldn't encourage unfree software at all. If users of unfree have to compile it themself it is a fine "punishment" IMO. </foss-hat>
<niksnut>
IIRC hydra only builds redistributable firmware
<niksnut>
andi-: well, that's basically the situation now
<andi->
and I like it :-)
<niksnut>
e.g. you have to build the nvidia drivers locally
<andi->
I switched to AMD & Intel GPUs because of that NVIDIA nonsense. It worked.
<averell>
with nvidia that's fine, but if you have to compile tensorflow (rather than tensorflow-bin), that's a several-hour punishment quite regularly.
<andi->
well it depends on unfree software
<andi->
if hydra is not to compile unfree software it can't compile stuff that depends on unfree software
<averell>
ok, but it means to stay usable for every-day people we for example definitely shouldn't remove binary options
<andi->
If you really need that to be up2date daily / in sync with nixpkgs channels then you should probably run your own CI/hydra that pushes the build to a binary cache.
<marek>
andi-: yes, it is
<marek>
I mean, if it's redistributable, there is no reason for us not to ship it, right?
<marek>
also, we end up in the whole definition of what is "free" :)
ixxie has joined #nixos-dev
<andi->
Simply put: cuda software is not free software as I am not free to modify the blobs that it depends on since I do not have the code are probably not allowed to.
<andi->
*and are
<marek>
ok, but that's not the case of for example wpscan, which you are free to mofidy it, but not to redistribute if it's for commericial usage
<marek>
there are several cases like that
<marek>
we should consider what *we* are doing, not the user, for example, we do redistribute and that's okay, but not every use case might match the license, so it's up to the user
<marek>
we should not stop shipping a software because for some percent of users the license does not apply
<marek>
and if someone wants to be *really* free, they can use guix :D
<andi->
uff, not really want to have that conversation.
* marek
nods
Jackneill has quit [Ping timeout: 245 seconds]
Jackneill has joined #nixos-dev
<nbp>
domenkozar[m]: I do not think we encourage people to use IFD, quite the opposite in fact.
<domenkozar[m]>
nbp: sorry?
<domenkozar[m]>
I said we don't encourage
<domenkozar[m]>
(I hope)
<nbp>
domenkozar[m]: oh, I thought you said the opposite in your blog post.
<yorick>
"We encourage more language tools to take advantage of this feature."
<domenkozar[m]>
we (CI) encourage use of IFD
<domenkozar[m]>
Hydra doesn't
<nbp>
domenkozar[m]: “Some Nix tools already embrace IFD […] We encourage more language tools to take advantage of this feature.”
<yorick>
I try to avoid it since it makes instantiating really annoying
<domenkozar[m]>
We, as in creators of Hercules CI
<domenkozar[m]>
we encourage it because it's very well supported, contrary to Hydra
<nbp>
However, it should remain forbidden within Nixpkgs.
<yorick>
nbp: callHackage uses it in nixpkgs
<clever>
yorick: and callCabal2nix too
<yorick>
but I agree it shouldn't end up in any nixpkgs package
<domenkozar[m]>
> Using different platforms (typically Linux and macOS) during IFD is one of the reasons why upstream forbids IFD, since their evaluator is running on Linux and it can’t build for macOS.
<{^_^}>
error: syntax error, unexpected ',', expecting ')', at (string):263:112
<roberth>
providing IFD-based tools is one thing, depending on it another
<domenkozar[m]>
> Our CI dispatches all builds during IFD back to our scheduler, so it’s able to dispatch those builds to either specific platform or specific hardware.
<{^_^}>
error: syntax error, unexpected ',', expecting ')', at (string):263:62
<domenkozar[m]>
for nixpkgs it makes sense, since software moves slowly
<nbp>
oh, the reason is simpler than that, looking for package names should not build anything.
<domenkozar[m]>
in development world, you need to constantly update Nix expressions if you don't use IFD
<domenkozar[m]>
well with flakes you can cache that
<domenkozar[m]>
or with any kind of update system in general
<roberth>
Nixpkgs has very specific requirements like name-based search
<yorick>
domenkozar[m]: no, we just look at the stack.yaml
<roberth>
many projects don't need that and benefit from the ease of use that IFD provides
<yorick>
(from nix)
<nbp>
I really against flake, until we prove it works within Nixpkgs.
<domenkozar[m]>
I don't think it's mean to be used within nixpkgs
<nbp>
it should!
<domenkozar[m]>
meant*
<nbp>
otherwise this is yet another non-reusible format.
<domenkozar[m]>
well the goal is to better support poly-repo
<domenkozar[m]>
so I don't see why it should be used within a repo
<domenkozar[m]>
nixpkgs itself could be broken down into mutiple repos, do you mean that?
<nbp>
The reason why it should, is that if you think of the recipe of a package as something which is provided by the package, then Nixpkgs is just an aggregate of these recipes.
<nbp>
Why should we express packages recipe differently in Nixpkgs than in Flakes.
<nbp>
Thus Nixpkgs should be composing Flakes.
<domenkozar[m]>
well essential flakes are solving the same problem as recursive Nix
<nbp>
If Flakes don't work in Nixpkgs, then this is a bad format and we should avoid it.
<yorick>
I think distros should have a final say in what/how software gets included and we should not leave packaging to the upstream (otherwise you get bloated spyware)
<nbp>
Recursive Nix is completely different than Flakes.
<nbp>
Flakes attempt to replace the shell.nix file.
<domenkozar[m]>
the implementation is different, but it solves a similar pain
<domenkozar[m]>
you have for example nixops with Nix expressions that are in the repo, then duplicated in nixpkgs
<domenkozar[m]>
that's the pain
<domenkozar[m]>
you can solve that with recursive Nix or flakes
<Taneb>
Or with IFD
<nbp>
flakes has nothing to do with IFD, nor Recursive Nix.
<domenkozar[m]>
exactly, that why we added support for it.
<domenkozar[m]>
but you can't use IFD on Hydra, while you will be able to use flakes
<aminechikhaoui>
I think I remember there is also some security problem with enabling IFD ? that would require restrict-eval to be false I think
<domenkozar[m]>
so flakes could work for solving this :)
<aminechikhaoui>
don't remember very well the problem
<domenkozar[m]>
aminechikhaoui: iirc that can be (was?) solved
<aminechikhaoui>
maybe, that's why I said I don't remember very well :)
<domenkozar[m]>
yeah I also don't :)
<aminechikhaoui>
does hercules-ci evaluate derivations on the frontend side or in the agents ? I guess it's a totally different architecture compared to hydra
<aminechikhaoui>
man I always need to look up the name of the CI :D
<michaelpj>
I agree that it's annoying to build stuff in order to work out what you can build, but I think it unfortunately reflects the reality of the situation, so long as we're converting other build tools' specifications into Nix
<domenkozar[m]>
aminechikhaoui: on the agents
<nbp>
I would not recommend Flakes, because of the gap between Nixpkgs and flakes which should not exists.
<domenkozar[m]>
nbp: I don't think you're against flakes in general, but the part that it manages dependencies?
<nbp>
I am not against the principle of having a standalone file to build project.
<nbp>
I am against the fact that we accept its flaws, while knowing they exists.
<domenkozar[m]>
aminechikhaoui: it was designed for evaluations not being a bottleneck, so that gives more space for IFD
<domenkozar[m]>
nbp: how would you solve the problem flakes are solving in nixpkgs?
<nbp>
By making a single file out of what we put in release.nix / default.nix / overlay.nix and shell.nix.
<nbp>
Then we can figure out incrementally how to make it simpler.
<nbp>
This is better, this applies to Nixpkgs as well, and follow the same principles as SOS.
<domenkozar[m]>
these changes are new syntax
<nbp>
The new syntax is just a way to express it.
<nbp>
This does not add any new semantic to the Nix language.
<domenkozar[m]>
I think there are different goals
<nbp>
What this synyax express can already be expressed in Nix.
<domenkozar[m]>
the goal of that document was to fix some shortcomings of Nix language as it's too basic
<domenkozar[m]>
while flakes solve dependency management and semantic atttributes instead of semi-semantic files
<nbp>
Flakes is equivalent to release.nix / shell.nix and default.nix, do we agree on this point?
<domenkozar[m]>
well that's part of it, but it adds much more
<nbp>
Today, the default.nix which live within project can be imported in Nixpkgs if we wanted to, and the opposite is true as well.
<domenkozar[m]>
like a way to identify what is a check (test)
<domenkozar[m]>
and versioning of the language (editions)
<nbp>
Flakes do not offer this possibility
<nbp>
Flakes is literally a different package format which does not help Nixpkgs.
<clever>
nbp: yeah, for simple projects, default.nix can be moved to nixpkgs with minor changes, and for more complex projects, i try to style default.nix after all-packages.nix, so i then have a foo.nix that can be dropped into nixpkgs
<michaelpj>
there's also all the registry stuff
<nbp>
This could be shared effort, instead of wanting to have a something alien to Nixpkgs.
<domenkozar[m]>
I see what you're saying, we should rely more on IFD, I agree :D
<nbp>
^ I do not agree on the IFD part.
<clever>
i think recursive nix would help complement things more, for example, i want the option to replace gcc with nix-build, inside a nix build
<clever>
if you can just `nix-build gcc.nix --arg source ./foo.c`, then you get all of the benefits of ccache
<clever>
by reusing existing logic in nix
<clever>
and it would be even better then ccache, because the nix sandbox can enforce purity
<clever>
build slaves also give it distcc like features at the same time
<adisbladis>
Quoting myself: "IFD is such a nice footgun"
<nbp>
I would agree on IFD if-and-only-if you could prove that the generated Nix script only depends on its inputs and is reproducible.
<clever>
nbp: the derivation your importing from still has to obey the normal nix rules, so it must either be based on its own inputs, or be fixed-output
<domenkozar[m]>
exactly
<nbp>
(random)
<domenkozar[m]>
so if you disallow impurities, IFD is also pure
<clever>
`import (pkgs.fetchFromGitHub args) {}` is fixed-output, so its both pure and cachable
<nbp>
Nix does not prove reproducibility, it checks it when asked, but does not garantie anything.
<clever>
`callCabal2nix "name" (fetchFromGitHub args) {}` depends on the specific build of cabal2nix, and a fixed-output
<domenkozar[m]>
IFD allows you to purely generate Nix expressions, so you don't have to :)
<infinisil>
Yeah so IFD is only pure with fixed-output derivations
<domenkozar[m]>
trust me, I worked on a project with 30 people, it's a pain.
<clever>
and builtins.exec is IFD's ugly cousin that nobody talks about :P
<nbp>
What ?!
<nbp>
Who introduced this function?
<adisbladis>
Tbh I'd rather spend the effort to ingest various lock-files in Nix than resorting to IFD
<michaelpj>
adisbladis: there are way too many lock-file formats for that!
<infinisil>
There's the commit that introduced it :)
<nbp>
Good!
<clever>
nbp: basically, it will take an argv array, execute it at eval time (as whatever user is doing the eval), and then parse whatever was on stdout as a nix expr, and return that parsed expr
<nbp>
at least you have a large command line footgun to aim at your-self before running it.
<clever>
nbp: related, there is another builtin protected by the same safety, that lets you load a dynlib directly into nix itself
<adisbladis>
domenkozar[m]: Because I don't like the implications of IFD.
<clever>
nbp: which can then return any nix value, including a new primop or attrset
<adisbladis>
My opinion may change if recursive Nix becomes a thing
<domenkozar[m]>
adisbladis: can you be more specific?
<adisbladis>
domenkozar[m]: Like the intermediate derivation becoming GCd
<domenkozar[m]>
those count as build derivations
<clever>
nbp: so i could just `fizzbuzz = builtins.importNative("/path/to/libfizzbuzz.so");` and then use the function as i see fit
<clever>
nbp: and IFD likely works here, so nix can build that for itself
<domenkozar[m]>
adisbladis: either way, Nix could handle it differently if wanted
<clever>
domenkozar[m]: thats a bug with hydra, its not aware of the IFD level dependencies, so it doesnt GC root them, and doesnt upload them to the cache
<domenkozar[m]>
in hercules ci we track all IFD builds during evaluation
<domenkozar[m]>
so we upload it all to cache, etc
<domenkozar[m]>
ok but then it's not IFD, it's hydra
<domenkozar[m]>
(like always)
<adisbladis>
domenkozar[m]: Yeah. some of my grievances can be fixed.
<adisbladis>
Also IFD impacts discoverability
<clever>
and IFD forces builds at eval time
<adisbladis>
I cant fire up a repl and just start exploring packages
<adisbladis>
Yeah, it's ugly
<clever>
ive seen hydra take 6 hours to do an eval, because it depended on ghc at eval time
<clever>
and it gives zero progress information when doing that
<adisbladis>
"yay"
<domenkozar[m]>
also a property of hydra, not IFD
<clever>
thats why i try to avoid depending on complex builds at IFD time
<infinisil>
clever: I've mentioned before how it would be really neat if you could use pure haskell functions from Nix, something like `builtins.haskell "f :: NixExpr -> NixExpr; f = ..."`. This would enforce SafeHaskell so it's actually pure
<michaelpj>
not doing IFD just means moving the computation to "human time" beforehand to generate the Nix. Which is error-prone and requires people to commit big generated files
<michaelpj>
surely we can do better mechanically
<clever>
infinisil: builtins.importNative would let you experiment with that, without having to recompile nix itself
<domenkozar[m]>
^^
<domenkozar[m]>
michaelpj: I don't think everyone has the same experience as we do, but once you do, you're there :)
Jackneill has quit [Ping timeout: 268 seconds]
<infinisil>
clever: Hm yeah, but that would also need an --unsafe flag or so
<infinisil>
The builtins.haskell thing can be enabled by default because it's actually safe
<domenkozar[m]>
the experience of working on a code that's modified hour-to-hour and using Nix leads to some different pains than 1000 commiting to nixpkgs
<clever>
infinisil: the importNative is more about testing it locally, before you commit to adding it into nix
* infinisil
nods
<niksnut>
nbp: I don't really understand your criticism of flakes, which seems to be that it doesn't solve a problem it's not intended to solve
<niksnut>
saying "it doesn't work inside nixpkgs" is like saying that crates don't work inside rust modules
<niksnut>
flakes are basically just a way to compose repos and then have hermetic evaluation
<nbp>
niksnut: The problem I have with flakes, is that Nixpkgs is a collection of packages, and flakes attempt to solve the solution for a single package.
<niksnut>
no, that's a misunderstanding
<nbp>
niksnut: Thus my question is why would it be different than what composes Nixpkgs.
<niksnut>
flakes can have as many packages as you like
<niksnut>
or no packages
<niksnut>
flakes don't care what nix expressions are in them
<nbp>
ok, so flakes are a set of packages, like we have in Nixpkgs, why would it be different than Nixpkgs then?
<nbp>
My concern is that Flakes would be one thing, and Nixpkgs would be another with no sharability between the 2.
<nbp>
And that would be a shame.
<niksnut>
no, they're a standardized interface to a set of nix exprs, which could be packages, modules, overlays, CI tests, ...
<etu>
I guess one could consider nixpkgs a big flake?
<niksnut>
yes, nixpkgs is a flake
<ryantm>
It's one of the main examples in the RFC.
<etu>
Yeah, I think flakes is a brilliant idea because it makes it more modular since flakes can depend on specific versions of other flakes.
<nbp>
Then why do we lose Nixpkgs properties of composability?
<nbp>
IF this is only a wrapper with types, we should not lose any of the features we already have in Nixpkgs.
<infinisil>
Huh, why would we lose anything in nixpkgs?
<nbp>
That's what is mentioned in the RFC.
<niksnut>
what do you mean with composability? overlays?
<aanderse>
there seems to be a fair bit of mention on how everything can be split out into separate repositories with flakes. i hope that "nixos core" wouldn't become a crippled os that doesn't do anything on its own, and to make it useful you have to add a bunch of different flake repos/whatever to it :\
<nbp>
Also, if Nixpkgs is a flake, and it is composed of project A, B, and C. IF each project A, B and C are flakes too, then Nixpkgs is a flake of flake ?
<nbp>
niksnut: I mean overideDerivation, override, and others. Yes.
<nbp>
aanderse: That's my hope as well, but apparently this is not what flakes are about, IIUC.
<niksnut>
nbp: sure, nixpkgs can depend on other flakes and re-export some of their stuff
<niksnut>
for example, nixpkgs/lib would be a candidate for turning into a separate flake
<aanderse>
nbp: yeah i guess i just have concerns that since nixos is a very fast paced and modern (feature wise) os this reflects on developer attitudes, and maybe developers will be more likely to say "why waste my time with nixos core when i can just do all my work in my own flake!"
<niksnut>
nbp: there are no overrides at the flake level at the moment, since functions like .override are killing us wrt garbage collection
<aanderse>
nbp: this doesn't mean that flakes cause the problem... but they are one method which makes it easier for a problem like that to develop
<niksnut>
but the outputs of flakes (like packages) can have overrides
<nbp>
niksnut: I wish to see how this could be done.
<niksnut>
I thought about enforcing that outputs.packages don't have overrides to ensure memory-efficient evaluation, but that's not the case yet
Jackneill has joined #nixos-dev
<nbp>
niksnut: especially without having `include nixpkgs {}` in each flake.
<nbp>
niksnut: one remark about
<niksnut>
well, you need to specify a dependency on nixpkgs if you use it in your flake, sure
<nbp>
NixOS avoid this issue by merging the declaration and failing if declaration do not define the same thing.
<niksnut>
which is very tricky in the context of the nix evaluation model
<niksnut>
the problem is that declarations like builders.unixPackage have no identify, they're just a value
<niksnut>
so you can't distinguish between multiple inherits of builders.unixPackage, vs. single inherits of 2 different parents that happen to declare conflicting fields
<nbp>
Yes, which is why NixOS module kind of ignore the problem.
<niksnut>
so we'd have to change the language to add some concept of top-level declarations
<niksnut>
and not have everything be an expression
<nbp>
why so?
<niksnut>
then you can have a decl like "class builders.unixPackage" so it has an identity
<nbp>
Why not just making the identity using the function that compose it?
<niksnut>
?
<nbp>
id = hash(attributes)
drakonis has joined #nixos-dev
<nbp>
the problem I see, is that anything which might want to just add a name propery would have to `extend lib.module.name`, in order to avoid conflicts with other name properties when merging.
<nbp>
The other option is to keep all of them in separated namespace.
<niksnut>
hash(attributes) is tricky because then you need to decide whether this should evaluate attributes deeply
<nbp>
Thus extending with multiple would actually have each entity existing.
<niksnut>
also attributes could contain functions
<nbp>
YEs, I meant the hash of the AST.
<nbp>
having different namespace would allow to disambiguate if needed. pkg.builder.name instead of pkg.name
<nbp>
but a definition would be used as a definition to all name option.
<niksnut>
that's the thing, in the current evaluation model you don't get to see "pkg.builder.name", just the value that attribute evaluates to
<niksnut>
you could even inline in
<niksnut>
e.g. 'let x = E; in < extend x; ... >' is equivalent to '< extend E; ... >'
<nbp>
maybe we could do like the with statement. `extend builder with { name }`
<nbp>
in which case, the other side would have to be defined based on the exposed options.
<nbp>
such as: `let a = < name | … >; b = < name | … >; in < extend a with {} b with { name = name; } >;` <-- inherit name from `a`, and copy the name result to `b.name`
<nbp>
<extend a b; > would raise an error as a.name and b.name collide.
orivej has quit [Ping timeout: 244 seconds]
orivej has joined #nixos-dev
peti has joined #nixos-dev
<peti>
niksnut: hydra.nixos.org appears to have a scheduling problem. Almost all builders are idle even though the queue is pretty long.
<hyperfekt>
Any feedback would be very welcome (here too if you want lower friction or to have a discussion)! https://git.io/fjjqj