<qyliss>
So you end up building with some random blob that was built with network access, that probably isn't reproducible
<qyliss>
Changes in dependencies are constantly changing the outputs of fixed output derivations
<qyliss>
Nobody even notices, generally.
<Profpatsch>
We know that fetchurl/fetchgit are well-behaved.
<Profpatsch>
They are pretty simple and you can understand what they do.
<tilpner>
Ahh, right. And the alternative are generators that write Nix expressions that use better-behaved fixed-output fetchers?
<Profpatsch>
buildRustPackage uses cargo as its fetchurl.
<Profpatsch>
And cargo does whatever. Updates to cargo change its interfaces. But you need to change the output hash to even notice.
<Profpatsch>
So the *only* time where a fixed output derivation is reasonable is if the fetcher will *never* *ever* change its behaviour/output in the future.
<Profpatsch>
In all other cases they are just a hack because people are lazy and software is bad (oh so bad)
<manveru>
i really think this is a legit use-case, but i also understand that people can easily abuse it
<Profpatsch>
manveru: Hm, you could write a nix plugin instead
<Profpatsch>
Maybe we just need some pushing people in the right direction so they stop abusing fixed outputs.
<manveru>
how's a nix plugin better? :P
<manveru>
sorry, i know nothing about the plugins or how they're run
<manveru>
but i think they have to be C++ as well, so not sure how that makes things easier
alp has quit [Ping timeout: 258 seconds]
<Profpatsch>
yeah, tru
<qyliss>
Has anyone tried porting stdenv to Bash 5 yet?
<qyliss>
I ask because my attempt to bootstrap using native stdenv is failing because my system's bash is too new.
<qyliss>
But would probably be nice for less niche use cases too.
alp has joined #nixos-dev
pie_ has quit [Ping timeout: 258 seconds]
<timokau[m]>
Profpatsch Profpatsch re "nobody notices": as I mentioned in https://github.com/NixOS/nixpkgs/pull/63645, apparently fixed output derivations *do* get rebuilt when their dependencies change. I remember that differently too, but that's what I saw when I tested it.
<{^_^}>
#63645 (by timokau, 5 hours ago, open): bazel: add fixed output test case
<Profpatsch>
timokau[m]: How would they? If their output hash (which is the input hash) is already in a binary cache somewhere, they are not rebuilt.
<Profpatsch>
Otherwise they’d have to be rebuilt all the time.
<timokau[m]>
Oh, it probably just rebuilt the package depending on the FOD and I confused the two
<timokau[m]>
But still, its not impossible in principle. Nix could use the same mechanism it always uses to determine when to rebuild something. Would probably make dealing with them a lot easier.
<timokau[m]>
It would then only rebuild the FOD, not all its reverse-dependencies (since the result of the FOD still has the same hash if everything worked right)
<timokau[m]>
Then if I change patchutils or bazel, I can just `nix-review` it and get notified when hashes change (and what the new hashes are)
<timokau[m]>
Profpatsch: I feel like the distaste for FOD is justified, but I wouldn't take it to the extreme you and qyliss do
<Profpatsch>
timokau[m]: I don’t see how we should teach nix to distinguish. It’s just not possible.
<Profpatsch>
The fixed input derivation can do arbitrary computation & arbitrary networking.
<timokau[m]>
Why wouldn't it be possible?
<Profpatsch>
And the hash is the only way to move from dirty imperative code to declarative.
<timokau[m]>
Yes, but if you assume the network to be pure you can treat the computation the same as you do with other derivations
<Profpatsch>
So if you don’t change the hash, nix has no way to see whether something changes (it’s the halting problem)
<Profpatsch>
You already do, that’s what the hash is for.
<Profpatsch>
You assume your fetcher & the network would always return the same binary if run again and the hash didn’t change.
<Profpatsch>
There’s nothing different you can do.
alp has quit [Remote host closed the connection]
<Profpatsch>
And by giving it a fixed output, you *promise* the fetcher always returns the same file.
alp has joined #nixos-dev
<Profpatsch>
Which is completely bonkers to do with an ill-designed piece of software like bazel workspaces.
<Profpatsch>
Or go modules for that matter.
<Profpatsch>
Or even rust crates
<timokau[m]>
You could at first treat it like any nix derivation and hash its inputs. Then it will be re-built when the inputs change. You can then hash the output and compare that to the output that was given with the fixed output derivation. Complain if they don't match.
drakonis_ has quit [Ping timeout: 244 seconds]
<timokau[m]>
That way it will have to be rebuilt every time one of the inputs change (assuming the network is pure, nothing we can do about that), but the hash will only need manual changes when the result changes.
drakonis_ has joined #nixos-dev
<timokau[m]>
Profpatsch: ^
<timokau[m]>
That'd also resolve my long running annoyance of having to come up with a false hash every time I do TOFU. Instead I could just leave in the old hash, update the url and nix would rebuild.
<manveru>
yeah, that'd be much more intuitive :)
<Profpatsch>
timokau[m]: Yeah, and good look doing that for every single source, because fetchgit needs git.
<{^_^}>
#48567 (by nh2, 35 weeks ago, open): fetchpatch doesn't notice changes to other arguments
<timokau[m]>
Profpatsch: what's the problem with fetchgit needing git?
<Profpatsch>
timokau[m]: That we invalidate our whole frigging source cache every time git changes
<timokau[m]>
ekleog: Thanks! I think my suggested approach would nicely fix that issue
<Profpatsch>
fetchgit is a fixed output derivation as well.
<Profpatsch>
Maybe that’s a bad example, because it moved into a builtin, but the argument still stands
<timokau[m]>
Profpatsch: Well we would need to rebuild it every time git changes, yes. But the actual results wouldn't change, so the rebuilts would stop there. I don't see the problem with that in principle. Just like we rebuild everything every time stdenv changes, we'd rebuild all the sources every time git changes. Seems sensible to me.
<Profpatsch>
timokau[m]: No, it really isn’t.
<Profpatsch>
Because we have a source cache.
<ekleog>
timokau[m]: downloading that much data at each git-or-one-of-its-dependency-upgrade is going to get hydra blacklisted everywhere
<Profpatsch>
Which today guarantees us that all fixed output derivations are binary-reproducible, even from years ago.
<Profpatsch>
That’s the whole deal, fixed output will never re-download.
<Profpatsch>
Which brings us to the first point, using them for anything non-trivial is abusing fixed outputs
<Profpatsch>
s/first point/original point/
<timokau[m]>
Profpatsch: As long as there is some deduplication, caching should still work
<timokau[m]>
ekleog: Good point
<clever>
2019-06-22 11:07:30 < timokau[m]> That'd also resolve my long running annoyance of having to come up with a false hash every time I do TOFU. Instead I could just leave in the old hash, update the url and nix would rebuild.
<clever>
timokau[m]: if you include the version in the name, you dont have to do that
<clever>
timokau[m]: that tar file, is fetched in a derivation with the name "hello-2.10.tar.gz"
<clever>
so if you change the name/url to hello-2.11.tar.gz, it wont find the old file, even using the same hash
<timokau[m]>
clever: Oh? That's a neat trick. We should make some of the common fetchers like `fetchFromGitHub` do that by default
<clever>
> multimc.src.name
<{^_^}>
"source"
<clever>
timokau[m]: i dont remember exactly why, but it was decided to name fetchFromGitHub plain "src"
<clever>
source*
<timokau[m]>
Ideally even do a lite version of what I proposed above and have an opt-in list of inputs that can influence the output hash, that'd solve the problem more generally without having to clobber the name
<Profpatsch>
“solve”
<timokau[m]>
clever: Would be great to name it something like `github-owner-repo-rev` instead
<Profpatsch>
It’s another hack around hacks
<timokau[m]>
I feel like you're a bit unnecessarily antagonistic. It would be better than the status quo.
<clever>
timokau[m]: in 16.09, fetchFromGitHub had name ? "${repo}-${rev}-src"
<clever>
timokau[m]: almost exactly what you said
<timokau[m]>
clever: Any clues as to why it was changed?
<clever>
timokau[m]: something to do with sharing the $out with builtins.fetchTarball i think, checking git history now
<Profpatsch>
timokau[m]: I have a strong distaste against adding corner cases for workarounds
pie_ has joined #nixos-dev
<clever>
Bisecting: 19638 revisions left to test after this (roughly 14 steps)
<qyliss>
Having to opt in to inputs that could change the hash would defeat the entire purpose of nix
<qyliss>
Which is to make sure that outputs always correspond to inputs
<timokau[m]>
clever: Thanks for digging that up! Although I don't quite understand why its so important to have the names match up (and I don't want to watch an entire NixCon talk to find out)
<timokau[m]>
qyliss: It'd just be a convenience for fixed output derivations, not the default for all of nix. It could fix some everyday annoyances, even if other problems remain. I don't see the downside of that (besides that someone would have to implement it).
<clever>
timokau[m]: the exact value of $src inpacts the build, so if builtins.fetchTarball is creating a different path from pkgs.fetchFromGitHub, then you cant share the resulting build
<timokau[m]>
`fetchurl` could depend on the url, `fetchFromGitHub` on repo and rev etc. Would be a nice quality of live improvement I think
<timokau[m]>
clever: Why would you want to use the two interchangably?
<clever>
timokau[m]: a hydra configured to download the latest X and build it every hour
<timokau[m]>
clever: The latest what? Wouldn't that still either use `fetchTarball` *or* `fetchFromGitHub`? Why would it sometimes use one, sometimes the other?
<clever>
timokau[m]: hydra internally uses nix-prefetch-git, which i think also calls the thing source
<timokau[m]>
clever: Okay, I'd probably need to dig deeper into the internals of hydra than I'd like to to figure out why that is a good idea / necessary
<niksnut>
Profpatsch: amen, we could start by banning buildRustPackage etc from Nixpkgs
<niksnut>
or ban all fixed-output derivations except fetchurl and fetchgit
<clever>
niksnut: what about adjusting builtins.fetchgit, pkgs.fetchFromGitHub, and nix-prefetch-git, to all name the output "source-${rev}" ?
<niksnut>
why?
<clever>
2019-06-22 11:07:30 < timokau[m]> That'd also resolve my long running annoyance of having to come up with a false hash every time I do TOFU. Instead I could just leave in the old hash, update the url and nix would rebuild.
<clever>
niksnut: because it would invalidate things when you change only the rev, and avoid the problem of the sha256 reuse breaking things
<timokau[m]>
Also much nicer logs and error messages for nix-build (especially if you include repo as well as rev) pretty much the original reasoning for https://github.com/NixOS/nixpkgs/pull/5521
<{^_^}>
#5521 (by shlevy, 4 years ago, merged): fetchgit: give output a nicer name
<clever>
but builtins.fetchgit should be able to query the rev it just fetched, and name things "source-65b5f177b5fbb1b0778ede047a13a3cee9c59cfe" instead
<niksnut>
that requires having a rev
<niksnut>
which isn't always the case
<niksnut>
for example in fetchTarball
<clever>
it just ran git to clone a ref, and can then just ask git what that ref pointed to
<clever>
yeah, that wont have a rev...
<clever>
but in what cases are you going to be switching between fetchgit, fetchTarball, and fetchzip, and fetchFromGitHub...
<niksnut>
probably not
<clever>
niksnut: for the same reason others want nix to work purely on a sha1 rev, i was thinking just "source-${rev}" is enough, repo name and owner name dont matter, but still keep the sha256 as well for obvious reasons
<timokau[m]>
clever: If we're already including info about the repo, why not also include the name? Would make log outputs much nicer. Only negative I can think of is that it is probably a bit annoying to figure out in `fetchgit` as opposed to `fetchFromGitHub`, but should be doable right?
<clever>
timokau[m]: nixpkgs and nixpkgs-channels wont match up if you do that
<timokau[m]>
clever: That is assuming that `nixpkgs` itself is used in a fixed output derivation *and* is for some reason interchangably used with `nixpkgs-channels`. Is that actually the case some where?
<clever>
timokau[m]: its an input to things like the manual creation
<clever>
timokau[m]: and if you are fetching the same rev, it doesnt matter which repo you got it from
<timokau[m]>
clever: And why would it be interchanged with `nixpkgs-channels`?
<clever>
timokau[m]: if the rev is the same, then the contents of the dir are the same
<timokau[m]>
clever: Fair enough. I think better logs are more valuable than that particular edge case, but thats matter of opinion
<timokau[m]>
clever: Yes, including the name would just be about the `nix-build` output
<clever>
timokau[m]: you can also `nix show-derivation` on the path, to see how the path was fetched
<timokau[m]>
I don't think the use case of wanting to change the source repository is common enough to worry about it, but as I said that's matter of opinion
<timokau[m]>
clever: Good to know. Still, I think the UX would be better if the store path was meaningful in the first place and it would be worth the tradeoff of not being interchagable with other repository names
<timokau[m]>
Anyway, just having the rev would be a major improvement
<MichaelRaskin>
Well, for actual readability of the build/fetch plan it would be better to have the repository name
<timokau[m]>
^ That's what I mean
bgamari has quit [Ping timeout: 252 seconds]
bgamari has joined #nixos-dev
orivej has joined #nixos-dev
drakonis has joined #nixos-dev
drakonis has quit [Read error: Connection reset by peer]
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 248 seconds]
orivej has quit [Ping timeout: 248 seconds]
drakonis has quit [Read error: Connection reset by peer]
drakonis_ has joined #nixos-dev
alp has quit [Ping timeout: 258 seconds]
drakonis has joined #nixos-dev
alp has joined #nixos-dev
drakonis_ has quit [Ping timeout: 259 seconds]
pie_ has quit [Remote host closed the connection]
pie_ has joined #nixos-dev
Guanin has joined #nixos-dev
justanotheruser has quit [Quit: WeeChat 2.4]
justanotheruser has joined #nixos-dev
drakonis has quit [Ping timeout: 250 seconds]
drakonis has joined #nixos-dev
<qyliss>
I guess before we can start getting rid of buildRustPackage etc we'd have to make sure there were alternatives?
<qyliss>
idk what state carnix is in these days
<LnL>
ofborg uses carnix, not sure how useful it is in nixpkgs however
emily has joined #nixos-dev
alp has quit [Ping timeout: 258 seconds]
orivej has joined #nixos-dev
<ekleog>
there have been debate about it generating too large swathes of nix code for them all to be checked in
<ekleog>
guess solving that would require either some kind of nix-in-nix/ret-cont-nix/IFD/similar or some kind of refactor to get carnix to share dependencies when they overlay, like what (I think?) we do for python & co
<ekleog>
and/or just do an RFC and get enough people to agree that the size bloat is not a meaningful issue
<qyliss>
what's the issue with the large generated code?
<qyliss>
but yeah, maybe this is RFC territory
<infinisil>
ekleog: qyliss: I thought the idea was to have it like haskell/node-packages, a single big file generated by a tool, used to generate all packages of that language
<LnL>
yeah, I like the haskell model for nixpkgs
<qyliss>
mmm yeah
<qyliss>
although that doesn't work when binaries aren't published on that language's package repo I think?
<LnL>
things like ruby currently generate lots of duplicates, that's not good in repo
<qyliss>
We've been working on Ruby!
<qyliss>
Well, mostly manveru has been
<infinisil>
qyliss: Why wouldn't it work?
<ekleog>
infinisil: AFAIK carnix doesn't currently support that, does it?
<infinisil>
ekleog: I don't know, but it probably doesn't indeed
<clever>
ideally, 2 different projects, both generating nix exprs for dep X, would compute to the same $out for X, and share
<clever>
so the only cost is the size of the exprs and eval time
<clever>
but the instant you stick in lock files, and the deps of X differ, the whole idea breaks down
<clever>
even if X is the same ver on both
<clever>
haskellPackages mostly solves that, by just ignoring the lock files
<clever>
stack2nix/haskell.nix have that issue though
<ekleog>
clever: right, the question is not that much about build time than it is about nixpkgs size -- as build time is shouldered by hydra, while every single user has to download a copy of nixpkgs
<ekleog>
(also, in rust if the deps of X do differ, then X's API could differ too, if it exposes elements from the API of its deps, so I don't think we could share stuff anyway)
<MichaelRaskin>
And also build time is paid once, but binary cache is stored for a long time
orivej has quit [Ping timeout: 245 seconds]
drakonis has quit [Ping timeout: 252 seconds]
<infinisil>
An alternative would be to do something like all-cabal-hashes
<infinisil>
This is a repository that has all hashes for all versions of all haskell packages on hackage
<infinisil>
And in nixpkgs it's included with a simple package definition
<infinisil>
And then you can use the hashes from it for building derivations
<infinisil>
Oh wait, that uses IFD doesn't it
<infinisil>
Never mind
drakonis has joined #nixos-dev
Guanin has quit [Remote host closed the connection]
drakonis1 has joined #nixos-dev
alp has joined #nixos-dev
drakonis_ has joined #nixos-dev
johnny101 has quit [Ping timeout: 248 seconds]
johnny101 has joined #nixos-dev
drakonis2 has joined #nixos-dev
drakonis has quit [Ping timeout: 252 seconds]
drakonis_ has quit [Ping timeout: 257 seconds]
orivej has joined #nixos-dev
ryantm has joined #nixos-dev
<ekleog>
Perchance, does anyone understand why `/nix/var/nix/profiles/per-user/root/channels` is in the default nix.nixPath? It sounds to me like the `nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos` should already take care of things
<qyliss>
That wouldn't let nixos.whatever work, would it?
<ekleog>
not sure what “nixos.whatever” represent here (aside: I've looked, and it appears to be here since basically forever, though I don't know why)
alp has quit [Ping timeout: 258 seconds]
<qyliss>
nix-shell -p nixos.hello
<qyliss>
I think that works?
<qyliss>
It's been forever since I used channels
<ekleog>
oh right I think I got it: it's for eg. `nix-build '<nixos>' -A firefox` that's not explicitly in the NIX_PATH but transitively is via this item
<qyliss>
Yeah
<ekleog>
(your example returns “error: value is a function while a set was expected, at (string):1:94”, but I guess that's just a distraction)
<qyliss>
If you add other channels they'll be usable like that too
<ekleog>
welp. so there's no just removing it without some hard backwards-compatibility breakage :/
orivej has quit [Ping timeout: 258 seconds]
<qyliss>
I'm not sure you'd want to remove it anyway
<qyliss>
It makes things usable when you add them with `sudo nix-channel --add`, I think
<ekleog>
I was thinking it takes part in the nixpkgs vs. nixos channel name mess
<ekleog>
though maybe that'd be better solved by just naming the channels correctly