<energizer>
if you want to do it yourself, clone nixpkgs, change the version, change the hash to lib.fakeSha256, and from repo root run nix-build -A wolfssl, it will fail and tell you the new hash. replace lib.fakeSha256 with the new hash. submit a pr.
<energizer>
otherwise file an issue on nixpkgs
<lechner>
that's a giant repo!
<infinisil>
lechner: git clone with --depth=1 will do too :)
<infinisil>
But yeah, it do be pretty big
<lechner>
energizer: the issue tracker reminded me: The CVE is a 9.8 on the NVD scale but is already public. Should I still just file un update request?
<lechner>
sorry poor speller
<infinisil>
lechner: If you can do a PR directly that would be best, but if not, an update request is probably the next best thing unless somebody else makes a PR for you
<lechner>
infinisil: do i need a virtual system to run 'nix-build' ?
Jackneill has quit [Read error: Connection reset by peer]
__monty__ has quit [Ping timeout: 256 seconds]
Jackneill has joined #nixos-dev
AlwaysLivid has joined #nixos-dev
<Mic92>
gchristensen: I appretiate your work on hydra.
<niksnut>
+1
stolyaroleh_ has quit [Ping timeout: 272 seconds]
<gchristensen>
thanks, Mic92 :) it, uh, isn't the most fun of work to be doing, but hopefully it pays off :D
<Mic92>
gchristensen: for sure. Longterm I want to have hydra status checks in nixpkgs-review
stolyaroleh_ has joined #nixos-dev
jonringer has joined #nixos-dev
mkaito has joined #nixos-dev
<gchristensen>
I'd love to get a review on this PR which fixes the bandwidth amplification deployed last week: https://github.com/NixOS/hydra/pull/860 I'm planning on deploying it in ...15 minutes ... unless I hear back
<{^_^}>
hydra#860 (by grahamc, 13 hours ago, open): Move evaluation errors from evaluations to EvaluationErrors, a new table
Jackneill has quit [Ping timeout: 240 seconds]
<abathur>
oh man, tock's tickin'
<gchristensen>
it is sort of a "how much worse could it get" kind of situation
<abathur>
ye
<abathur>
I looked at it last night after you posted it but have no clue :]
<gchristensen>
:D
<gchristensen>
thanks
copumpkin has joined #nixos-dev
copumpkin has quit [Client Quit]
copumpkin has joined #nixos-dev
Jackneill has joined #nixos-dev
<gchristensen>
what if Hydra had a graph view, which showed all the derivations in the queue and their dependencies and how they related to each other
<gchristensen>
with live-updates... for example: hydra is working on this job, which means it has to work on these dependencies first
<infinisil>
That would be nice!
<Taneb>
I think that would be very useful
<infinisil>
That feels like it should be a Nix feature though, with hydra just exposing it
<gchristensen>
it sort of is a nix feature already, isn't it?
<infinisil>
gchristensen: Like the Nix output tree mockup you did
<infinisil>
It is?
<gchristensen>
I dunno :x
<gchristensen>
and this is unique to hydra, really, since nix doesn't have so much of a job scheduler
<infinisil>
Hmm, maybe Nix should have a job scheduler? Or does it not need one?
<gchristensen>
nix the client has a job scheduler, nix the daemon does not, really
rajivr has quit [Quit: Connection closed for inactivity]
<gchristensen>
can I specify a list of substituters for hydra-queue-runner, different from the system's list of substituters?
<infinisil>
bwrap it with a different /etc/nix/machines :P
<infinisil>
gchristensen: I recently did such a thing for another program, works surprisingly well
<tilpner>
gchristensen: Perhaps with NIX_CONFIG="substituters = ...", on nixUnstable
<gchristensen>
hmm
<tilpner>
I don't know if it reads other files, may need an include
<infinisil>
Ah NIX_CONF_DIR might work
<infinisil>
Ah NIX_CONFIG too
<tilpner>
Or do the wrapper dance, and do PATH=${customNixWithOptionFlags}/bin:$PATH :P
<gchristensen>
lo0l
<gchristensen>
hydra has its own c++ integration with Nix, so not sure that'd work :P
<tilpner>
Ahh, not an option if hydra-queue-runner links libnix*
<ris>
has anyone (perhaps I've even mentioned it before) considered adding an e.g. `fromSource = false` flag to meta that would allow people to filter binary packages in the same way they can filter nonfree packages?
<ris>
might address some of the concerns that some guix people have
<infinisil>
ris: The concern being?
<ris>
you don't really know what you're getting when you request a nix package for installation
<ris>
supply chain attack concerns etc
<supersandro2000>
you don't know that on any OS
<supersandro2000>
binary packages are currently not treated any different meta wise
<ris>
rrrrrrrright reflections on trusting trust yes yes, but that's a false dichotomy
<ris>
no, i know they aren't, but perhaps flagging them would give people the option
<supersandro2000>
I think that sounds like a RFC
<ris>
phaps
<supersandro2000>
we only download binaries if upstream has no source (eg often ufnree) or it is very tedious to build
<ris>
weeeeeeell, there are quite a few binaries that aren't _that_ hard to build. in my experience, a lot of our binaries are "nobody's got around to doing a source package for that yet, binary is better than nothing"
<infinisil>
lechner: Ahh yeah, use a `0` instead of a `2`
<V>
supersandro2000: signal desktop
<infinisil>
The hash format requires a 0 or 1 at the start
<V>
For one
stolyaroleh_ has quit [Quit: Konversation terminated!]
<ris>
V: this was the example given by a guixy colleague
<V>
I wanted to patch it a while back, went in, and discovered that it was using the upstream binaries
<V>
While I could probably fuck around with the app.asar, it would be much better if all the source-available packages were packaged from source, and everything packaged from binary are available strictly via package names ending in -bin or such
<V>
That would lead to far less confusion, I think
<infinisil>
Oh wow, never seen that error in practice
<ris>
V: than having a meta attribute?
<lechner>
n00bs can screw up anything!
<V>
ris: no, than the current state of things
<ris>
oh i see
<V>
IIRC Gentoo has -bin packages
<ris>
i think meta has an advantage in that people so-minded can block binary dependencies too
<gchristensen>
yea
<V>
Just block everything with a name ending in -bin, easy :p
<gchristensen>
lol
<V>
Name comparisons are how people currently whitelist unfree packages, anyway
<V>
So it's not that crazy of an idea
<gchristensen>
that was sort of a last-resort option
<gchristensen>
anyone want to rename that to allowlist?
<V>
Isn't nixpkgs a large collection of those? :p
<gchristensen>
no
<lechner>
infinisil: the error persisted even after 'rm -rf /nix'
<supersandro2000>
ris: java eco is not great currently
<infinisil>
lechner: Oh, that *never* fixes build issues with Nix
<infinisil>
lechner: Try running the `nix-build` with -K
<infinisil>
This should output something like "keeping directory bla bla bla"
<infinisil>
Somewhere in /tmp probably
<infinisil>
Hmm actually never mind
<infinisil>
This doesn't let you debug the cyclic dep
<gchristensen>
--keep-failed keeps the paths in /nix
<lechner>
is this a link cycle?
<infinisil>
Oh it does? (how is that safe?)
<gchristensen>
it isn't a registered path and is deleted at the next GC
<infinisil>
AH gotcha
<gchristensen>
the issue is that, like, the .dev output revers to .out and .out refers to .dev
<gchristensen>
or something
<V>
lechner: there's stuff in the dev output pointing into the lib output and vice versa
<infinisil>
lechner: Could be, but could also be just a string reference (e.g. if one /nix/store path has a script that calls the other /nix/store path)
<gchristensen>
and the easy fix is to just not split outputs
<V>
IIRC
<V>
I closed the paste a while ago
AlwaysLivid has quit [Remote host closed the connection]
<infinisil>
gchristensen: Hm yeah maybe that's best for now
<infinisil>
lechner: So what you can do to fix it, is remove the `"lib"` and `"dev"` strings in the `outputs = ` assignment in the default.nix
AlwaysLivid has joined #nixos-dev
<lechner>
infinisil: then i get: mv: cannot move '/nix/store/7zl5bw3m3fa0xw7xv7nxy1jj6srrl04x-wolfssl-4.6.0/bin/wolfssl-config' to '/bin/wolfssl-config': Permission denied
<infinisil>
lechner: Ah, remove the `moveToOutput` command below
v0|d has quit [Ping timeout: 258 seconds]
AlwaysLivid has quit [Remote host closed the connection]
AlwaysLivid has joined #nixos-dev
saschagrunert has quit [Ping timeout: 264 seconds]
<infinisil>
lechner: Btw, you can build a curl (the only package that has some builtin support for wolfssl) with `nix-build -E 'with import ./. {}; curl.override { sslSupport = false; wolfsslSupport = true; }'`
<infinisil>
Although actually when I tried that out, I got a curl that can't even do https curls
<lechner>
please tell me more: cURL is a wolfSSL product!
<gchristensen>
I didn't know wolf could own cURL :)
<infinisil>
lechner: Oh I'm just noticing the comment "fix recursive cycle" hehe
<lechner>
infinisil: yeah, i was trying to leave the thinp in an itelligible state, but i am too far behind you
<infinisil>
lechner: I think if you use -K as suggested above, you should be able to inspect the cyclic paths
<infinisil>
And figure out which file causes that to happen, and then move it manually (like the command you uncommented)
<infinisil>
If you have a cyclic dependency between /nix/store/...-lib and /nix/store/...-dev, you can grep for `/nix/store/...-dev` in `/nix/store/...-lib` to figure out where there's a link, and same for the other way around
<supersandro2000>
Can we invalidate all ofborg results?
<supersandro2000>
I just broke master by mergeing stdenv.lib, again
<gchristensen>
supersandro2000: no there isn't such a way
<gchristensen>
just have to go comment `@ofborg eval` on them
<supersandro2000>
mmmmm
<supersandro2000>
I am not going to comment that on all
AlwaysLivid has quit [Remote host closed the connection]
drakonis has joined #nixos-dev
<lechner>
infinisil: how can i invalidate my build product? on rerunning, I just get a laconic /nix/store/fc6zbif820c528rbd87nyshvaa8qpv7d-wolfssl-4.6.0
<cole-h>
--check
<lechner>
cole-h: thanks!
red[evilred] has joined #nixos-dev
<red[evilred]>
How does one go about mapping dependancies in nixos?
<red[evilred]>
I have a fresh install with nothing in systemPackages yet it's still pulling in half the universe and I'm trying to work out why
<infinisil>
In Nix you never need to invalidate any caches
<infinisil>
Because if anything changes about the build, it will cache it in a different place
<infinisil>
So if you get the same result, it's because nothing that would influence the build changed
<infinisil>
--check is mostly only useful if the build is non-reproducible (which is the exception)
<red[evilred]>
lassulus (IRC): interesting - that shows very few things.. so why do I have all these other things
<red[evilred]>
I'll kick this around - thanks for the starting point!
<lassulus>
hmm, maybe /run/current-system doesn't show everything
<lassulus>
but not sure
<lassulus>
ah yeah, try /run/current-system/sw
<tilpner>
red[evilred]: environment.noXlibs might produce a smaller closure at the cost of lower cache hit rate
<lechner>
infinisil: what if i added -K
<infinisil>
lechner: Still the same, -K just keeps the build artifacts after it's done
<lechner>
i don't think it restarted the build
<tilpner>
Although... I just checked the module, and I apparently misremembered how much it changed. Perhaps dbus, but otherwise none of these should be on your system
<infinisil>
lechner: Then the build succeeded already
<red[evilred]>
so interesting
<infinisil>
lechner: What's your current state of nixpkgs?
<lechner>
cloned earlier today
<lechner>
i found it btw
<red[evilred]>
if I give nix-du -r /run/current-system/sw instead of just /run/current-system it detects all the other packages
<red[evilred]>
but doesn't have any dependency information for them al all
<infinisil>
lechner: Found what?
<lassulus>
it just looks at all the symlinks in /run/current-system/sw
<red[evilred]>
ah
<lassulus>
so stuff would be in environment.systemPackages to end up in there, maybe nixos-option environment.systemPackages helps you?
<lechner>
infinisil: the circular ref is in /nix/store/xagmdww3f1fpq8fmbv064aw71f12169s-wolfssl-4.6.0-dev/bin/wolfssl-config https://dpaste.com/47VMJKEBW
<infinisil>
lechner: Hmm, so that points to -dev right?
<infinisil>
Oh *and* -lib
<lechner>
i am looking the other way right now
<lechner>
the script is like a homebaked pkgconfig
<infinisil>
So we have wolfssl-config which the `moveToOutput` moves to -dev
<infinisil>
And has references to both -dev and -lib
<lechner>
i think we cn delete it
<infinisil>
So then the -lib needs to not have any references to -dev, otherwise we have a cycle
<cole-h>
If it's part of the package, it has to be there for a reason, right? Even if we don't use it, packages that depend on wolfssl might.
<infinisil>
Yeah I think so too ^
<ris>
reality check - can patches supplied in `patches = ` include (git) binary sections?
<red[evilred]>
oh - I found an issue in nixpkgs tracking it
<lassulus>
red[evilred]: ah yes, all this bloat was tried to be fixed a couple of times by different people. But somehow it never went far
<cole-h>
From what I've seen, the whole point of wolfSSL is embedded, no?
<lechner>
cole-h: i meant in a reduced development environment without pkgconfig for experimental boards, for paying customers
<lechner>
infinisil: the reference tho otheway comes from libtool's .la file, which i think is useful https://dpaste.com/DHFWB6YW8
<lechner>
although i don't currently ship that in Debian either
<infinisil>
Okay so I think just getting rid of the -dev and -lib outputs is good then
<cole-h>
ris: You could try to convert it into base64 and then decode the base64 in postPatch or whatever
<infinisil>
lechner: In Nix there's the convention to fall back to the default output if -dev or -lib don't exist, so that shouldn't be a problem
<ris>
cole-h: well, it's from github so i think the simplest thing is just going to be to fetchurl the file and dump it where it needs to go
<lechner>
okay, it's like shipping the headers with the ELF library, right?
<cole-h>
ris: That works too :P
<ris>
don't think there are any pretty solutions here
<ris>
in an ideal world there would be a `patchCommand = ` override where packages that need it could just inject git apply blah blah
<ris>
(i don't really want to write my own whole patchPhase)
<lechner>
hey, i have a couple of general questions, please: why do you prepend the hash, please. wouldn't it be easier to find stuff the other way around?
<gchristensen>
the /nix/store is not generally looked in, generally, software should act as though /nix/store cannot be enumerated
<samueldr>
I seem to recall there was a reason from the last few times it came around in discussions
<gchristensen>
maybe b/c hashes distribute more evenly
<samueldr>
but already there's the issue where the nix store on developer machines will generally not be in a situation where you can just ls them :)
<samueldr>
too many cooks^W derivations
<samueldr>
I believe it's half-assumed to be opaque, where you will access what you know about, but shouldn't try to reason the whole store
<gchristensen>
+1
<lechner>
okay, and then why are circuler references disallowed for outputs even from the same build?
<gchristensen>
the store paths are registered sequentially, and every store path must be valid once it is registered. you can't register 2 things at once
<lechner>
is that your way or normalizing path names?
<samueldr>
if it's $out -> $lib -> $out, it's still two distinct outputs even though they're from the same build
<lechner>
of
<gchristensen>
a path is valid only if all of its dependencies are present
<lechner>
i see
<gchristensen>
finally, there is no reason to support circular references like between lib and out: they cannot exist independently, so don't split them
<lechner>
just so you know, i have been super enthusiastic about nix since hearing about it, but most people i have met are skeptical
<gchristensen>
yeah :)
<gchristensen>
sometimes people have a bit of a hard time stomaching the rules
<samueldr>
I often have seen people who "can't believe it won't be butter", who can't believe it works
<lechner>
actually, they told me it doesn't
<gchristensen>
jdjde
<gchristensen>
hehe.
<lechner>
does nix have speed issues due to finding or stat'ing files in all those folders?
<samueldr>
since a derivation knows about store paths, and doesn't resolve them at runtime, not AFAIK
<samueldr>
maybe garbage collection and store repair operations type of operations?
<gchristensen>
there is some small penalty in the rpath search, but usually the filesystem cache covers it and the impact is not significant
<gchristensen>
there is some searching, but not very much
<lechner>
but there is a lot of rebuilding?
<samueldr>
exactly as much as needed
<gchristensen>
:D
<lechner>
:)
<lechner>
has gentoo looked at mix?
<lechner>
nix
<gchristensen>
I doubt it, it is fairly different
<lechner>
they have the local infrastructure to rbuild via portage
<lechner>
do you impose tight controls on your tool chains? (or do builds simply fail and don't get installed)
<gchristensen>
I think the answer is yes, but I'm not sure, and I'm thinking it is because our control is so tight that I'm imagining you're asking a different question
<samueldr>
if a build fails, the whole dependent chain will fail, there is no automation about changing configuration if things fail
<samueldr>
a precise question is asked (a derivation, with precise inputs) and you will either get the answer, or a failure :)
<samueldr>
there should be no partial state
<lechner>
i am just trying to learn
<samueldr>
there _is_ no partial state
<gchristensen>
lechner: sure, of course :) no worries, you're asking good questions
<gchristensen>
people often say tight control over tool chains means saying gcc x, or gcc x.y, or gcc x.y.z
<gchristensen>
our tight control of tool chains is gcc x.y.z with hash abc123 built with this version of bash and everything else
<lechner>
but the number of possible permutations will forever prohibit binary distribrustion, right?
<gchristensen>
we are a binary/source hybrid distribution. almost all users download precompiled binaries
<gchristensen>
if you want to customise something, nix will try to fetch a precompiled version, fail, and then build it locally it
<lechner>
is nixos here to stay, or is it mainly a proof of concept on the road to getting the package manager accepted in a major distro?
<gchristensen>
nixos has been a thing for about 18 years now, and is used by a number of very large organizations
<gchristensen>
I don't think it is going anywhere
<samueldr>
if something, a debian/ubuntu kind of situation is more likely to happen with NixOS/?????? than trying to get Nix into something existing
<samueldr>
it's definitely not an academic project done only for advancing research :)
<immae>
It is quite frustrating to be asked to write a fix, write the fix as asked, and then getting ignored when the fix is written... :(
<gchristensen>
ack, I'm sorry immae, I missed it in a pretty hectic day. I'll test it and merge it tomorrow
<immae>
ok thanks :)
<gchristensen>
immae: do you know how to drop a constraint only if it already exists?
<immae>
hmm let me see
<immae>
do you think it can happen that a constraint would not already exist?
<gchristensen>
wait ...
<gchristensen>
one sec, immae ...
<gchristensen>
ahh... hmm..., the thing I was thinking about is `check (schedulingShares > 0),` has no specific name in hydra.sql, but is named in the migration adding it: "alter table Jobsets add constraint jobsets_check check (schedulingShares > 0);"
<gchristensen>
this seems tricky.
<immae>
Oh I see
<immae>
> ALTER TABLE jobsets DROP CONSTRAINT if exists jobsets_foo
<{^_^}>
error: syntax error, unexpected IF, expecting ')', at (string):471:37
<immae>
This kind of syntax seems to work (no matter what {^_^} says)
<gchristensen>
:D
Jackneill has joined #nixos-dev
<gchristensen>
I wonder if we need to drop the existing constraints and re-add them with unique, specific names
<immae>
Normally they are predictible though (if you add them in the same order, then you should have the same names)
<immae>
but yes giving a unique name is usually a good idea :)
<immae>
Do you want me to add that to the PR, or should it be in a separate one maybe?
<immae>
(there are several checks, not limited to jobsets table)
<immae>
Oh not so much actually, only Builds table and Jobsets table
<gchristensen>
hmm... probably should be a different PR. I'm a bit concerned about how the names will have played out for different deployments of hydra over time... well, we can find out. I'll add a comment to the PR
<immae>
Well, if we don’t add the "if exists" then we are sure that at the end of the migration script it either worked or failed. And it "should" fail only for people who develop on hydra (and thus have occasions to play with the database schema)
<gchristensen>
right
<immae>
On the other hand in the case of my PR the new checks are strictly more constrainted than the existing ones, so having two more would be harmless
<gchristensen>
true
<immae>
Oh well your comment is worrying because the production hydra doesn’t hav the same checks as a brand new one :D
<immae>
(checks names)
<gchristensen>
:)
<immae>
So a way to do it in the most generic way is to delete all the constraints of table jobsets in the migration, and then re-adding them in that same migration
<immae>
(I don’t know if it is feasible)
<gchristensen>
I commented an idea of how I think we could do it
<immae>
Yes it seems reasonnable
<gchristensen>
if you decide you've bitten off more than you care to chew, it is okay
<immae>
Maybe a bit over-causious (I’m not sure if it is really a bad thing to have an unconstrained table for a few milliseconds)
<gchristensen>
I have no evidence this is true, but I sort of hope postgres will notice it already has an identical constraint and skips checking :P
<immae>
It might refuse to create an identical check though :p
<immae>
(I’m trying that)
<gchristensen>
I have a copy of prod that we can do cheap experiments on
<immae>
So the good news is, we can have several checks that are exactly identical (psql doesn’t care as long as it doesn’t have the same name)
<gchristensen>
nice
<immae>
My "over-cautious" question was rather about your "Then we don't at any point lack the constraint. Does this seem reasonable?" sentence: is it a problem if a table lacks the constraint for a few milliseconds during the migration?
<immae>
(i.e. do we expect hydra to regularly try to enter offending entries, or is it just a "safety check"?)
<gchristensen>
no I don't
<gchristensen>
but it would suck for the migration to fail :P
<gchristensen>
and avoiding it in this case seems fairly cheap and easy
<immae>
your plan of action is fine indeed, but it assumes that we can predict the name of the check in all situations (it turns out I was wrong about that since prod-hydra is diffrent from immae-local-hydra). There seem to exist a query that drops all constraints on a table (so without knowing their name), which garantees to end up with an homogeneous list of check names at the end of the migration
<gchristensen>
back in a bit ... time for supper, thank you a lot for thinking through this with me
<immae>
sure :)
<immae>
(I might be asleep by then but we can resume tomorrow)