drakonis_ has quit [Read error: Connection reset by peer]
drakonis_ has joined #nixos-dev
drakonis has quit [Read error: Connection reset by peer]
Drakonis__ has joined #nixos-dev
drakonis_ has quit [Ping timeout: 245 seconds]
drakonis has joined #nixos-dev
Drakonis__ has quit [Ping timeout: 258 seconds]
drakonis1 has quit [Quit: WeeChat 2.3]
callahad has quit [Ping timeout: 245 seconds]
callahad has joined #nixos-dev
FRidh has joined #nixos-dev
init_6 has quit []
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 252 seconds]
orivej has quit [Ping timeout: 272 seconds]
orivej has joined #nixos-dev
<LnL>
what build-max-silent-time do the hydra builders use?
init_6 has joined #nixos-dev
phreedom has quit [Remote host closed the connection]
phreedom has joined #nixos-dev
<infinisil>
Was there ever a case where some tests of a package actually failed?
<gchristensen>
like the checkPhase tests?
<infinisil>
I'd think, as long as the build phase of a package succeeds and upstream didn't publish some release that failed tests, this can't really occur
<infinisil>
Yeah
<gchristensen>
oh my yes
<gchristensen>
go and rust, for example, have tests patched out
<infinisil>
Oh, we need to also ignor test failures due to the sandbox
<gchristensen>
many tests talk to the internet
<infinisil>
I mean actual failures that indicate an error in the package itself
<gchristensen>
probably
<infinisil>
I haven't seen any
<infinisil>
And those are the ones the tests are intended for
<simpson>
infinisil: Come explore the Python subsystem sometime. It does happen that +
<simpson>
*that a package's tests cover real functionality.
<gchristensen>
I think usually a package's tests failures indicate the packaging isn't yet correct
<gchristensen>
infinisil: what're you thinking about?
<MichaelRaskin>
I am almost sure some of LibreOffice tests that we disable indicate some kind of mismatch in what we pass the package, but I have no idea how to debug this
<infinisil>
gchristensen: Disabling tests completely. Because if failures due to the package actually having an error are so so rare (or even existent), then it doesn't make sense to waste so much CPU on them
<gchristensen>
that seems pretty unwise
<MichaelRaskin>
Some of the Julia tests were indeed used to catch missing tricks around dependencies
<gchristensen>
knowing their own tests pass gives us much better confidence in their functionality, even if they usually do pass
<infinisil>
Okay so let's assume we have a package where the tests fail because of an error in the package. What do we do?
<infinisil>
Disable tests? Fix the error with an upstream patch? Complain to upstream that they released a version where the tests fail?
<infinisil>
Don't take that version at all?
janneke is now known as janneke[m]
<gchristensen>
yes
<MichaelRaskin>
… actually evaluate the specific situation?
<simpson>
Usually yes, some combination of that. In the Python tree, we usually fix or disable selected tests, and I presume I'm not the only one working with upstreams.
<gchristensen>
all of the above
<infinisil>
MichaelRaskin: Yeah :P
janneke[m] is now known as janneke_
janneke_ is now known as janneke
<infinisil>
Hmm yeah
<MichaelRaskin>
The real goal is to have the situation where tests _not_ failing means autobump
<MichaelRaskin>
And tests failing means having to look at the problem
<infinisil>
Does anybody by chance have an example perhaps?
<simpson>
I am *extremely* interested in alternative ways to have found this problem.
<infinisil>
MichaelRaskin: Oh, you mean like, run the build with tests, if they succeed, bump the package, but don't run the tests in nixpkgs itself (because we already tested it once)
<infinisil>
Because that sounds very neat and solves the problem
<simpson>
infinisil: Tests may have changed with the bump, so we gotta run the tests on the new package. I think that they mean more like what ryantm does.
<MichaelRaskin>
Well, I think the tests did fail because of dependencies upgrade before
<MichaelRaskin>
That usually leads to pinning
<infinisil>
simpson: Hmm thanks
<infinisil>
Yeah I guess tests can also start failing with upgrades of dependencies, for interpreted things especially
<infinisil>
But for compiled languages it might work (that is: only running the tests for the package bump PR, but turning off the tests in nixpkgs)
<MichaelRaskin>
Tell me more about it.
<MichaelRaskin>
SBCL bump broke some code
<gchristensen>
sounds like it could end in unhappy situations of tests being used to verify a program's exec()'d tools behave as expected, but it turns out they don't, and then the program doing something destructive because of it (and not in the nix build sandbox) -- thinking especially around things parsing the glorious text-only output of well defined unix commands.
<MichaelRaskin>
Runtime-only: standard library fixed a rare problem and got a more frequent race condition. Compilation of the code broken by the change didn't suffer.
<gchristensen>
IMO, running package tests on the Hydra infrastructure is not a waste of time. I think it is a good use of time.
<gchristensen>
humans wrote the tests to be automatic verification, automatic builds are doing verification -> humans save time
jtojnar has quit [Quit: jtojnar]
<infinisil>
I just feel like it's almost not worth it, because the tests in 99.9% fail because of the sandbox or other Nix reasons, which is not what the tests are for
<infinisil>
If we were to do some cost-benefit analysis, it costs us a whole lot to run these tests for very little benefit
<simpson>
How'd you measure that? That's quite a stark number.
<gchristensen>
99.9% of packages with tests disabled?
<infinisil>
Just a guesstimate, because I think all of the tests I've seen failing have been due to network/sandbox/test infra broken or something like that
<infinisil>
Probably only compiled languages though
<gchristensen>
the thing to do in those cases is to disable that one specific case, not all the tests
<infinisil>
Eh whatever, it's just a thought, maybe it would be worth investigating this a bit further and get some actual measurements
<infinisil>
I'm bringing this up because I'm bisecting something right now, and I had to do a couple stdenv builds, and the tests take so much build time, they get run even when just curl gets an update, which seemed pointless
<gchristensen>
yeah, stdenvs are slow to build for sure
<gchristensen>
you could try skipping stdenv rebuilds?
<simpson>
Yikes. That's unfortunate.
<infinisil>
gchristensen: I think the error is in some staging updates
<MichaelRaskin>
infinisil: what exactly you are bisecting?
<gchristensen>
if ! nix-build . -A stdenv -j0 ; then git bisect skip; fi might get you closer
<infinisil>
teamspeak_client, it's failing on master with some odd QT message
<MichaelRaskin>
(asking because I was bisecting last weekend and ended up in staging)
<MichaelRaskin>
Hm. I wonder if _that_ is related to X.org bump, too…
<infinisil>
This is the error: "/run/current-system/sw/bin/ts3client: relocation error: /run/current-system/sw/bin/ts3client: symbol calloc version Qt_5 not defined in file libQt5WebEngineCore.so.5 with link time reference"
<infinisil>
(That's the only line of output)
<MichaelRaskin>
Ouch. Ouch. calloc having version Qt_5 does sound scary
<Profpatsch>
infinisil: Why not mumble?
<MichaelRaskin>
I think separating tests, and giving options about running/ignoring them — that might be a feasible sell
<infinisil>
Oh, looking at the possible commits that broke this (down to ~100), it's probably 8e811ec295424649bd7fca81ec136e2d28bf8230 (qt5: 5.11 -> 5.12)
<gchristensen>
MichaelRaskin: impure env var for running testst?
<MichaelRaskin>
Nope
<infinisil>
Profpatsch: I might try that instead. All my friends are currently using teamspeak on a friends server
<Profpatsch>
MichaelRaskin: If you have some upstream use for them, I’m not against putting it into nixpkgs
<MichaelRaskin>
I don't really need a lot of old build data, is scraping build information a heavy load for Hydra's web UI or would it be OK
<Profpatsch>
I wanted to use it to run the bazel tests, so I might upstream it myself.
<gchristensen>
MichaelRaskin: go slow and stop if you get 500's? :)
<MichaelRaskin>
Profpatsch: I am not going to lead any test work in Nixpkgs until I see a way to run GUI integration tests without booting a NixOS Qemu for a minute
<infinisil>
MichaelRaskin: How would that work without a VM?
<MichaelRaskin>
gchristensen: I have a suspicion that if I scrape with an optimal rate it might degrade everything for everyone.
<gchristensen>
(uhoh)
<gchristensen>
depends on your reference frame for the definition of optimal :P
<Profpatsch>
infinisil: Maybe over wayland somehow?