<gchristensen>
IMO a secondary repo is dead before it starts
<gchristensen>
what I'd suggest is start with something less "grand-plan" and more "just a little thing" and see how it goes
<ekleog>
I'm not sure something can be less “grand-plan”, in order to get lightweight tests :/
<ekleog>
that said a secondary repo can be useful, so long as enough committers politely ask people to add tests there before merging the PR, and could constitute an argument in favor of including it in nixpkgs
<ekleog>
(I know I'd try to remember adding a test there before adding in PRs that add/update packages, were one to constitute an argument in favor of these)
<gchristensen>
a second repo won't work because that isn't a reasonable thing to ask
<gchristensen>
there is a lot of room to work in smaller scale which doesn't require grand buy-in from the greater community to prove out tech
<gchristensen>
I happen to agree with some of the feedback Eelco left w.r.t. waiting for X and what-not, but I also think the lighter tests are useful for cases that don't require X
<ekleog>
the issue is, how would you do a lighter test case?
<ekleog>
there's currently no way to do test cases across packages
<gchristensen>
sure
<ekleog>
or by “just a little thing” do you mean “just the same PR but forgetting the fact it's possible to do complex things with it”?
<ekleog>
(that's the only solution I'm thinking of right now)
<ekleog>
(and thus removing tests that require X, and add them later on when some tests for CLI applications interworking have already been added)
<gchristensen>
Eelco had two primary objectionss to the code as was written: (1) it does service management, many useful tests can be written without service management
<gchristensen>
(2) the tests were too far from the package expressions
<gchristensen>
neither of these are "I don't want tests"
<ekleog>
I can get 1 (which would mean a stripped-down version of the PR, as I understand it, what I meant by “complex things”), but I don't get how it's possible to fix 2 while having inter-package tests ./
<ekleog>
like, if a test tests two packages, necessarily it can't be neither in one package nor in the other
<ekleog>
s/can't/can/
<gchristensen>
well don't fix that part then
<gchristensen>
or, pick your favorite package to put it under
<gchristensen>
is the taxonomy of the testing infrastructure the burning need, or is it having tests for packages?
<ekleog>
(just posting a message I was preparing, that could still be useful:) so to sum it up, I'd say if we want to rework the PR to give it a chance under these conditions, it'd be needed to : 1/ remove the `coreutils` tests (it tests only `coreutils`, thus would be subject to debate), 2/ remove `evince`, `libreoffice`, `youtube-dl` and `zathura` tests, 3/ remove the associated helpers. This leaves `texlive`,
<ekleog>
`maxima` and `imagemagick`
<ekleog>
actually I'm starting to think of a taxonomy that would make sense: just have a meta.tests = listOf derivation; attribute set, that must all build for the package to be considered as having passed tests
<gchristensen>
sure
<ekleog>
that's enough to, later on, add tests shared between multiple packages
<gchristensen>
sounds low-barrier
<ekleog>
which would just be fetching the derivation from a global attrset
<ekleog>
and then we can think of service management and more complex stuff much later on, but likely it wouldn't require large modifications to the model
<ekleog>
this way changes'd be incremental
<gchristensen>
initially probably avoid a global attrset
<ekleog>
indeed, so just add all these tests under a single one of the tested packages
<gchristensen>
IMO I'd stick to a list of functions in meta.tests, each can be called by callPackage
<ekleog>
and refactor that later on
<gchristensen>
yeah
<ekleog>
hmm, I'm not sure to get what you mean by “a list of functions”?
<gchristensen>
like meta.tests = [(import ./test.nix)] where test.nix contains: { stdenv, mypkg, foo }: stdenv.mkDerivation { ...
<gchristensen>
or whatever
<gchristensen>
anyway
<gchristensen>
it sounds like we've come across a more minimal approach to get the test train going
<ekleog>
oh indeed, I didn't think of the fact that pkgs isn't passed to the derivations, and instead they get all the dependencies independently
<gchristensen>
yeah
<gchristensen>
that way the runner can be (map callPackage pkgs.meta.tests)
<ekleog>
hmm there is still a question of how to cope with overridden packages, then
<gchristensen>
a great question
<gchristensen>
I'm sure one of the many valid questions to be raised
<ekleog>
I guess first question is, what exactly do we want to test?
<ekleog>
I'd say it's the package as its name appears in `all-packages.nix`, so maybe it's not that big an issue, actually
<gchristensen>
yeah
<ekleog>
there's still the question of packages that should be callPackage_i686, though
<gchristensen>
my general approach is, when it doubt cut scope
<gchristensen>
get something working and Nixpkgs will be better for it
<ekleog>
indeed :)
<gchristensen>
then Sonarpulse will want it to work great for cross testing too and he and his compatriats work on it
<ekleog>
well, good night! and MichaelRaskin ^ just in case :)
<gchristensen>
good night :)
<gchristensen>
(what is your TZ?)
<ekleog>
(Europe/Paris, getting quite late here)
<gchristensen>
yikes
<gchristensen>
it is almost my bedtime here in America/New_York
<ekleog>
guess you wake up earlier than I do :p
<gchristensen>
0630
orivej has joined #nixos-borg
jtojnar_ has joined #nixos-borg
jtojnar has quit [Read error: Connection reset by peer]
<dtz>
do compatriats of Sonarpulse get fancy hats? Asking for a friend >_> <_<
jtojnar has joined #nixos-borg
jtojnar_ has quit [Ping timeout: 240 seconds]
<MichaelRaskin>
gchristensen: re: secondary-repo-dead: if I get to the point where adding the test for the current state of a graphical package and asking ofborg to run it is the same amount of hassle as checking out the PR branch, building locally and checking manually (which I find believable) and then start doing that and get a couple of active members join me (again, judging from the conversation, believable), we could have something that tests that nothin
<MichaelRaskin>
ekleog: gchristensen: re: service management: the packages that depend on nothing are the packages where there is a chance that upstream tests work and are sufficient (also the least annoying to test inside nix-shell --pure, etc.) For the tests I actually care about, there are some environment setup requirements, but running a full NixOS image is just not an efficient way to satisfy them.
jtojnar has quit [Quit: jtojnar]
jtojnar has joined #nixos-borg
jtojnar_ has joined #nixos-borg
jtojnar has quit [Read error: Connection reset by peer]
jtojnar_ is now known as jtojnar
<MichaelRaskin>
I think with our kind of PR backlog, a large fraction of people who ever see me doing «nixpkgs-test-merge-after 48h» will ask what they need to do to get the same treatment
MichaelRaskin has quit [Quit: MichaelRaskin]
IdleBot_e8d8e38f has joined #nixos-borg
<IdleBot_e8d8e38f>
Just in case there is some test discussion…
<IdleBot_e8d8e38f>
IdleBot-s are what I (MichaelRaskin) use to hang around for multiple days when I want to avoid risk of clashes with the main client instance
jtojnar_ has quit [Ping timeout: 240 seconds]
<makefu>
ha!
jtojnar has joined #nixos-borg
jtojnar has quit [Quit: jtojnar]
jtojnar has joined #nixos-borg
jtojnar has quit [Ping timeout: 264 seconds]
<ekleog>
MichaelRaskin: I was thinking that service management for tests could be added later on, after tests not requiring service managements have already been widely accepted and it's one fewer bikeshedding topic, so that the discussion could concentrate on exactly how to properly integrate service management (eg. maybe running a dedicated runsvdir, or similar low-footprint already-made service manager, or…)
<ekleog>
and politically speaking the secondary repo has issues, as then if you want to merge it you have to get over the fact that you “went around” the previous refusal, even if you have users to prove it's useful
<IdleBot_e8d8e38f>
ekleog: given the enthusiastic reaction to the idea of a -next branch for Nixpkgs as a whole in a separate repository…
IdleBot_e8d8e38f has quit [Remote host closed the connection]
IdleBot_4e8513a6 has joined #nixos-borg
<IdleBot_4e8513a6>
I am not sure what tests we want to add that require zero setup and still test something. I mean, there are definitely Julia tests from upstream, but we have actually stopped running them in the build because they can fail for reasons we are not sure we want to fix immediately…
<IdleBot_4e8513a6>
Checking return codes for all executables to be non-special might be interesting, but probably mostly as a sweeping change
<ekleog>
I was thinking of at least the coreutils, texlive, maxima and imagemagick tests
<ekleog>
among the tests you already pushed to the previous PR
<ekleog>
and then once enough people have started adding tests like this for “simple” programs, and it's become obvious these tests are here to stay (likely in tests/ as the need to mutualize tests between programs will have arisen since then), we will be able to have a calm discussion about service managers
<ekleog>
s/since/by/
<IdleBot_4e8513a6>
Well, texlive test is meaningless on its own (without the Zathura/Evince tests providing the other dimension of the test matrix) — if NixOS manual builds as PDF, TeXLive works somehow
<ekleog>
hmm I seem to remember you used pdftotext?
<ekleog>
not enough to catch the “evince can't read something produced by pdftotext” yet, but it's already a good first progress, so that people see the point of nixpkgs-tests, don't you think? :)
<IdleBot_4e8513a6>
I am not sure what is the incentive to have such verification — empirically if TeXLive happens to build any manual, it is very unlikely to do it completely wrong
dtz has quit [Ping timeout: 248 seconds]
dtz has joined #nixos-borg
<Mic92>
ekleog: IdleBot_4e8513a6 was there not a a good talk on package-level tests on nixcon?
<Mic92>
The basic idea was not to seperate tests from the actual derivatoin
<Mic92>
*derivation
<Mic92>
so even if the test fails, the package is still usuable
<Mic92>
s/was not/was/
<gchristensen>
from profpatsch
<IdleBot_4e8513a6>
Mic92: the premise of the talk, though, is that the tests already exist
<Mic92>
IdleBot_4e8513a6: I am a friend of regression tests
<Mic92>
also moving out the tests provied upstream to a seperate test derivation would be sometimes helpful
jtojnar has joined #nixos-borg
orivej has joined #nixos-borg
MichaelRaskin has joined #nixos-borg
<MichaelRaskin>
So do we have a recent problem with a package that doesn't work after a nominally succesful build, which doesn't make reverse-dependencies in Nixpkgs fail the build and which doesn't need any environment to reproduce/demonstrate?
<makefu>
MichaelRaskin: there seemed to be some issues regarding backwards compatibility of simp_le. not sure where it came from but the script hung because the certificates the previous version created were tipping off the new version. however i am not sure how to reproduce. i am also not sure if this is a good example as the simp_le and certbot test suite is enabled AND pretty exhaustive
<MichaelRaskin>
I would expect something not caught by their test script to need a mock ACME server to reproduce…
<makefu>
they actually boot up a mock ACME server
<MichaelRaskin>
So I guess it wouldn't be a good case for a test localised with the package…
<makefu>
yes, unfortunately
jtojnar has quit [Quit: jtojnar]
jtojnar has joined #nixos-borg
jtojnar_ has joined #nixos-borg
jtojnar has quit [Ping timeout: 240 seconds]
jtojnar_ is now known as jtojnar
<MichaelRaskin>
Plan B doesn't seem to work well…
<ekleog>
n
<ekleog>
whoops, mistook weechat for vim, tried to switch to next buffer in hotlist with n…
<LnL>
I've sent more then just one vim command to irc before :p