<ekleog>
is it only me or do installer tests always blink for some reason? just checked the 18.09 dashboard, and…
* ekleog
wonders whether there's an actual reason for all these failures or whether it's just transient failures all the time
<ekleog>
(reason: matrix people don't stop yelling that the latest upgrade is critical and to upgrade now, so I'm hoping the bump can make it through asap)
<ekleog>
if it's just transient failures all the time, then maybe something is wrong with the installer tests in particular
<ekleog>
(and given the few I've looked at are all “Waiting for the VM to connect”, I'm assuming it's transient failures? but given it's installer tests I'm not sure at all)
<samueldr>
ekleog: there *seems* to be a transient failure going on in the tests
<samueldr>
it's (probably) not caused by the increase in time the tests had between 17.09 and 18.03
<samueldr>
(that increase was caused by a new test to verify the store contents in some tests)
<samueldr>
other than that, not much is known :/
<samueldr>
the tests *could* be hanging at a spot, but there's a lack of data to know whether it's because the limit is too conservative or not
* samueldr
must open that PR
<ekleog>
the installer tests appear to be particularly heavily struck by this issue, though they are not the only ones
<ekleog>
(I'm assuming it's not just that they are rebuilt more often, without particular data to back this assumption other than gut feeling)
* ekleog
wonders what's particular in them
<samueldr>
AFAIUI all tests involving in some way the installer is rebuilt for any git revision change
<samueldr>
since the nixpkgs revision changes (I might be wrong)
<samueldr>
I know any change to the repo while testing my "add more data to tests" changes caused the installer media to be rebuilt, every time
<gchristensen>
gross. :|
<samueldr>
the image rebuild every time? annoying, not gross, the repo *does* change every time
<samueldr>
there's probably some way to pass *another* nixpkgs than the one being built though
<samueldr>
than the one being used to build the test*
<gchristensen>
right, fair
<ekleog>
hmmm that kind of makes sense, we do need to pass the whole nixpkgs to the installer tests… ideally it'd be possible to extract only the portion of the whole nixpkgs that will actually be used by the installed system, but that'd be *hard*… I guess this issue is intractable then
<ekleog>
*but* somehow nixos.tests.installer.simpleProvided.x86_64-linux looks too green for an installer test… or maybe this one doesn't include the whole nixpkgs image?
* samueldr
needs to check those assumptions
<samueldr>
possiblty the "really booting" images are having more issues?
<samueldr>
I'm finishing something (eating + some show) then doing the PR and taking a look
<samueldr>
the thing is those tests failures *could* be caused by a host being overloaded, if so, it's annoying to test any fixes
<ekleog>
yeah, another thing I noticed is that all the installer test failures appear to be on packet-epyc-1
<samueldr>
maybe we should scrape X evals and see where it fails and where it passes?
<samueldr>
#53827 might be cool to have, to better understand the "macro" profile of our tests, though I'd like it if someone with perl experience could confirm everything is fine :)
MichaelRaskin has quit [Quit: gchristensen last call: if nobody is in the process of writing a good post about medium-term RFC SC selection rules, I will write a quick-hack «please state your opinions» one]