<{^_^}>
input-output-hk/iohk-ops#409 (by cleverca22, 22 weeks ago, open): [DEVOPS-1012] adding first version of hydra-util to git
<clever>
gchristensen: this PR adds a util that will expose some timing details from the hydra db
<gchristensen>
nice
<MichaelRaskin>
gchristensen: Hydra could also provide a single-request export of public jobset jobs across time…
<clever>
gchristensen: basically, it allows you to render a graph, showing the timespan every step was building for
<clever>
and then you can see if a group of steps are waiting for one step
<MichaelRaskin>
Actually, with graph it might be better to allow dumping and processing locally
<clever>
or if there is a gap, where nothing built at all (slaves busy with other things)
<MichaelRaskin>
Although if it is JS, a webpage with scripts to process locally will arise
<clever>
it also deals with other jobsets stealing builds and caching them
<gchristensen>
huh
<clever>
if you query for every step in the current eval, you wont see cached builds done by another jobset, on the same drv
<clever>
but the above util will query the closure of everything in the eval, then find the steps that built it, no mater what eval/jobset built it
orivej has joined #nixos-dev
<gchristensen>
I'm a bit suspicious that hydra doesn't properly utilize hardware across architectures / features
delroth has quit [Quit: WeeChat 2.3]
delroth has joined #nixos-dev
<gchristensen>
to clarify, nixpkgs:nixpkgs-unstable might have a zillion aarch64 build jobs, but be deprioritized b/c it is out of shares, meanwhile no other jobset has aarch64 jobs
<MichaelRaskin>
I dunno, is hydra.nixos.org the web frontend underloaded enough for scraping it being a better solution than negotiating a data dump?
<MichaelRaskin>
(of course, that won't give an answer to questions like «does a nonpublic jobset compete»)
<gchristensen>
there are no non public jobsets
<gchristensen>
unless even admin rights can hide them ...
<MichaelRaskin>
Wait, then _what_ are the privacy implications of a Hydra data dump?
<clever>
private jobsets are a joke in hydra
<clever>
MichaelRaskin: i believe the sha1 of passwords is in there, if using hydra based login
<gchristensen>
user accounts
<MichaelRaskin>
I mean, come one, not exporting a table is not that complicated.
<gchristensen>
pass me the command :)
<gchristensen>
preferrably the dump will go to stdout, not the fs
<MichaelRaskin>
Skipping a _column_ might be harder
<MichaelRaskin>
I think user table doesn't affect the queue in any way anyway
<clever>
i believe the queue-runner will mostly be checking the closure of a given .drv (nix-daemon intensive), then checking what paths exist already (s3 intensive), then computing what steps remain to build, and adding them to an internal queue
<clever>
you can also reverse the ssh a bit, ssh hydra@hydra -t "pg_dump ... | pv | gzip | aespipe"
<gchristensen>
heh, true
<clever>
or ssh anything@hydra -t "sudo -u hydra ..."
<gchristensen>
that is very complicated
<clever>
-t forces it to allocate a tty, so sudo and friends will work as normal
orivej has quit [Ping timeout: 268 seconds]
<MichaelRaskin>
Why aespipe if it is supposed to be a dump of public data?
<simpson>
It's good to be in the habit. And that's the right order, too.
<gchristensen>
and, just in case I accidentally included non-public data
<simpson>
Well, um. `pv` should be later in the pipe. You probably don't care how fast Pg goes, but how fast Pg and gzip together go. Unless this is one of your forty-something-core boxen, I guess?
<clever>
gchristensen: oh, you can make the gzip faster...
<dtz>
maybe let ssh handle compression/encryption (setting those options explicitly if needed)?
<clever>
pigz.out 126,848 x /nix/store/hx121lsjmk768nv5x928bhpn2xyz1n2k-pigz-2.3.3/bin/pigz
<clever>
pigz, will gzip in parallel
<dtz>
but that's not really important and this way it stays that way :)
<clever>
`gzip -9` will take 900kb blocks of input, and compress each, in series
<dtz>
*stays encrypted, never leaves host unenc by accident, etc.
<simpson>
dtz: ssh can't not encrypt. The idea is that the blob will be encrypted *even if* the other end is controlled by Eve or Mallory.
<clever>
pigz, just takes each 900kb block, and compresses it in a different thread, and then re-assembles the compressed blocks
<gchristensen>
simpson: I do want to know what the write end is doing
<gchristensen>
though, really, I'd like a pv at the start and end of that chain
<simpson>
dtz: And then application of Cryptographic Principle of Doom forces us to compress before encrypting, defeating any compression ssh might apply. (ISTR that SSHv2 compression is risky anyway?)
<simpson>
gchristensen: I think that you can do that! TIAS with another pipe first.
<gchristensen>
yeah, I could have
<gchristensen>
also, compressing prior to encryption exposes some attacks (CRIME)
<simpson>
I don't know if there's CRIME for SSHv2 out there, but the principle exists, and I found a mailing-list post where somebody who is not you is asking about whether CRIME could affect their use of OpenSSH to transfer Pg backups.
<samueldr>
(additionally, not sure the e-mail from aszlig to the linux-unioinfs list made it :/)
ckauhaus has quit [Quit: WeeChat 2.2]
<dtz>
:(
<samueldr>
ouch
<samueldr>
even though they maybe fixed a bug (honouring this constant) this is breaking the userspace
<dtz>
that's my understanding, yeah. And even if they bump the default, or we bump ours... it's uncomfortable to rely on folks' kernels not enforcing the limit it's had for ages lol :/. idk hopefully there's a good resolution ... O:)
<samueldr>
aszlig: sorry to ping you again, that's just in cas your client is sensitive to beginning of a line; I'm really not a mailing list person, so I wouldn't know what to do about the presumably lost e-mail
<aszlig>
samueldr: well, i could bounce it again, but not sure that would change anything
<aszlig>
samueldr: another way would be to ping the maintainer via github...
<samueldr>
maybe reply without CCing anyone nor the list to the original recipient, without the long e-mail, asking if they received it?
<samueldr>
I'm thinking maybe the CCs+attachments are the root cause?
<aszlig>
samueldr: well, the CCs shouldn't be a problem... but maybe the attachments were?
<samueldr>
anything is plausible
<aszlig>
samueldr: i'll send again without the attachments
<aszlig>
5A560980DD95A: to=<linux-unionfs@vger.kernel.org>, relay=vger.kernel.org[209.132.180.67]:25, delay=4, delays=0.36/0.05/1.1/2.5, dsn=2.7.0, status=sent (250 2.7.0 nothing apparently wrong in the message. BF:<H 0.000907487>; S1726506AbfBBR3a)
<aszlig>
samueldr: now it worked
<samueldr>
I just realized the android app for gmail shows you as an octogonal interrogation mark, which means "could be a spammer"; the description is "Unauthenticated email; Gmail couldn't verify that nix.build actually sent this message (and not a spammer)." which sounds like something that with other criterions would count against you :/
<aszlig>
samueldr: yah, that's probably because i'm missing an SPF record
<samueldr>
yeah
<aszlig>
which shouldn't be an issue with the ML
__Sander__ has quit [Quit: Konversation terminated!]
<aszlig>
i mean, when looking through the archives i found real spam with forged sender addresses, which went through
<aszlig>
samueldr: i'll post the mailing list link as soon as it's on spinics.net, because marc.info pretty much garbles up the mail =)
<samueldr>
thanks again <3
<samueldr>
once the ball is rolling with a dicussion there I hope some progress is going to happen
drakonis1 has joined #nixos-dev
<gchristensen>
ooooooooooh crap
<samueldr>
uh?
<gchristensen>
the shebang problem :D
<samueldr>
right
* samueldr
is swimming in kernel regressions
<samueldr>
gchristensen: what's your opinion for the new_kernel medias (iso image, sd image) and zfs? what should be done?
* samueldr
gasps in java
<samueldr>
in job ‘nixpkgs.chromium.aarch64-linux’: You must accept the Oracle Binary Code License Agreement for Java SE at ...
<gchristensen>
imo? leave it failing until 0.7.13 is out
<samueldr>
gchristensen: I figure it will continue failing in the future
<gchristensen>
oh?
<samueldr>
new_kernel is a moving target that will not be updating in lock-step with zf
<samueldr>
zfs*
<gchristensen>
sure..
<gchristensen>
what is this new_kernel thing?
<gchristensen>
and, what is its promise / does it get published anywhere?
<samueldr>
uses linuxPackages_latest instead of the default (which is usually latest LTS)
<samueldr>
not published except on hydra, but its usefulness is in two parts
<gchristensen>
anyone have a feel for how the kernel team sees NixOS?
<samueldr>
I only reported one issue to the kernel, but haven't felt *anything*, for my report it looks like it was an implementation detail to the issue (the issue was provably not nixos specific)
<gchristensen>
hrm
<gchristensen>
ok
<aszlig>
first need to have a vm test that actually reproduces the problem...
<MichaelRaskin>
gchristensen: what do you want to report that kernel developers' opinion of NixOS is omportant?
<gchristensen>
I don't understand the question
<MichaelRaskin>
> <gchristensen> anyone have a feel for how the kernel team sees NixOS?
<{^_^}>
error: syntax error, unexpected ')', expecting ID or OR_KW or DOLLAR_CURLY or '"', at (string):218:1
<gchristensen>
I don't care too much about what they think, but am aware that the context might matter. "breaking userspace for debian" is a different scale than "breaking userspace for Hannah Montana Linux".
<MichaelRaskin>
Well, they also probably break some amount of userspace pip installations if they strengthen the check
<gchristensen>
yeah
<MichaelRaskin>
For Nixpkgs kernels we might start to bump the limit already.
<gchristensen>
and anyway, just curious
<gchristensen>
it is cool to see (for example) the broad change in opinion of Debian team members of Nix :)
<MichaelRaskin>
Well, I guess we did demonstrate the ability to survive long enough to be worth _discussing_ an exception seriously
<gchristensen>
yeah :) definitely
<MichaelRaskin>
The argument «FHS is to prevent surprises, if you install Nix explicitly, you will be _surprised_ not to see /nix» is actually a reasonable one
<gchristensen>
we should have a 16-year party on june 15
<ekleog>
2**2**2-year party
<ekleog>
next one will be in 67539, so it's a once-in-a-lifetime chance
worldofpeace has joined #nixos-dev
<gchristensen>
:D
<infinisil>
> pow 3 (pow 3 3)
<{^_^}>
7625597484987
<infinisil>
> pow 2 (pow 2 (pow 2 2))
<{^_^}>
65536
<infinisil>
Ahh
<aszlig>
ekleog: or if you use bits, it's in 2035 which would at least be twice-in-a-liftime :-)
<infinisil>
Question: For a NixOS option that controls whether the configuration is declarative via NixOS or stateful, what should it be named?
<infinisil>
Should I have a `stateful` option, or a `declarative` one?
<infinisil>
Existing options of similar purpose: `services.znapzend.pure` and `security.dhparams.stateful`
<worldofpeace>
Declarative operation should be expected, so I'd go with `stateful`.
<infinisil>
One thing I should mention is that I'll have to make it stateful by default for backwards compatibilities sake
<worldofpeace>
Though better names are always better :D
<infinisil>
worldofpeace: You still think it should be `stateful` with a default = true?
<worldofpeace>
infinisil: I think so
<infinisil>
Because I almost feel like `declarative` with a default of false would be better, since then the user needs to set a positive value if they want it declaratively (which is something positive)
<ekleog>
infinisil: there's also `services.openldap.declarativeContents`
<ekleog>
I personally like `declarative`
<infinisil>
Me too right now
<ekleog>
as for the backwards compatibility matter… well, we *can* break backwards compat between releases if we want to, can't we? whether it's a good idea is another question
<worldofpeace>
A release should suffice then, no?
<infinisil>
I personally am annoyed every time when something breaks, I'd rather not bring this on people
<ekleog>
well, I'm too, but I run release-18.09 for this reason, and read release notes
<ekleog>
(and am definitely annoyed when a release note is missing, but that's another question :p)
<ekleog>
ideally there'd be an intermediate release of only warning, though
<infinisil>
Okay so the concrete usecase is minecraft-server. Currently you can only run it statefully in NixOS, but there's a PR to add declarative options
<worldofpeace>
Now I get it :D
<infinisil>
And the config file of minecraft always sets all options, even when it could just use the defaults, so it would be a bit of a pain to have to figure out how to translate that to a NixOS option
<infinisil>
Because you don't want to explicitly set the defaults, but also you're not really sure what the defaults are
<ekleog>
why wouldn't you want to explicitly set the defaults? they're set in the stateful config file anyway, IIUC
<infinisil>
Yeah, but you don't need them, I don't want to set all 25 values in my configuration.nix when all of them but one are the defaults anyways
<ekleog>
well, you don't need them, but I mean it wouldn't change any visible behavior if you did (and actually not setting them would change visible behavior after an upgrade of the server that changes the defaults)
<infinisil>
And anyways, there is also a whilelist option, which needs playername and playeruuid, which can be annoying, since I believe minecraft has commands to add to that file ingame
<ekleog>
that… sounds like a PITA
<ekleog>
no way to have those split to a separate config file I guess?
<infinisil>
By 'those' you mean?
<ekleog>
the whitelist's playername&uuid
<infinisil>
Oh, whitelist is a different file
<ekleog>
hmm'k
<ekleog>
so it means having some script for updating the whitelist like for /etc/passwd / groups, I guess?
<infinisil>
Eh, I think it's fine to be stateful by default
<infinisil>
There's no harm to this
orivej has joined #nixos-dev
<ekleog>
indeed, sounds like it's hard-to-impossible to not be stateful anyway
<ekleog>
like the nextcloud config and issues that ensue :/
<infinisil>
If you don't care about minecraft changing your configs then it shouldn't be a problem
<infinisil>
(because it can't if its declarative)
<infinisil>
Alright I think I'll name the option `declarative`
<infinisil>
I'll also need a `.declarative` file in the dataDir to detect state/decl, so this fits
drakonis1 has quit [Quit: WeeChat 2.3]
orivej has quit [Remote host closed the connection]
orivej has joined #nixos-dev
orivej has quit [Remote host closed the connection]
orivej has joined #nixos-dev
orivej has quit [Read error: Connection reset by peer]