gchristensen changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | 18.09 release managers: vcunat and samueldr | https://logs.nix.samueldr.com/nixos-dev
init_6 has quit [Ping timeout: 250 seconds]
<gchristensen> samueldr: back to ZHF, where does that script live
<gchristensen> ?
<gchristensen> oh hi
<samueldr> it's parsing the DOM from hydra responses, as the API responses have less information
<gchristensen> O.o
<gchristensen> awesome
<samueldr> (at that point I didn't use nokogiri as I thought it would be fine without it, it is for the eval page, but not for the build page)
<gchristensen> I was tempted to just nix-instantiate and query the cache for each drv's results but it wouldn't link to build failures
<bennofs[m]> samueldr: what info is missing from the API?
<samueldr> I don't remember
<samueldr> ah, bennofs[m] the https://hydra.nixos.org/eval/1496375 as json only returns build numbers, not the status
<samueldr> I could have alternatively hammered the API with 30k requests to get the build status
<samueldr> (though likely to be fewer than 30k as I would have cached the replies as I do anyway)
<bennofs[m]> samueldr: ah. I think when https://github.com/NixOS/hydra/pull/529 is deployed it'd cover that
<{^_^}> hydra#529 (by bennofs, 48 weeks ago, merged): feat: add /eval/<id>/builds endpoint
<samueldr> I didn't see an API doc so I assumed only endpoints with HTML pages existed :)
<bennofs[m]> it's not yet live on hydra.nixos.org
<samueldr> yeah, as I just saw
<samueldr> so, bennofs[m], you're... used to the hydra codebase?
<bennofs[m]> no. only added that because wanted it for nix-index
<samueldr> :(
<samueldr> I wanted you to check a PR :)
<samueldr> my experience with perl, and the framework used, is pretty minimal
<gchristensen> maybe they can help anyway :)
<samueldr> maybe, hydra#619 is the PR, but it might be dependent on good knowledge on the framework
<{^_^}> https://github.com/NixOS/hydra/pull/619 (by samueldr, 2 weeks ago, open): Adds error messages to lazy tabs
<gchristensen> oh cool
<gchristensen> never heard of a lazy tab
<samueldr> the dashboard is lousy with lazy tabs :)
<samueldr> ideally, I would like to expand the "lazy" paradigm so that the same route will know whether to render the full page or only the lazy tab. From there, instead of using anchors in URLs to distinguish tabs, it would be able to fully render the DOM server-side and be a "real" page
<samueldr> while still allowing the lazy action to happen
<samueldr> but I'm 99.9% sure the implementation is "not great"
<gchristensen> bennofs[m]: deployed, but I can't seem to make it work
pie___ has joined #nixos-dev
<gchristensen> I'd check logs, but hydra-update-gc-roots is running and that is not a nice time to examine logs
<samueldr> gchristensen: it works
pie__ has quit [Remote host closed the connection]
<samueldr> curl --header "Accept:application/json" https://hydra.nixos.org/build/86015916 |
<samueldr> jq | less
<samueldr> oops
<samueldr> no
<samueldr> wrong url
<samueldr> (but same principle, you'll need the header)
<samueldr> seems to "work"
<gchristensen> right, gotcha, cool
<samueldr> it's still loading
<samueldr> while a 404 is instant
<bennofs[m]> hmm it times out... maybe this is too much for hydra
<samueldr> other than a boatload of data, is there anything performance-wise specific to the hydra hosting instance?
<samueldr> (I was thinking the other day that I may need to generate placeholder data to try things in hydra)
<bennofs[m]> not sure why it'd take that long though. as it should only be a simple select matching on the evaluation id...
<samueldr> OUCH
<gchristensen> didn't time out for me
<samueldr> it worked
<samueldr> didn't have time to copy the line
<samueldr> but it's... big?
<gchristensen> but I sort of wished I hadn't printed it to stdout
<bennofs[m]> yeah it has all the infos for all the builds in the eval
<bennofs[m]> might be just the perl json serializer that takes some time? :D
drakonis has joined #nixos-dev
<gchristensen> I mean
<gchristensen> it has to do a lot of DB stuff, and then Perl stuff, and then serializing stuff
<gchristensen> its just a lot
<samueldr> and it's really a boatload of data :)
<bennofs[m]> oh ok maybe we need to find another way to do this
<samueldr> oh, at least it doesn'T look memory starved
<gchristensen> is that a joke?
<samueldr> oops, misread the graph
<gchristensen> ah :)
<gchristensen> maybe that graph should be reversed
<samueldr> maybe
<samueldr> my home machine's memory graph looks like that https://stuff.samueldr.com/screenshots/2018/12/20181218202849.png
<samueldr> yes, the green bit at the bottom is the important one :/
<gchristensen> I haven't seen rrd graphs in a hot minute, nice
<samueldr> so I kinda made the assumption I was looking at the same kind of graph
<samueldr> I use munin on my machines, seems "worse is better" in my case, no central authority
<gchristensen> heh, if I use the query: (((node_memory_MemFree_bytes{instance=~"^($instance):.*",role=~"^($role)$"}+node_memory_Buffers_bytes{instance=~"^($instance):.*",role=~"^($role)$"})/node_memory_MemTotal_bytes{instance=~"^($instance):.*",role=~"^($role)$"})) it works fine
<gchristensen> ohhh nevermind
<gchristensen> bennofs[m]: that API endpoint took 7min to return, and produced 45.7M of data
<bennofs[m]> perhaps we can generate the data once per evaluation instead and serve it via the cache, like store-paths.xz?
<gchristensen> well it is useful for other stuff too, like reporting on evals
<gchristensen> json may just not be the right tool for the data
<bennofs[m]> not sure if perl is the right thing to process those amounts of data either
<gchristensen> perls fine
<gchristensen> its just 45M / ~50,000 rows
<gchristensen> anyway, what were you going to use this endpoint for?
<bennofs[m]> I guess the table isn't sorted by evaluation id?
<bennofs[m]> would be a nicer way to generate the nix index, without requiring "large" (>4G) amounts of ram for evaluation the jobsets :) also I wouldn't need to ensure everything is the same between hydra & client for nix-index, but instead could point it at an evaluation and index everything built there
<gchristensen> the evaluation table and build table are joined by a jobsetevalmember table, with a foreign key / index
<bennofs[m]> oh now it makes sense that it takes long. I thought the build table just had a direct key to the evaluation id
<gchristensen> well it shouldn't make a big difference, both of those indexes should be clean hits
<gchristensen> yeah, that query took 7 seconds. the rest of the time was probably in serializing
<gchristensen> at any rate, we probably shouldn't be having clients hit this API endpoint with regularity
jtojnar has quit [Quit: jtojnar]
jtojnar has joined #nixos-dev
jtojnar has quit [Client Quit]
Lisanna has quit [Remote host closed the connection]
<samueldr> ah! a prime sample of why one would desire lazy "pages" to fully render a page. https://hydra.nixos.org/jobset/nixpkgs/trunk/jobs-tab?filter=uboot
sir_guy_carleton has joined #nixos-dev
worldofpeace has joined #nixos-dev
pie__ has joined #nixos-dev
pie___ has quit [Ping timeout: 250 seconds]
Lisanna has joined #nixos-dev
sir_guy_carleton has quit [Quit: WeeChat 2.2]
lassulus_ has joined #nixos-dev
lassulus has quit [Ping timeout: 272 seconds]
lassulus_ is now known as lassulus
drakonis has quit [Quit: WeeChat 2.3]
Lisanna has quit [Ping timeout: 272 seconds]
copumpkin has quit [Read error: Connection reset by peer]
copumpkin has joined #nixos-dev
pie___ has joined #nixos-dev
pie__ has quit [Remote host closed the connection]
FRidh has joined #nixos-dev
__Sander__ has joined #nixos-dev
FRidh has quit [Read error: Connection reset by peer]
Lisanna has joined #nixos-dev
worldofpeace has quit [Quit: worldofpeace]
<bennofs[m]> Anyone know what happened at https://github.com/NixOS/nixpkgs/pull/52454? Doesn't ofborg check tarball evaluation on all platforms? (Tarball failed at https://hydra.nixos.org/build/86015607)
<{^_^}> #52454 (by bennofs, 1 day ago, merged): adoptopenjdk: autogenerate sources + add support for aarch64
<bennofs[m]> Or am I misunderstanding what ofborg does?
<{^_^}> nixos-weekly#72 (by domenkozar, 2 weeks ago, open): Call for Content: 2018/14
<domenkozar> I'm sure gchristensen has something :)
<sphalerite> bennofs[m]: I think ofborg only checks evaluation of packages
orivej has joined #nixos-dev
asymmetric_ has joined #nixos-dev
pie___ has quit [Remote host closed the connection]
pie___ has joined #nixos-dev
<domenkozar> niksnut: is there a reason why https://github.com/edolstra/nix/commit/58d1980de426f1c79c28d31d6e37c75366b0dd17 is not in master?
orivej has quit [Ping timeout: 250 seconds]
<niksnut> domenkozar: it was mostly needed for recursive nix and multithreaded daemon, which are not in master either
<niksnut> also, it changes the schema
zimbatm has quit [Ping timeout: 252 seconds]
zimbatm has joined #nixos-dev
init_6 has joined #nixos-dev
<domenkozar> niksnut: that's when not using the daemon right?
<domenkozar> I guess it's not much better than current form where it also fails? :)
<asymmetric_> is it possible to e.g. set env variables in a shell.nix, or start daemons?
<asymmetric_> for a dev environment
asymmetric_ is now known as asymmetric
<niksnut> domenkozar: hm, what fails?
<domenkozar> well two nix-store -r fail if they touch same paths
<domenkozar> with the deadlock message
<asymmetric> (sorry, wrong chan)
<{^_^}> nix#2207 (by nh2, 28 weeks ago, open): acquiring/releasing lock: Resource deadlock avoided
<niksnut> ah, forgot about that
<niksnut> yeah that might be a reason for cherrypicking it to master
<niksnut> if somebody can confirm it fixes the problem for them
<domenkozar> can do that
<domenkozar> also maintenance branch?
<niksnut> no
<niksnut> since it changes the schema
<niksnut> and it's far from clear that it will fix the problem
<domenkozar> you mean it changes behaviour if two different nix versions operate on nix store?
<domenkozar> there are no schema changes per se
<niksnut> right, they won't see each other's locks so it can lead to store corruption
<niksnut> e.g they could both start building the same path concurrently
<domenkozar> but shouldn't that basically be idempotent?
<domenkozar> or are some sql actions not? :)
<niksnut> it's not about the sql database (which uses its own locking) but the locks on store paths etc.
<niksnut> /nix/var/nix/db/schema it's really the schema of the sql db but the entire store
<niksnut> s/it's/isn't/
<domenkozar> I mean, what's the harm if in the edge case of two nix having different locking building the same paths
<domenkozar> (besides the obvious time wasted)
<domenkozar> not sure if the store path creating is atomic operation :)
<domenkozar> creation*
orivej has joined #nixos-dev
drakonis has joined #nixos-dev
sir_guy_carleton has joined #nixos-dev
<domenkozar> seems like current locking isn't multi-process safe anyway :)
init_6 has quit []
<sphalerit> domenkozar: lgtm
<domenkozar> tnx!
<asymmetric> > Nix community attended Reproducible builds summit - maybe i'd say "The Nix community" here
<{^_^}> undefined variable 'Nix' at (string):214:1
<asymmetric> and also "the Reproducible summit"
sir_guy_carleton has quit [Quit: WeeChat 2.2]
sir_guy_carleton has joined #nixos-dev
<domenkozar> too late :/
domenkozar has quit [Changing host]
domenkozar has joined #nixos-dev
ikwildrpepper has quit [Changing host]
ikwildrpepper has joined #nixos-dev
makefu has quit [Changing host]
makefu has joined #nixos-dev
infinisil has joined #nixos-dev
infinisil has quit [Changing host]
zimbatm has quit [Changing host]
zimbatm has joined #nixos-dev
sphalerit has joined #nixos-dev
sphalerite has joined #nixos-dev
sphalerit has quit [Changing host]
sphalerite has quit [Changing host]
srhb has quit [Quit: ZNC 1.7.1 - https://znc.in]
orivej has quit [Ping timeout: 240 seconds]
drakonis has quit [Quit: WeeChat 2.3]
srhb has joined #nixos-dev
<timokau[m]> I think skipping the "the" is fine for something headline-y
<ekleog> just noticed something: 10 calendar days mean the NixOS FCP announcement in NixOS Weekly comes out the day before its end… guess there's little other way to handle it other than make FCPs much longer, though
<ekleog> (though it's kind of weird, was a nixos weekly delayed? should have been 3 days, or maybe we missed one weekly?)
<domenkozar> there was a single item last week
<domenkozar> so I usually don't send the email in that case
<ekleog> oh, so that's why :)
<ekleog> makes sense
<domenkozar> but this is a good proof that maybe I should? :)
<ekleog> just felt weird reading the FCP announcement and noticing it's actually today (in $timezone)
<ekleog> I guess we'll soon have an official RFC process \o/
<Synthetica> Soon ™️
<ekleog> domenkozar: as for nixos weekly, I guess maybe information that has limited validity in time (eg. FCPs, announcements for eg. the end of the sale of nixcon tickets, etc.) may trigger a weekly even if there is nothing else? and/or keep a log of ~2 non-deadline'd items so that if such an empty week with a deadline'd item comes up it's possible to fill it
ekleog has quit [Quit: back soon]
ekleog has joined #nixos-dev
janneke has joined #nixos-dev
drakonis has joined #nixos-dev
<domenkozar> ekleog: that makes sense, I would send it out if I noticed the content of the single item :D
drakonis_ has joined #nixos-dev
drakonis has quit [Ping timeout: 244 seconds]
__Sander__ has quit [Quit: Konversation terminated!]
orivej has joined #nixos-dev
jtojnar has joined #nixos-dev
pie__ has joined #nixos-dev
pie___ has quit [Remote host closed the connection]
asymmetric has quit [Ping timeout: 250 seconds]
drakonis_ has quit [Ping timeout: 252 seconds]
orivej has quit [Ping timeout: 250 seconds]
drakonis has joined #nixos-dev
<samueldr> I'm about to send a PR to re-enable aarch64-linux as supported in nixos/release-combined; but it was reverted more than a year ago due to issues with memory usage on hydra
<gchristensen> butt.holdings
<samueldr> anything I can do to check what's up, if it's going to cause issues?
<gchristensen> just do it and see what happens?
<samueldr> other than that :)
<gchristensen> watch hydra to see if it stops evaluating
<samueldr> (I'll do it anyway, just asking nicely before)
<samueldr> I want hydra to produce sd_image and iso for aarch64 for nixos-unstable
<gchristensen> yes please
<samueldr> (and it does already for 18.09, but not a combined three-platform eval)
<gchristensen> will this make aarch64 a blocker for the channel?
<samueldr> hmm, I think it might
<gchristensen> great
<samueldr> unless it's added to limitedSupportedSystems
<samueldr> depending on what's desired :)
<gchristensen> if we can manage it, everything is much simpler by having aarch64 be part of the channel,
<tilpner> Is there a reason other than "not overwhelming users with channel selection" for why channels depend on multiple systems?
<samueldr> right now tests often fail due to failing deps :/
<gchristensen> tilpner: for one, it is extremely annoying to have your nixops network depend on several versions of nixpkgs
<samueldr> (e.g. qemu timing out fails all vm tests)
<tilpner> I would guess many people use just one platform, and only get disadvantages from their channels blocking on multiple platforms
<tilpner> But of course the granularity of channels offers a lot of space for discussion
<tilpner> Hmm, I wouldn't know. I deploy my machines separately and without nixops :/
drakonis_ has joined #nixos-dev
<gchristensen> really though, anything we want to produce very good support for should be a blocker
drakonis has quit [Ping timeout: 252 seconds]
<gchristensen> even if not everybody uses it
hedning has joined #nixos-dev
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 252 seconds]
asymmetric has joined #nixos-dev
drakonis has quit [Ping timeout: 252 seconds]
<samueldr> right, I'll be finishing that later tonight, there may be an issue with how `tested` is built; where it maps all supported platforms on everything and... a couple tests will fail to eval on aarch64-linux
<samueldr> (well, not fail to eval, but don't have an aarch64-linux attribute, so *that* will fail to eval)
<gchristensen> send a simple revert and PR it?
<samueldr> revert to?
<gchristensen> or does ofborg not check that... hrm...
<samueldr> nah, I did eval, and I remember that ~2 weeks ago I tried something that also resulted on seeing the same issue :)
<samueldr> either tonight I'll add it as a limited supported platform (so it will at least block on sd image and iso failures) or I'll look into what it takes to fix these issues
drakonis has joined #nixos-dev
drakonis has quit [Read error: Connection reset by peer]
drakonis_ has joined #nixos-dev
orivej has joined #nixos-dev
pie___ has joined #nixos-dev
pie__ has quit [Ping timeout: 268 seconds]
hedning has quit [Quit: hedning]
asymmetric_ has joined #nixos-dev
asymmetric has quit [Ping timeout: 240 seconds]
drakonis_ has quit [Ping timeout: 252 seconds]
drakonis_ has joined #nixos-dev
sir_guy_carleton has quit [Ping timeout: 250 seconds]
apaul1729 has joined #nixos-dev
boredom101 has joined #nixos-dev
<boredom101> Just wondering, is there a desire for pull requests that add new options to existing packages?
<simpson> Sure, the options aren't set in stone. What did you have in mind?
<boredom101> I was thinking more options for KDE applications.
<boredom101> is that possible?
<simpson> Probably. Do you have code for your PR ready?
<boredom101> no, sorry. But this is somethings I would like to try to do.
boredom101 has quit [Quit: Konversation terminated!]