<ekleog>
I feel like this was already discussed but forgot the answer: it isn't possible to get aggregate statistics on the number of eg. monthly downloads per cached path, right?
<gchristensen>
correct
<ekleog>
is it for technical reasons, or for anonymity reasons?
<ekleog>
(And in the second case, have we thought about how we could do it anyway? I'm thinking of a package I maintain, that I wonder whether I could not just remove it because I'm ~sure basically no one is using it any longer)
<gchristensen>
it doesn't really say nobody uses it, just that nobody using the cache uses it
<gchristensen>
also you're almost guaranteed to see some cache hitsfetching it due to mirroring
<gchristensen>
maybe a better approach would be to mark it broken?
<ekleog>
(it's still working and afaik secure, but it probably reduces discoverability and contributes to making nixpkgs host to very niche packages that are hard to distinguish from actually well-established packages — I often find myself having to look at other package repositories to check I'm not installing someone's almost-hobby-project when installing something)
<ekleog>
hmm right but I was thinking that we could establish some “noise level” by eg. looking at the first percentile (or at the 0.1%?) of downloads or something, and decide that this likely means packages below that are unused enough that it'd maybe be better to remove them
<gchristensen>
well still it doesn't actually mean nobody uses it since there are lots of users who don't hit the cache
<ekleog>
(said package I maintain is something I did as a hobby project, that got added to nixpkgs after asking people when two of us used it in the nixpkgs community, and now I stopped using it though afaik it still works well so I'm not sure there's a greater-than-one number of people using it… and even the other one I'm not sure is still using it)
<gchristensen>
yeah, so maybe marking it as broken as the first step
<ekleog>
right, but OTOH if only people who don't hit the cache are using some software… doesn't that hint at the package being better off outside of nixpkgs?
<gchristensen>
that'll help find out if people use it or not
<ekleog>
hmm it'd be nice if it were possible to add a message to meta.broken, to eg. tell people “this package is scheduled for removal after 21.10, please contact foo@bar if you still use it” or something similar
<gchristensen>
maybe add that to the description?
<ekleog>
(ISTR that adding lib.trace to a package is a bad idea because it breaks the tarball job, so… not that I guess)
<gchristensen>
right
<sterni>
ofborg also doesn't like lib.trace
<gchristensen>
both of these things don't like lib.trace to keep nixpkgs' evaluation clean for end users, and keep nix-env usable
<ekleog>
Anyway, this discussion, I think, properly describes the current best practice on how to deprecate and hopefully remove a package one knows is useless. But, unless we have something like cache stats, how can we know which packages we think are used (or even, which packages no longer has a maintainer and no one even know exist) are actually unused?
<gchristensen>
we've talked about popcon data many times
<gchristensen>
it'd be interesting to see a coherent popcon proposal
<samueldr>
deprecation is something we have no policy about and is bound to end up causing issue
<samueldr>
issues*
<gchristensen>
g'night :)
<samueldr>
'night!
<gchristensen>
(but before I go, cache stats are not adequate for popcon.)
<ekleog>
'night!
<ekleog>
so I wonder, if a coherent popcon proposal that included cache stats as one among other indicators were written, would it be thrown away because it's actually technically / legally impossible; or would it stand a chance
ris has quit [Ping timeout: 268 seconds]
<ekleog>
(cc gchristensen for ^, hopefully enough time has passed that you're no longer looking at IRC and will only see that ping tomorrow :) as I don't think many other people would be able to answer that question unfortunately :/)
<samueldr>
ekleog: if it's your package, that you maintain, just remove it, with the usual alias explaining the removal "maintainer didn't care anymore about the package, but cares about nixpkgs <3"
<samueldr>
(not literally that, but you know what I mean I hope)
<ekleog>
i actually would like the world to be such that this exact message would be ok
rajivr has joined #nixos-dev
<lovesegfault>
ekleog: it's fine IMHO?
<lovesegfault>
like, better to find a message that says "I don't want to maintain this, sorry" than to see that the package is in nixpkgs, go to `nix-shell` it, and then find out it's broken
<lovesegfault>
open an issue, and have the maintainer take forever to answer because they don't care about the package
supersandro2000 has quit [Disconnected by services]
supersandro2000 has joined #nixos-dev
<samueldr>
I think ekleog meant that the wording is a bit cavalier :)
<ekleog>
lovesegfault: right, I meant literally the exact message s.mueldr was writing, with the <3 and all
<ekleog>
as opposed to something like "This package was removed for suspicion of lack of users. If you still used this package, please contact foo@bar to see whether it would make sense to re-add it" as will probably end up being the result
xwvvvvwx- has joined #nixos-dev
adisbladi has joined #nixos-dev
msirabella has joined #nixos-dev
Gaelan_ has joined #nixos-dev
jlotoski-znc has joined #nixos-dev
dottedmag_ has joined #nixos-dev
dottedmag_ has quit [Changing host]
dottedmag_ has joined #nixos-dev
pbogdan has joined #nixos-dev
sgo has joined #nixos-dev
globin has joined #nixos-dev
danderso1 has joined #nixos-dev
<aanderse>
ekleog: while i completely agree unfortunately we'd push away a large audience from nixpkgs if we did that :(
<sterni>
this is unfortunate for what I wanted to do -.-
teto has joined #nixos-dev
<rmcgibbo[m]>
Adding earlyoom to r-rmcgibbo really helped. Rust packages, man...
<sterni>
which reminds me I still need to track down the mystery failure for uunf
<rmcgibbo[m]>
maybe https://github.com/NixOS/nixpkgs/issues/113903 will be good, but the kernel oom killer is just... not amazing. it's pretty lame for the system to be completely unreachable by ssh and still not have the kernel oom killer act. i forgot about all of this since putting 64GB of ram in my deskop minipc.
misuzu has quit [Remote host closed the connection]
misuzu has joined #nixos-dev
costrouc has quit [Quit: costrouc]
rj_ has joined #nixos-dev
rj_ has quit [Ping timeout: 268 seconds]
rj_ has joined #nixos-dev
cole-h has joined #nixos-dev
jonringer has joined #nixos-dev
rj_ has quit [Ping timeout: 268 seconds]
rj_ has joined #nixos-dev
mkaito has joined #nixos-dev
mkaito has joined #nixos-dev
mkaito has quit [Changing host]
justanotheruser has joined #nixos-dev
rj_ has quit [Ping timeout: 268 seconds]
rj_ has joined #nixos-dev
<rmcgibbo[m]>
I notice that sometimes for r-rmcgibbo reviewing PRs to staging, it requires rebuilding stuff like `bootstrap-stage0-glibc`. I assume that's because a lot of new stdenv stuff has gone into staging and hydra hasn't kept up (which is kind of the point of staging). It might make sense for me to try having it build the PR "as merged into" the last evaluation of staging, rather than the current HEAD of staging. Does that
<rmcgibbo[m]>
make sense?
<rmcgibbo[m]>
(Even though the PR doesn't actually modify stdenv, AFAICT. And it's not really a problem -- r-rmcgibbo just skips it. But it does result in some things not getting build that probably could be built.)
rj_ has quit [Ping timeout: 268 seconds]
rj_ has joined #nixos-dev
<supersandro2000>
it should merge the PRs in probably for all PRs like all other tools
<siraben>
doesn't nixpkgs-review perform the merge before building?
<siraben>
I thought r-rmcgibbo did the same
<siraben>
I guess not
<rmcgibbo[m]>
It does merge the PR into the HEAD branch, yes.
<rmcgibbo[m]>
My question was whether that behavior makes sense when the base is staging and there's no stdenv currently in hydra for the HEAD of staging.
rj_ has quit [Ping timeout: 268 seconds]
rj_ has joined #nixos-dev
AlwaysLivid has quit [Ping timeout: 268 seconds]
<supersandro2000>
siraben: yes and ...
<supersandro2000>
didn't it do that for a while?
<supersandro2000>
I kinda remember it did
<supersandro2000>
Maybe you should requeue the task if there is no stdenv available?
<supersandro2000>
or explicitly mention that it is not merged
<rmcgibbo[m]>
Right now, the build just gets skipped (it realizes that rebuilding stdenv would take > 2 hours and bails), and so it doesn't do any work. So this is why some things like https://github.com/NixOS/nixpkgs/pull/115570 got skipped, even though otherwise it's perfectly capable of only rebuilding a subset of the changed attrs.
<rmcgibbo[m]>
But a different alternative would just be to merge the PR branch with a slightly older commit of staging. I guess it might be confusing, but at least that way it would be able to build a nonzero number of changed attrs.
<gchristensen>
sounds like they have much much more active repos that they engaged with
<samueldr>
and "push errors", what are push errors here? anyone getting push errors with Nixpkgs?
<samueldr>
ah
<samueldr>
I see
<samueldr>
they do maintenance work in the background every n pushes
<cole-h>
nothing about unicorns when searching for issues though :(
<tazjin>
gchristensen: there's some large corp repos, iirc Uber amongst others, hosted with them - those will have waaaaay more traffic than nixpkgs does
<gchristensen>
yep
<endocrimes>
we had a lot of issues with the cocoapods specs repo's back in the day
<endocrimes>
but that was partially because of very wide directories and shallow clones being common, both of which kinda suck for git remotes
<siraben>
tazjin: wow, way more active than Nixpkgs?
<endocrimes>
large corp monorepos see a lot of traffic
<endocrimes>
(hundreds of commits per hour and more)
<gchristensen>
there are lots of stories of going on a few days vacation and coming back and waiting ages for `git fetch` to finish
<endocrimes>
hahahaha git fetch on our p4 mirror over VPN from DE is slow enough that $employer sent me a second workstation so I could have it handle that and dependency caching
<gchristensen>
lol.
<endocrimes>
like... if i need to do a clean clone for any reason thats a day+ gone
<samueldr>
Nixpkgs updates at a leisurely pace that a single human being can follow
<samueldr>
(which is good!)
<tazjin>
siraben: we have a little under 100k devs at work and approximately 1 commit per second (though this isn't git), Uber has 2k engineers so if this scales linearly (completely disregarding the amount of commits done by automation) they might see a commit every minute or so
<tazjin>
which is more than nixpkgs :p
<tazjin>
and note that's just actual commits merged, disregarding pushes to branches etc.
<tazjin>
so the actual number is probably multiple pushes per minute during working hours
<endocrimes>
tazjin: huh I think we're more like ~5-10k devs and don't have that much less traffic than you
<endocrimes>
maybe 15-20k. I have no idea outside of my part of the org tbh
<tazjin>
endocrimes: unfortunately the metrics I know are averaged over the day, it likely peaks way higher during California working hours
<tazjin>
also note that these things are kind of naturally bursty, people will push a revision before their meeting starts to kick off their presubmit runs, but most meetings start on the hour etc.
<endocrimes>
I think our main bottleneck is presubmits because for anything in my org they take a full workday
<tazjin>
O_o
<tazjin>
you might wanna .. invest in that :p
<endocrimes>
lol we're fitting more DCs for them
<endocrimes>
can only fit so much server in so little space
<tazjin>
are these necessary integration tests and such consuming resources, or are there a lot of redundant builds?
<endocrimes>
a looot of redundancy rn. reducing that has been a multi year project so far
<endocrimes>
had to get off an old in house build system first xD
<endocrimes>
now we're mostly over to bazel and it's getting easier to build tooling
ris has joined #nixos-dev
<ekleog>
“bazel” “easier” (sorry)
<tazjin>
bazel is great if you have thousands of people
evils has joined #nixos-dev
justanotheruser has quit [Quit: WeeChat 2.9]
weechat3 has quit [Ping timeout: 240 seconds]
weechat3 has joined #nixos-dev
justanotheruser has joined #nixos-dev
<supersandro2000>
rmcgibbo[m]: I don't think that would be a good idea and not that valuable
<supersandro2000>
I get at least once a that my git push or fetch hangs for no reason
<rmcgibbo[m]>
supersandro2000: Okay, I'll leave the behavior as-is (full merge commit into the base HEAD).