<gchristensen>
garbas: Nick is happy to give us the plan with read-only elasticsearch auth tokens, but before he definitely commits to giving this to us for free he wants to see our schema, dataset size, and queries a bit. he doens't want us to tell him, though, he wants us to use it and then he'll look at it all and evaluate it from his side.
<gchristensen>
with regards to dataset size, I think we should probably create a new index each time the channel updates, and then rename it to be the one the frontend searches
<gchristensen>
and then delete the old one
<garbas>
i was thinking in another direction, to just keep everything in one index
<garbas>
all packages of all evaluations of all channels of all projects
<niksnut>
I mean, do we *really* need this? add a complex dependency on yet another proprietary service, for something that mostly works fine today?
<gchristensen>
niksnut: it isn't proprietary
<niksnut>
in the same way that github isn't
<gchristensen>
huh?
<garbas>
niksnut: it works fine but we would like to expand the functionality
<gchristensen>
bonsai provides elasticsearch hosting, that is all, and elasticsearch is truly oss
<niksnut>
we're talking about hosted elasticsearch right?
<gchristensen>
yeah
<garbas>
yes
<niksnut>
and github provides git hosting
<garbas>
in the worse case we can host git or elasticsearch it ourself
<gchristensen>
and issue trackers and wikis and PR workflow :P
<garbas>
s/it//
<gchristensen>
bonsai doesn't add much (anything?) on top of elasticsearch
<niksnut>
anyway my main objection is the complexity
<niksnut>
the current solution is much, much simpler
<garbas>
for a simple search yes, i agree. if we want to do something more (which i think we want) i would like to explore this option.
<garbas>
we wont remove current search but try to improve it along side
<tilpner>
You said this would allow future functionality expansion. What specifically could it provide over the current search, except speed and bandwidth improvements?
<gchristensen>
I think there are nice advantages to a proper search engine, for example we could index documentation too
<garbas>
then we can see if this bring value. if not, we trash it and close all the tickets that ask for improvements of current search
<gchristensen>
the current method is much simpler, that is true
<niksnut>
what we discussed yesterday (move to releases.nixos.org, compress with brotli) is a much simpler solution to the current netlify problem
<garbas>
tilpner: i'll write a proposal with mentioned features we would like in a ticket/PR and we can comment there. i think chat is not the best to have this discussions
<garbas>
niksnut: this solution that was proposed is actually already step one of moving to elasticsearch.
<garbas>
niksnut: we discussed how we could improve the search with samueldr when it hit us that we could just start with already planned step one and solve also the netlify issue
<tilpner>
garbas: Thank you. I'm asking because these (far-away?) scope expansions might alter the usage pattern (and required resources), and any free hosting plans currently in negotiation might not be sufficient then
<garbas>
tilpner: this is actually done with simplification in mind (in a way). i'd like to have nixos-homepage really slim and content oriented. currently having a packages/options search adds quite a build step with all the npm packages. separating this in a separate project (eg. search.nixos.org or similar) will make changes to nixos.org actually easier.
<gchristensen>
tilpner: I would be surprised if this free plan stopped being free, but it is possible
<tilpner>
I realise how dangerous scope expansion is, so feel free to tell me to go away. But a few months (year?) ago, I was talking with gchristensen about making the current Hound instances (search.nix.gsc.io, search.tx0.co) more official. Should that be considered at this point?
<niksnut>
but if we move packages/options to release.nixos.org, it's already no longer part of the website
<garbas>
niksnut: i was refering to the search code (javascript), the data is moved out yes.
<garbas>
tilpner: the hound search would be a nice addition to help developers contribute. the search which i want to improve is more geared towards end users and people exploring nix/nixos
<tilpner>
garbas: I'm biased (search.tx0.co is mine), but I think it's very useful to quickly find examples of how to use NixOS
<garbas>
tilpner: and regarding scope expansion. i might be wrong so please correct me, but if there is a clear way how to remove something from the whole picture i think is worth expanding too. especially if it bring value. which you don't know until you try it. and even then you might not konw it (should we track usage? :P)
<tilpner>
Sorry, I don't understand your last message. What are you saying here?
<garbas>
happens a lot to me :)
<garbas>
tilpner: i don't see scope expansion being a problem if this is something that can be removed at any time.
<tilpner>
Well, it takes more time and reduces chances of this getting done
<garbas>
also you need to give it a try to actually know if something is useful. otherwise it is just speculations
<tilpner>
But a unified search portal for the entire ecosystem sounds too nice to not bring it up right now
<tilpner>
Well, we did give it a try already
<garbas>
you mean with hound?
<garbas>
or you are talking about packages/options search on nixos.org?
<garbas>
tilpner: ^
<tilpner>
Yes, both Hound instances have been used for quite a while
<tilpner>
People are generally happy to find them, but they are not very discoverable right now
<garbas>
hound is a developer tool, packages/options are end user tools. i think both have a place to be somewhere more discoverable.
<tilpner>
No, you missed my point. Hound is not just a developer tool
<garbas>
that is why i started working on nixos.org. since a lot of good things are not discoverable :)
<garbas>
oh it is not. sorry i must have missed it
<tilpner>
Some people learn by examples, and my Hound instance allows them to quickly search through many user configurations at once
<garbas>
so it shows nix code?
<tilpner>
Well, it shows anything from the indexed repos, but that's just because it was easiest for me
<garbas>
can you configure it to search among packages? and options?
<tilpner>
But NixOS users have to write Nix too, so it's not just for developers
<tilpner>
No, Hound is a text search tool
<tilpner>
But I'm not suggesting we adopt Hound, I'm asking if we can expand the scope of your hypothetical search.nixos.org to also search user configurations and repositories of the nixos organisation
<tilpner>
Probably not at first, it would be enough to account for that possibility in the design
<garbas>
oh yes, ofcourse. i never said no. just we need to think how to present this into a single page
<tilpner>
I don't mind multiple search pages, if there's a header to navigate between them
<garbas>
tilpner: once we have something around ourpart of the search would it be ok to ping you and we can bring also hound into it? this will probably in a month or so
<tilpner>
Sure. I want the functionality, not certain we want to keep hound. I have no idea what elasticsearch can do, and if it could be made to do what we're currently using hound for
teto has quit [Ping timeout: 246 seconds]
teto has joined #nixos-infra
<gchristensen>
apparently postgresql on zfs (haumea) should have wal_init_zero = off, wal_recile = off
<gchristensen>
apparently postgresql on zfs (haumea) should have wal_init_zero = off, wal_recycle = off