samueldr changed the topic of #nixos-infra to: NixOS infrastructure | logs:
zimbatm has joined #nixos-infra
{`-`}_ has joined #nixos-infra
{`-`} has quit [Ping timeout: 246 seconds]
<gchristensen> garbas: Nick is happy to give us the plan with read-only elasticsearch auth tokens, but before he definitely commits to giving this to us for free he wants to see our schema, dataset size, and queries a bit. he doens't want us to tell him, though, he wants us to use it and then he'll look at it all and evaluate it from his side.
<gchristensen> with regards to dataset size, I think we should probably create a new index each time the channel updates, and then rename it to be the one the frontend searches
<gchristensen> and then delete the old one
<garbas> i was thinking in another direction, to just keep everything in one index
<garbas> all packages of all evaluations of all channels of all projects
<niksnut> I mean, do we *really* need this? add a complex dependency on yet another proprietary service, for something that mostly works fine today?
<gchristensen> niksnut: it isn't proprietary
<niksnut> in the same way that github isn't
<gchristensen> huh?
<garbas> niksnut: it works fine but we would like to expand the functionality
<gchristensen> bonsai provides elasticsearch hosting, that is all, and elasticsearch is truly oss
<niksnut> we're talking about hosted elasticsearch right?
<gchristensen> yeah
<garbas> yes
<niksnut> and github provides git hosting
<garbas> in the worse case we can host git or elasticsearch it ourself
<gchristensen> and issue trackers and wikis and PR workflow :P
<garbas> s/it//
<gchristensen> bonsai doesn't add much (anything?) on top of elasticsearch
<niksnut> anyway my main objection is the complexity
<niksnut> the current solution is much, much simpler
<garbas> for a simple search yes, i agree. if we want to do something more (which i think we want) i would like to explore this option.
<garbas> we wont remove current search but try to improve it along side
<tilpner> You said this would allow future functionality expansion. What specifically could it provide over the current search, except speed and bandwidth improvements?
<gchristensen> I think there are nice advantages to a proper search engine, for example we could index documentation too
<garbas> then we can see if this bring value. if not, we trash it and close all the tickets that ask for improvements of current search
<gchristensen> the current method is much simpler, that is true
<niksnut> what we discussed yesterday (move to, compress with brotli) is a much simpler solution to the current netlify problem
<garbas> tilpner: i'll write a proposal with mentioned features we would like in a ticket/PR and we can comment there. i think chat is not the best to have this discussions
<garbas> niksnut: this solution that was proposed is actually already step one of moving to elasticsearch.
<garbas> niksnut: we discussed how we could improve the search with samueldr when it hit us that we could just start with already planned step one and solve also the netlify issue
<tilpner> garbas: Thank you. I'm asking because these (far-away?) scope expansions might alter the usage pattern (and required resources), and any free hosting plans currently in negotiation might not be sufficient then
<garbas> tilpner: this is actually done with simplification in mind (in a way). i'd like to have nixos-homepage really slim and content oriented. currently having a packages/options search adds quite a build step with all the npm packages. separating this in a separate project (eg. or similar) will make changes to actually easier.
<gchristensen> tilpner: I would be surprised if this free plan stopped being free, but it is possible
<tilpner> I realise how dangerous scope expansion is, so feel free to tell me to go away. But a few months (year?) ago, I was talking with gchristensen about making the current Hound instances (, more official. Should that be considered at this point?
<niksnut> but if we move packages/options to, it's already no longer part of the website
<garbas> niksnut: i was refering to the search code (javascript), the data is moved out yes.
<garbas> tilpner: the hound search would be a nice addition to help developers contribute. the search which i want to improve is more geared towards end users and people exploring nix/nixos
<tilpner> garbas: I'm biased ( is mine), but I think it's very useful to quickly find examples of how to use NixOS
<garbas> tilpner: and regarding scope expansion. i might be wrong so please correct me, but if there is a clear way how to remove something from the whole picture i think is worth expanding too. especially if it bring value. which you don't know until you try it. and even then you might not konw it (should we track usage? :P)
<tilpner> Sorry, I don't understand your last message. What are you saying here?
<garbas> happens a lot to me :)
<garbas> tilpner: i don't see scope expansion being a problem if this is something that can be removed at any time.
<tilpner> Well, it takes more time and reduces chances of this getting done
<garbas> also you need to give it a try to actually know if something is useful. otherwise it is just speculations
<tilpner> But a unified search portal for the entire ecosystem sounds too nice to not bring it up right now
<tilpner> Well, we did give it a try already
<garbas> you mean with hound?
<garbas> or you are talking about packages/options search on
<garbas> tilpner: ^
<tilpner> Yes, both Hound instances have been used for quite a while
<tilpner> People are generally happy to find them, but they are not very discoverable right now
<garbas> hound is a developer tool, packages/options are end user tools. i think both have a place to be somewhere more discoverable.
<tilpner> No, you missed my point. Hound is not just a developer tool
<garbas> that is why i started working on since a lot of good things are not discoverable :)
<garbas> oh it is not. sorry i must have missed it
<tilpner> Some people learn by examples, and my Hound instance allows them to quickly search through many user configurations at once
<garbas> so it shows nix code?
<tilpner> Well, it shows anything from the indexed repos, but that's just because it was easiest for me
<garbas> can you configure it to search among packages? and options?
<tilpner> But NixOS users have to write Nix too, so it's not just for developers
<tilpner> No, Hound is a text search tool
<tilpner> But I'm not suggesting we adopt Hound, I'm asking if we can expand the scope of your hypothetical to also search user configurations and repositories of the nixos organisation
<tilpner> Probably not at first, it would be enough to account for that possibility in the design
<garbas> oh yes, ofcourse. i never said no. just we need to think how to present this into a single page
<tilpner> I don't mind multiple search pages, if there's a header to navigate between them
<garbas> tilpner: once we have something around ourpart of the search would it be ok to ping you and we can bring also hound into it? this will probably in a month or so
<tilpner> Sure. I want the functionality, not certain we want to keep hound. I have no idea what elasticsearch can do, and if it could be made to do what we're currently using hound for
teto has quit [Ping timeout: 246 seconds]
teto has joined #nixos-infra
<gchristensen> apparently postgresql on zfs (haumea) should have wal_init_zero = off, wal_recile = off
<gchristensen> apparently postgresql on zfs (haumea) should have wal_init_zero = off, wal_recycle = off
<gchristensen> I swear I'm an adult
tilpner has quit [*.net *.split]
tilpner has joined #nixos-infra
<samueldr> still checking everything works
<samueldr> I got hung up on segfaulting generate-programs-index until I realised it was because of lack of memory
<garbas> samueldr++
<niksnut> samueldr: note that I've added and to nixpkgs as hydra jobs
<niksnut> so the channel mirror script doesn't have to generate them
<samueldr> oh
<samueldr> that'll remove some of the gunk
<samueldr> niksnut: haven't looked yet, will later, has it also been added to current stable?
<niksnut> I didn't backport them yet
<niksnut> but it should be pretty safe to do
<niksnut> btw we'll need to set a Content-Encoding on the brotli objects in S3
<niksnut> to decompress them automatically in the browser
<samueldr> yeah, that's something I was juste thinking that I left off the PR
<garbas> also a channels.n.o redirect needs to be created for packages.json and options.json
<garbas> and for .br files as well
<garbas> repology and some others are using it.
<niksnut> we don't need .json *and*
<garbas> i think adding it at around line 320 would be the way to do it
<niksnut> should be enough
<garbas> +1
<niksnut> okay I've cherry-picked packages/options.json to 19.09 and 20.03
<garbas> niksnut: do you maybe know where sync script lives?
<niksnut> no
<garbas> ok i'll dig around to see if they pass the correct headers to use our brodli
<garbas> s/our //
<garbas> i wonder if we should also add a redirect in netlify.toml for or just tell people to use channels.n.o