<ottidmes>
Every time I want to have some functionality on Android or Windows, I always remember how rare a thing Linux distributions are... no ads with every package, no remembering to uncheck certain checkbox to prevent unwanted software
zybell_ has quit [Ping timeout: 264 seconds]
zybell_ has joined #nixos-chat
<infinisil>
gchristensen: Now we almost need to write Nix bindings too.. Even bash can do it!
<gchristensen>
jeeze
<zybell_>
infinisil:bash can load extensions. to the language.
<infinisil>
Are you saying we need a dynamic grammar for Nix??
<gchristensen>
MichaelRaskin: what sort of things does your spider do?
<MichaelRaskin>
Well, there is this mistaken concept that regularly looking at stuff in a browser is a good UX
<MichaelRaskin>
My spider grabs some interesting Twitter feeds, some livejournal/dreamwidth feeds (mjg59, for example), lobsters, slatestarcodex…
<MichaelRaskin>
Converts stuff to text files in a uniform way.
<MichaelRaskin>
Then I can launch Vim and go through the feed
<MichaelRaskin>
Of course, when I remove files I actually remove the symlinks on QueryFS, so the stuff just gets marged as read in the database.
<gchristensen>
of course, why wouldn't you do it any other way? :P
<MichaelRaskin>
Mozilla's Raindrop was a nice inspiration. Too bad they gave up approximately immediately.
<gchristensen>
I've been wanting a way to consume a few choice Facebook accounts, but without all the rest of it.
<MichaelRaskin>
I wonder of there should be a project that just takes every way the scripts can access HTML/CSS layout/rendering result and publish a way how it can be used for tracking. Repeat until Mozilla explodes and makes layout/rendering one-way street in Private Browsing mode. Then we could at least hope for an accessibility-focused/content-only browser engine that could still access Javascript-only content…
<MichaelRaskin>
gchristensen: I think without account one can follow a link to a Facebook post, but not get a list of recent posts by account.
<ekleog>
like, weboob?
<ekleog>
(about the script set that converts stuff to text files)
<MichaelRaskin>
Well, mine is generic
<gchristensen>
I may need to have an account to do it properly, unfortunately.
<gchristensen>
MichaelRaskin: how does yours work generically?
<MichaelRaskin>
My approach is OK with JS-only stuff and could be extended to logins, actually
<MichaelRaskin>
I launch a Firefox with Marionette
<gchristensen>
oh this sounds robust
<MichaelRaskin>
Ask it to load whatever, inject a script that Makes Hidden Things Evident (a.k.a appends link targets to the link text, more or less), ask for innerText
<MichaelRaskin>
Yeah, it requires determination to break this workflow
<ekleog>
did you ever think of releasing it? also, does it work properly headless? that sounds like something great for eg. generating RSS feeds from anything (which likely would be my use of such a setup)
<MichaelRaskin>
Well, it actually runs in xdummy
<MichaelRaskin>
The problem is that currently it runs in full-paranoia mode, i.e. fresh UID per browser instance
<ekleog>
hmm… I guess it could use userns for this? (assuming the host kernel doesn't have them disabled, but iirc some stuff in nix requires them anyway nowadays?)
<MichaelRaskin>
I might at some point continue putting together a nerfed version of that that only uses nsjail
<MichaelRaskin>
We-ell
<MichaelRaskin>
I use separate UID, then a mount namespace, then a network namespace
<MichaelRaskin>
(ah, I think there is a couple of UID-namespace switches in the process, and it ends up being unprivileged UID inside a namespace that maps to unprivileged UIDs only anyway)
<ekleog>
oh
<MichaelRaskin>
I really really need the network namespace — Marionette insists on running on a TCP port, and I don't want to juggle TCP ports anymore, so I use a netns to convert TCP port into a unix domain socket with socat
<gchristensen>
MichaelRaskin: so ... with lang-os, where should I start looking.
<ekleog>
^
<gchristensen>
is there a logical entrypoint to start?
<MichaelRaskin>
Well, there is Firefox wrapper generator
<MichaelRaskin>
You could just use that inside a single-use UID and have this netns-isolated Firefox with Marionette control
<MichaelRaskin>
Obviously, you need to pass through some proxy access
<MichaelRaskin>
Ouch
<MichaelRaskin>
No, I think this wrapper launchers Firefox with Marionette and socat, but expects external code to set up netns
<MichaelRaskin>
Which lives in lang-os/lisp-os-helpers/subuser.lisp
<MichaelRaskin>
I have multiple system-lisp start scripts, maybe it is time to add one that doesn
<MichaelRaskin>
doesn't do anything like trying to grab tty's…
<ekleog>
hmm I guess I'll come back to this some other day :° sounds really nice but too complex for me to grasp past midnight :)
<MichaelRaskin>
(Obviously, when this daemon _is_ managing the system, I always want it to spawn basic stuff)
<MichaelRaskin>
The problem is that all this is currently written and used for the case of PID1=sinit, the system is controlled by a Common Lisp daemon that spawns all the services and reacts to requests
<MichaelRaskin>
There is no example where it isn't responsible for grabbing TTYs, for example.
<gchristensen>
does sinit do anything beyond starting your clisp daemon?
<MichaelRaskin>
sbcl, not clisp!
<MichaelRaskin>
It reaps orphans, and it spawns the shutdown script.
<gchristensen>
gotcha, but otherwise no sysv stuff
<MichaelRaskin>
Nope
<MichaelRaskin>
Should probably eventually start using some non-PID1 daemon supervision tool. Maybe runit
<ekleog>
hmm, why do you need a proxy on :3128?
<MichaelRaskin>
Because Firefox is netns-isolated?
<ekleog>
oh, if you want to isolate it securely, indeed
<ekleog>
I'd have thought just adding a veth and bridging would do it
<ekleog>
but that's not the same aim
<MichaelRaskin>
Also if I want to run Marionette on :2828 without cross-instance conflicts
<ekleog>
veth pair*
<MichaelRaskin>
veth is actually more complicated and root-only
<ekleog>
hmm starting a netns is root-only anyway, isn't it?
<MichaelRaskin>
Squid also allows me to ban a few CONNECT targets
<MichaelRaskin>
Nope
<ekleog>
oh? always thought it required root rights to setup a netns
<MichaelRaskin>
The only thing that works in Marionette-grab is starting something in a netns even without root
<ekleog>
hmm `ip netns add test` -> Permission denied, here (on `mkdir /var/run/netns`)
<MichaelRaskin>
Well, maybe you need a UID-ns root for netns
<MichaelRaskin>
But not real root
<MichaelRaskin>
And for eth outside of netns you actually need real root
<ekleog>
oh, and then veth is necessarily real-root, as it's outside of the netns, indeed
<MichaelRaskin>
Yep
<MichaelRaskin>
Given that Nix daemon creates a PID NS, I wonder how feasible it is to run systemd system instance inside a Nix build.
<ekleog>
hmm… I guess you'd want it for tests-PR v2?
<gchristensen>
I don't think you can, MichaelRaskin
<MichaelRaskin>
Well, it doesn't really solve the problem that NixOS is a ton of overhead for testing youtube-dl, but it could cut the overhead in half or something
<ekleog>
(if so, maybe runsvdir would do it? I'd guess it's simpler to use for tests than systemd, also, as writing a sv service is really simple)
<MichaelRaskin>
I don't actually need an init system
<gchristensen>
youtubedl is probably a bad test case for a testing infra, since it is fairly low value
<MichaelRaskin>
I need something that can use enough of NixOS service system to render Eelco's objection non-applicable
<gchristensen>
aye
<MichaelRaskin>
I mean, just launching stuff with a bash script is multiple times faster than using NixOS tests
<ekleog>
NixOS service system is way too much entangled with systemd :(
<MichaelRaskin>
And yes, my scripted test is fast and simple enough to use it for low-value youtube-dl
<MichaelRaskin>
Actually, it is even not: you can extract a runner script out of most stuff
<MichaelRaskin>
But even if I just extract the neccessary services and run s6 over all that, Eelco will probably still complain
<ekleog>
well, his complaint was about reimplementing all service management, and things like Xdummy aren't packaged as nixos modules anyway, so there wouldn't be any duplication?
<MichaelRaskin>
gchristensen: re: youtube-dl being low value: the point is that it is low-enough value to just auto-merge the automated updates to it with _only_ automated sanity check
<MichaelRaskin>
ekleog: the problem is: there is no good way out of his objection
<gchristensen>
ah
<MichaelRaskin>
Either I use something actually working (shell scripts, runit, sv) and then it is duplication
<ekleog>
well, I think next time I get an upgrade to one of the packages I'm maintaining I'll start to first trigger a discussion about just adding a meta.tests = listOf derivation; attribute, should be a good first step
<ekleog>
and there's no real objection against that in Eelco's arguments, afaicr
<MichaelRaskin>
Or I use NixOS services with systemd which means VM which means full boot which means overhead a few times over the actual test
<MichaelRaskin>
Well, yeah, for that you might want to coordinate with Profpatsch.
<ekleog>
oh yeah I heard about his talk at nixcon, do you know of a link?
<MichaelRaskin>
ekleog: you can declare the list of tests, there is just no sane way to implement tests
<ekleog>
the thing is it doesn't need to be sane
<ekleog>
the idea is that then we'll notice some programs duplicate meta.tests items, and will build up a top-level tests. attrset or whatever
<ekleog>
or we'll notice we want to link to nixos tests and same result
<MichaelRaskin>
I mean, there are NixOS tests. I recently added a _trivial_ check into the Chromium test. This work takes _forever_
<gchristensen>
oh yeah that is miserablee
<ekleog>
and then we'll be in a good position to think about only the optimization of nixos tests, without having to think about also adding all the things that are “free”
<MichaelRaskin>
I don't want just to tag Eelco _there_ and ask whether we can get some test approach that is actually usable
<MichaelRaskin>
You cannot actually optimise NixOS tests
<MichaelRaskin>
Without wither adding an actual init system (which Eelco opposes)
<ekleog>
and hopefully the change will come less as dumping a whole lot of new stuff, only as an improved way of running tests, because all the other new stuff will already have been taken care of :)
<ekleog>
I'm not sure he'd oppose it so blatantly if it didn't come with all the other changes and wasn't hand-rolled (then I only followed the PR, don't know whether you got any feedback from any other source)
<MichaelRaskin>
Also. NixOS tests are dependent on Perl.
<MichaelRaskin>
My PR had _no_ changes.
<MichaelRaskin>
It had only additions.
<ekleog>
I mean, even I would be reluctant to do service management in bash
<ekleog>
well, conceptual changes?
<ekleog>
all I'm saying is I think if we split this patch in small easy-to-digest steps, it could come up more easily :)
ma27 has joined #nixos-chat
<MichaelRaskin>
I mean, LibreOffice test needed everything at once anyway
<MichaelRaskin>
Also, NixOS tests use OCR in a suboptimal way, but debugging Perl code where a single run takes multiple minutes is just not going to happen.
<MichaelRaskin>
I just want some package update to break multiple NixOS tests.
<MichaelRaskin>
Ah wait, nixos-unstable is already stuck because of the docker image, and noone cares anyway
<ekleog>
Well, I was thinking of like dkimproxy. If it ever receives an upgrade, I guess I could add in some derivation that spawns it and pipes smtp data in/checks it receives the right stuff out. And that'd be a first meta.tests argument. And then once we got this running go forward with more complex tests like LibreOffice :) (and even dkimproxy might be a too-big test, something like a CLI tool would likely be
<ekleog>
an even better candidate for a first test)
<ekleog>
basically making hcanges as small as possible and incremental
<MichaelRaskin>
An easily testable CLI tool is unfortunately likely to have its own tests.
<MichaelRaskin>
I am still not completely convinced whether something crashing and burning is a neccessary condition of progress
<ekleog>
well, even if it has its own tests, it can't hurt to add in what the maintainer does to check the update
<MichaelRaskin>
Well, maybe.
<MichaelRaskin>
To be honest, for SBCL I could just add maxima as a test (and lispPackages.iolib)