Synthetica has quit [Quit: Connection closed for inactivity]
drakonis_ has joined #nixos-dev
drakonis has joined #nixos-dev
pie__ has joined #nixos-dev
pie__ has quit [Remote host closed the connection]
pie__ has joined #nixos-dev
delroth has quit [Quit: WeeChat 2.4]
drakonis has quit [Quit: WeeChat 2.5]
orivej has quit [Ping timeout: 245 seconds]
drakonis_ has quit [Ping timeout: 248 seconds]
drakonis_ has joined #nixos-dev
delroth has joined #nixos-dev
pie__ has quit [Ping timeout: 258 seconds]
pie__ has joined #nixos-dev
Jackneill has joined #nixos-dev
orivej has joined #nixos-dev
Jackneill has quit [Remote host closed the connection]
Jackneill has joined #nixos-dev
Jackneill has quit [Remote host closed the connection]
Jackneill has joined #nixos-dev
Jackneill has quit [Remote host closed the connection]
Jackneill has joined #nixos-dev
Jackneill has quit [Remote host closed the connection]
ciil has joined #nixos-dev
Jackneill has joined #nixos-dev
Synthetica has joined #nixos-dev
<layus>
globin: My work on nested build systems has stalled. I had a lot of work as TA at university, and focussed on building firefox with tup. Next step is to build FF incrementally, possibly with an implementation of https://github.com/NixOS/rfcs/pull/40
<layus>
globin, I will be available during summer to work on it.
<FRidh>
gchristensen: likely you're env already contains PYTHONHOME (or PYTHONPATH), causing breakage. This is typically the case when say a python 2 program calls a python 3 program in a subprocess. If the Python 2 program has a wrapped env where we set PYTHONHOME, then the python 3 program inherits that, unless it sets/unsets it itself
init_6 has joined #nixos-dev
Synthetica has quit []
Synthetica has joined #nixos-dev
averell has quit [Ping timeout: 272 seconds]
<ekleog>
FRidh: Hey! Wanted to run an idea by you, about the python infrastructure: what would you think about (ab)using sys.meta_path, to bring `RPATH` to python (by having an element of sys.meta_path read the nix-support/python-deps file), which would allow us to bring static linking and get rid of both the need for `propagatedBuildInputs` and `python.withPackages`?
<niksnut>
sounds great
drakonis has joined #nixos-dev
drakonis_ has quit [Ping timeout: 268 seconds]
drakonis has quit [Read error: Connection reset by peer]
drakonis has joined #nixos-dev
drakonis_ has joined #nixos-dev
drakonis has quit [Read error: Connection reset by peer]
drakonis has joined #nixos-dev
<Profpatsch>
ekleog: What does static linking look like for python packages?
drakonis_ has quit [Ping timeout: 252 seconds]
<ekleog>
Profpatsch: it'd be something like putting the package name -> store path in a nix-support/python-deps file, and having meta_path refer to this file at each import
<ekleog>
right now I'm trying to do a quick draft of the idea, and the biggest issue I'm facing is python doesn't apparently provide the path of the importing file to the meta_path element, which means that it'd have to rely on extracting the traceback and “parsing” it, which sounds a tad break-worthy -- but it should break only on python upgrades, and if no python dependency works we should quickly notice it
orivej has quit [Ping timeout: 248 seconds]
<Profpatsch>
ekleog: Ah, so it’s more like dynamic ELF linking instead of implicitly constructung the PYTHONPATH with wrappers.
<Profpatsch>
You put the information in the code itself (backwards-references) instead of propagating the transitive dependency chain forward by appending PYTHONPATH
<ekleog>
ah right, it's dynamic linking when using compiled languages' terms (though compared to python's usual it's static -- to be precise, it's doing RPATH for python), sorry for the misnomer :)
<Profpatsch>
ekleog: I could imagine something like browserify for python, where all the code is put in one static file :)
<Profpatsch>
Though with python’s imperative approach to module loading and modules not being first class, I don’t think that would work very well in practice.
<ekleog>
whew yeah right ^^'
<pie__>
ekleog, why is .withPackages necessarily bad?
<pie__>
or, i guess this python-deps mechanism somehow bypasses the need for that?
<pie__>
ekleog, how does the traceback get the informatio
<ekleog>
pie__: .withPackages is a PITA when you need to handle multiple pythons because two dependencies conflict, as for the traceback it's all in `traceback.extract_stack`
<gchristensen>
hrm. "warning: installing Nix as root is not supported by this script! performing a single-user installation of Nix..."
<pie__>
ekleog, hm...how would this fix that
<ekleog>
see above
<pie__>
i guess i just dont understand the details well enough
<ekleog>
well, I'll be able to explain it in detail when it's actually done
<pie__>
is it a problem with the withpackages interface as well or just the implementation
<ekleog>
but for now I need to yell at python… it assumes that a package name uniquely identifies the package :(( which means that importing two packages under the same name with our custom meta_path doesn't actually import the second one because it's thinking it has dependency-looped
<pie__>
im asking because its probably relevant to my work on the interface
<ekleog>
pie__: it should remove completely the withPackage interface, if I manage to do what I want, making it unnecessary
<ekleog>
don't hold your breath though
<pie__>
ekleog, like what about if there was an r.withPackages and otherstuff.withpackages
<pie__>
is that something i should abandon altogether
<ekleog>
pie__: well I only have an idea for python, for the time being
<ekleog>
so other languages' withPackages will stay until someone has ideas for them too
<pie__>
ekleog, a lot of othe rlanguages have completely adhoc implementations which is why i was looking into makign a common withpackages interface
<pie__>
if the interface is bad i should be doing something else
<pie__>
ekleog, but how / why would you want to combine two different python versions?
<ekleog>
well, the interface is bad, but often better than nothing -- the only good interface I see is the C one
<ekleog>
it's not two python versions, but depending on packages A and B that both depend on package C in different versions
<pie__>
hm
<pie__>
is that safe?
drakonis has quit [Read error: Connection reset by peer]
<pie__>
well i guess thats not different than with C i guesss
drakonis has joined #nixos-dev
<FRidh>
Jjust for clarity, we have currently two approaches. 1) we inject a snipped that extends sys.path in every executable in every python package. This is so it's possible to directly run applications that are written in Python. 2) we have .withPackages which is for composing a set of packages. It works by symlinking packages, and then wraps the executables setting PYTHONHOME so all Python modules are found.
<ekleog>
grmbl sounds like sys.modules will make things hard, if applications rely on a single set of name -> module association
<FRidh>
Note that with 1) we *also* create wrappers. I think we should still get rid of those.
orivej has joined #nixos-dev
<FRidh>
1) works fine for its purpose. There are some cases where the sed substitute that is used for injecting the snippet fails though, and the wrapper causes trouble because ...environment variables. 2) is problematic, because again, environment variable.
<ekleog>
FRidh: right, ideally if I can figure out a way to use sys.meta_path (by patching python's lib/python3.7/importlib/_bootstrap.py I think) it should also allow us to get rid of the sys.path manual extension
<FRidh>
ekleog: we aren't really using PYTHONPATH. We use it in the wrappers that are currently created, but those aren't actually needed because we inject already code into the entry points.
<FRidh>
ekleog: have you checked sitecustomize.py ?
<{^_^}>
#25985 (by FRidh, 2 years ago, open): Python: add sitecustomize.py, listen to NIX_PYTHONPATH and NIX_PYTHON_SCRIPT_NAME
<FRidh>
it would be good if we can indeed read it from a file, because that way we can skip injecting code and skip creating wrappers
<ekleog>
FRidh: I was thinking on something completely different, actually
<ekleog>
of*
<ekleog>
that is, having our python code dynamically resolve imports based on the importer's declared dependencies
<FRidh>
in principle that can be done with any finder, though not using a path based one should speed it up
<FRidh>
yes, that I understood. You will somewhere have to add that finder to sys.meta_path, and have somewhere your set of paths
<ekleog>
that set of paths would be in nix-support/python-deps different for each library
<ekleog>
so each library would have its own dependencies, allowing mix-and-match versions like nix allows
<ekleog>
the hard thing being that python appears to want a single module namespace, with sys.modules -- I wonder how many applications or libraries actually do meaningful stuff with it, though
<FRidh>
that's indeed not supported by python
<ekleog>
right now I'm investigating module-level getters and setters to see whether I can override sys.modules and have it provide a different view to each file that accesses it
<ekleog>
but it sounds like module-level getters and setters require setting the module itself in sys.modules, which would be the snake biting its own tail
<ekleog>
(python doesn't by default support it, I'm hoping we/I can find a way to make our python support it without breaking applications)
<pie__>
"cant we jsut get python to support it?" :P
<FRidh>
That's an interesting experiment, but I am not very optimistic that will work. In any case, what I do think is achievable for Nixpkgs is using a file in nix-support/ and using a finder that loads it from that file.
<ekleog>
so still keeping a single module name -> module association?
<pie__>
from my point of view .withPackages as a user seemed like just an interface thing, what would the alternative be? just listing all the python packages in buildinputs?
<FRidh>
ekleog: yes
<ekleog>
hmm that doesn't solve my problem, unfortunately :/
<FRidh>
pie__: that's also still possible. It just gets problematic when you have python 2 and 3 and packages for each as they'll end up on one PYTHONPATH
<FRidh>
because, during build-time, PYTHONPATH is used by python's setupHook
<pie__>
FRidh, i wasnt worried about "still" because i didnt know how it worked to begin with. ive just experienced some problems with nix-shell
<pie__>
im just asking from a completely user perspective of what this is suppsoed to look like in the ideal case
<FRidh>
Hydra issues with the Darwin builders.
<gchristensen>
what kind? I thought I got them all fixed up
<FRidh>
gchristensen: there were auth issues again. I just restarted all jobs, things seems better