<gchristensen>
I have this pattern of building which I replicate across a few projects, where I instantiate locally, build remotely, and then copy to a third location.
<gchristensen>
in a few projects' cases it involves local instantition, aarch64 and x86 builders, and then a single third machine which hosts them. it would be pretty nice to have a little tool to make this easy for me. for example right now it is a major slow-down since I instantiahte all of arm, copy, build all of arm... instantiate all of x86, copy, build all of x86. it could be much faster
<gchristensen>
I guess really what I want is to have the third machine use the aarch64 and x86 machines as remote builders
<gchristensen>
sounds like a heck of a yak to shave
drakonis has quit [Quit: WeeChat 2.3]
drakonis has joined #nixos-chat
Synthetica has quit [Quit: Connection closed for inactivity]
drakonis has quit [Quit: WeeChat 2.3]
<gchristensen>
solved it
drakonis has joined #nixos-chat
<ottidmes>
gchristensen: you managed to shave the yak? :P
<gchristensen>
yeah
<ottidmes>
gratz!
<gchristensen>
I'll share my nightmare solution
<ottidmes>
it is that I dont have any machines with different archs, but sounds interesting
pie___ has joined #nixos-chat
<gchristensen>
I gave up thath delightful simplicity what feels like a long time ago
pie___ has quit [Remote host closed the connection]
<ottidmes>
I truly hate this application I am trying to fix, everytime I think, yay, maybe now I can apply logic again to solving my issue I get stuff I do not get. Now I have the issue that /home/admin/proj/name does not work, all owned by the user and group admin with the right permissions, but if I place it in /var/www/name, it just works... the filesystem is just all under / with ext4, no fancy stuff going on,
<ottidmes>
no symlinks, I am at a loss, so I guess I am just moving it to /var/www and hope I dont run into problems until I can replace the application
<gchristensen>
...hrm
<gchristensen>
if you become the user / group, can you `cd` to that dir and `ls`?
<gchristensen>
a parent directory might not be +x for that user, andthus not allow the user to bein the subdir
<gchristensen>
in particular, this seems likely if the application doesn't run as `admin`
<ottidmes>
gchristensen: I am logged in as that user, and nginx and php-fpm are all running as it as well
<ottidmes>
configured and checked with ps aux
<gchristensen>
weird :/
<ottidmes>
yep :P this application is somehow capable of creating weird stuff like this I never run into
<gchristensen>
you could strace it to see whats up
<ottidmes>
so I am moving it to /var/www test it, and hope I dont have to touch it if I can help it
<ottidmes>
but you gist seems cool, so you now create those builders on the fly?
<gchristensen>
yea
<gchristensen>
it updates a config file, and then I can run ./build.sh which creates iPXE netboot images for a dozen types of servers
<gchristensen>
and then I can run ./test.sh which boots a server per type, and watches it do a NixOS install and verifies it behaves properly
<ottidmes>
so the goal is to test a package on different archs as once?
<gchristensen>
the cycle times are hell -- 30 minutes to verify a change, but way better than the manual way.
<ottidmes>
yeah, humans tend to be bad at things you could automate with a script
<gchristensen>
I build/maintin Packet.net's tools for when customers request a server with NixOS as the OS, and I'm building netboot images that boot, erase the disk, and install NixOS
<ottidmes>
ah right, so this script automates building the things you need to allow you to provide others with NixOS by prebuilding e.g. the necessary iPXE images on the possible archs?
<samueldr>
(and relatively obvious since sqlite added a test case apparently)
<gchristensen>
wow
<ottidmes>
BTW, I looked into, set -euo pipefail, and I would never use it myself, it makes something that you already have to be careful about, have even more unexpected behaviors
<samueldr>
huh?
<samueldr>
-o pipefail I could see how one would avoid it, but -eu is... pretty dependable
<samueldr>
imho most of the reasons to avoid it are ways you probably shouldn't code in bash :/
<gchristensen>
truth
<samueldr>
the moment you're going into arithmetics or convoluted conditionals, maybe get to a real programming language?
<samueldr>
something like perl or php
* samueldr
sets trap
<samueldr>
;)
<ottidmes>
maybe, but still, its unnecessary and it messes with your intuition and is inconsistent, no thank you
<gchristensen>
I completely agree
<ottidmes>
yes! I got gchristensen to agree :P
<ldlework>
bash is such an insane language
<samueldr>
bash is already inconsistent :/
<ottidmes>
so why make it more so?
<samueldr>
more so?
<ottidmes>
why make it worse than it already its
<ottidmes>
* is
<samueldr>
the reason set -e will exist is because bash is inconsistent :)
<emily>
even that very page has
<emily>
rking's personal recommendation is to go ahead and use set -e, but beware of possible gotchas. It has useful semantics, so to exclude it from the toolbox is to give into FUD.
<emily>
on it
<samueldr>
it won't make those inconsistencies go away, you would maybe need to know about these anyways
<emily>
shell never behaves like you expect. at least with "set -e" its behaviour tends towards the more robust/reasonable
<samueldr>
good thing I had nothing weird in my clipboard
<emily>
the exercises are more like "so you think bash is OK, huh?"
<samueldr>
looks like I raced with my screenshot util
<samueldr>
emily: pretty much :)
<emily>
joke's on you, BashFAQ. i have no illusions about your cursed language
<samueldr>
`set-e` isn't about making bash better, but your glued together bits and bobs of commands you think can't fail fail early
<ottidmes>
I will stick with the default semantics, I am used to their gotchas
<gchristensen>
this is why we need XMshelL
<ottidmes>
cause turning everything in XML makes it so much better XD
<gchristensen>
:D
<ottidmes>
I have to admit, I was loving the whole XHTML era, to bad that sentiment went away mostly after HTML5
lassulus_ has joined #nixos-chat
lassulus has quit [Ping timeout: 250 seconds]
lassulus_ is now known as lassulus
<ottidmes>
samueldr: I am curious though, what "real" programming language do you use when you are not just gluing things?
<samueldr>
when I was saying "real" I was saying mostly one where you are confident in its general programming purposes
<samueldr>
whatever I use isn't relevant
<samueldr>
but the quick and dirty goes through ruby
<ottidmes>
samueldr: I of course knew what you meant about not using shell to program in, so it is relevant, because it was my whole point of the question, but Ruby seems like a good choice
<samueldr>
just making sure it was clear I wasn't prescribing anything :)
<ottidmes>
I tend to pick whatever is the easiest, if there is some good Python library that does most of what I need, I will pick Python, even though I dont really like the language
pie__ has joined #nixos-chat
pie___ has quit [Remote host closed the connection]
hedning has joined #nixos-chat
ottidmes has quit [Quit: WeeChat 2.2]
<infinisil>
Haskell for me o/
sir_guy_carleton has quit [Quit: WeeChat 2.2]
hedning has left #nixos-chat [#nixos-chat]
etu has quit [Quit: WeeChat 2.3]
etu has joined #nixos-chat
hedning has joined #nixos-chat
nckx has quit [Quit: Updating my GNU GuixSD server — gnu.org/s/guix]
nckx has joined #nixos-chat
__monty__ has joined #nixos-chat
__monty__ has quit [Quit: leaving]
drakonis_ has joined #nixos-chat
drakonis has quit [Ping timeout: 250 seconds]
<gchristensen>
sure wish jD91mZM2 came around more
<ghostinthenet>
Trying to get away from the docker networking model. Rkt is much simpler to play with. I’m certainly open to other options though.
<gchristensen>
I'm wondering if I can use jD..'s rnix library to emit Nix code from Rust, it'd be nice to not have typos anymore
<ldlework>
ghostinthenet: what about the networking models has your focus?
drakonis_ has quit [Read error: Connection reset by peer]
<ghostinthenet>
Trying to take NAT out of the picture mostly. It •can• be done with the docker model, but so far as I’ve found, that mostly means resorting to bridging. Adding the NAT layer just makes things harder to track.
<ghostinthenet>
Of course, I’m fairly new to things on that side, so I may be missing some of the options.
<ldlework>
by NAT do you mean the fact that containers have private IP's on the host?
<gchristensen>
in a point release they added some iptables commands which took down my prod network, once.
<ghostinthenet>
Idlework: not so much that as that the application’s internal addressing doesn’t match how it’s referenced externally. private addressing is essential, but if the application thinks it’s at 10.0.0.1, I don’t want to be accessing it at 172.24.3.5.
<ghostinthenet>
Sort of. I still want the load balancing option though. :)
<ldlework>
Is it http? Can you reverse proxy?
<ghostinthenet>
Not necessarily http, no… though much of it might be.
<ghostinthenet>
All of that said, I haven’t actually looked at what docker networking can do in a couple of years. I may have to have another look, though I still think the container engine really doesn’t need to be bound to the networking engine.
<ldlework>
when i worked there, listening to the people who worked directly on the daemon, especially the socket plane people who worked on the networking, i got the impression they were much smarter than me
<ldlework>
those unikernel people too, jeeze
<gchristensen>
I work with some of those unikernel people now. must say: jeeze
<ghostinthenet>
Unikernels are pretty hardcore.
<ghostinthenet>
I know this is strange coming from a networking guy, but I like my connectivity to be as simple as possible. Overlays complicate things. Sometimes that’s necessary, but standardizing on it is another thing.
<joepie91>
ghostinthenet: I'd imagine it's similar to programming; once you're good enough at it, you realize that complexity is your enemy :)
<ghostinthenet>
Exactly. Not to be feared, but let’s not live for it either. Job security through complexity is no better than security through obscurity.
<simpson>
ghostinthenet: So, looking just at the k8s objects, a Pod (a group of containers) has some set of ports routed to a private IP inside the cluster. A Service maps a port on a public IP to a port on a private IP. So this public-private disconnect happens with very basic k8s objects.
<simpson>
This is useful because a Deployment or ReplicaSet can have many copies of a Pod underneath it all running independently, and then a Service can be mapped to the Deployment/ReplicaSet instead of to a single Pod.
<simpson>
We *do* sometimes need to tell application servers about the public IP, rather than letting them automatically guess, since their guess is going to be wrong.
<ghostinthenet>
simpson: True, but at least it’s isolated to the pod. (I can feel my network purism slowly dying as I’m faced with the grim reality of things.)
<simpson>
Yes, Pod isolation is good.
<simpson>
To give a practical example, I have some k8s Ingress objects. Ingress is basically a vhost router on top of Services. For one public IP, I have a vhost with a half-dozen domains mapped to it, and traffic goes first through a TLS terminator and then to the Service for that domain, which has several Pods behind it.
<simpson>
Part of this mapping isn't even IP-based, but vhost-based.
<ldlework>
If we're being real, I'm a Django programmer. And with Docker I have been able to architect and deploy fairly sophisticated distributed systems so I think there's something to it. It always takes code (and complexity) to implement the abstractions which simplify things somewhere else. There's no hope for me if I had to solve the problems that container engines are doing for me.
<ldlework>
Someday I'll look into the k8s abyss. That k8s is an abstraction and an abyss tells you something about how -intrinsically- complex the problem of distributed services really is.
<simpson>
I do wonder how much more complex than containers it can be.
<simpson>
To me, k8s's object model is simpler than Docker's networking model.
<ldlework>
Well I think for people like me with lots of container experience there's hope.
<ldlework>
But I've seen a whole company stare into the k8s abyss and go "Nahhhhh"
<ldlework>
The idea that ingress and all of that is simpler than "docker network" command seems like crazytown to me but, you know
<simpson>
Ingress is only really needed when doing HTTP. Another example: I have Tahoe-LAFS storage servers. One single Python daemon. To manage that, I have a Deployment, a Service for talking to the outside world, a PersistentVolumeClaim for persistent disk, and also a ServiceAccount and ClusterRoleBinding for portably getting the Service's LoadBalancer's external IP. Messy kind of capability there, really, but
<simpson>
it works.
<simpson>
3 objects for the essentials plus 2 objects because the k8s API isn't perfect yet.
<simpson>
The tradeoff is that this is sufficient abstraction to move to any k8s provider. I've been turning up new storage servers in under 1hr, including new account and cluster creation.
<ldlework>
well i respect the overall problem so i don't diss for k8s for not being perfect
<ldlework>
don't worry soon enough we'll all be so edgy and dispassionated we'll have some similar epitaph towards complex orchestration solutions as we do with "its just someone else's computer" about the cloud even though we've had mainframes forever but not totalistic automation of everything involved :)
<simpson>
I just think that if folks want to look at what portability looks like, then a good insight is that k8s *is* a portable API developed by many cloud professionals with the hopes that it would be implementable anywhere.
<simpson>
e.g. the StorageClass object type is not really something I can create. They're provided by the cloud provider. But I can query them. https://bpaste.net/show/f459689cc044
<ldlework>
"it just puts containers on servers"
<simpson>
It's like a relatively small portable API for putting containers on servers, yes. It's designed to be extended with custom object types, rather than carrying an object type for every single provider-specific bit of functionality.
<simpson>
In return, providers have actually not been doing custom types. So instead custom types are where people are putting more complex concepts.
<ldlework>
simpson: my own orchestrator was based on an agnostic driver system
<ldlework>
we only ever implemented an AWS backend but we were totally ready to throw down with Azure if Windows containers came around
<ldlework>
(they have since of course)
<simpson>
e.g. using kubeless (https://github.com/kubeless/kubeless) one gets new kinds of objects, like CronJobTriggers, HTTPTriggers, and Functions.
<simpson>
ldlework: That's how a lot of tools end up. kops is still effectively AWS-only, because the community has the direction of un-AWS'ifying it.
<ldlework>
I didn't have arbitrary objects, we tried to fit a common denomenator over them. But hey, it's me and it was very early.. no orchestrators around except whispers of swarm.
<simpson>
Whereas kubicorn has had a good spread of support across many vendors because they didn't build on one cloud.
ghostinthenet has quit [Quit: ghostinthenet]
<ldlework>
btw, the person who wrote the automat docs has never actually brewed coffee
<gchristensen>
lol, yeah?
<ldlework>
well maybe they have an amazing coffee machine and I'm an impoverished pleb
<ldlework>
but generally you put grinds not beans into a coffee machine
<ldlework>
but realized after I said that, that maybe some of them can grind beans
<gchristensen>
heh
<ldlework>
oh crap, this isn't the channel we were talking about automat in
<gchristensen>
colemickens: you here?
<colemickens>
gchristensen: yup
<gchristensen>
I have questions about usability and the packet.net nixos installer (cc clever, disasm, too)
<gchristensen>
oenuthoenuthoenuth it is complicated. I want to support ZFS
avn has quit [Remote host closed the connection]
<gchristensen>
so my thought was to have a "secret" (documented) escape hatch in the userdata which says hey swap over to that cool ZFS mode ... but really I shouldn't, because Packet already supports CPR https://support.packet.com/kb/articles/custom-partitioning-raid -- but this doesn't support ZFS and only works on reserved hardware.
endformationage has joined #nixos-chat
rawreraw has joined #nixos-chat
rawreraw has quit [Client Quit]
rawreraw has joined #nixos-chat
rawreraw has quit [Quit: WeeChat 2.3]
<elvishjerricco>
Got the GPU and audio devices using the VFIO driver pretty trivially. The USB controllers aren't playing ball though :/
<samueldr>
nice, here I tried, but I *think* that my gpu isn't playing nice because of its age
<samueldr>
a gtx 570, the VM sees it, but it apparently can't initialize it
ghostinthenet has joined #nixos-chat
<samueldr>
I had trouble finding good information, but even by dumping its rom and passing it, nothing works
<samueldr>
(on a linux guest)
<samueldr>
though it's listed in lspci
<samueldr>
I'm thinking it might be cause of its age since apparently every cards listed as working properly have a UEFI option rom, while AFAIUI mine doesn't
<elvishjerricco>
samueldr: Have you made sure you're using the real rom, not a shadow rom?
<elvishjerricco>
When the EFI initializes your GPU for the boot screen, for some reason it makes it so that dumping the rom dumps some "shadow rom" instead, not the real rom.
<elvishjerricco>
This doesn't happen if the gpu in question is the one used for the boot screen
<elvishjerricco>
s/is/isn't'/
<samueldr>
dumped while not being in use for display purposes
<elvishjerricco>
samueldr: Must also have never been used for display purposes throughout the current boot, even if it no longer is in use at the time of dumping.
<samueldr>
and not used at all*
<samueldr>
I have anohter nvidia gpu that was in use
<elvishjerricco>
Then I dunno :P But initialization problems in the guest does sound like rom problems
<elvishjerricco>
(to my novice ear, anyway)
<samueldr>
(though now I'm thinking that I never checked if I can do it using the 630, mostly to see if it works there)
<elvishjerricco>
Anyway, I can't seem to find a way to prevent `xhci_pci` from loading. Blacklisting it doesn't work, and that's the driver which steals my USB controllers before vfio can get to them
<samueldr>
main issue with my searches is not finding anything below 7xx series of GTX gpus, so it's possible the whole family the 570 is part of is not well documented
<elvishjerricco>
why would blacklisting a kernel module not result it in not being loaded, even when it has no dependents?
<samueldr>
could it be in another list, which gets loaded? I'm thinking initrd early init or something
<samueldr>
you know, so the keyboard works when inputting an passphrase
<elvishjerricco>
samueldr: You don't think `boot.blacklistedKernelModules` would prevent that in initrd?
<samueldr>
I may be wrong, not looked up documentation, but if memory serves blacklisting only stops the automatic loading
* samueldr
lookups doc
<elvishjerricco>
Oh. So you're saying someone is asking for it explicitly?
<samueldr>
that's my hunch
<elvishjerricco>
Ah, the `hardware-configuration.nix` has `boot.initrd.availableKernelModules = ["xhci_pci" ...];`
<elvishjerricco>
Would that do it?
<samueldr>
that's what my hunch is based on, mainly
<samueldr>
maybe a bit simpler: boot.blacklistedKernelModules is not referenced (AFAICT) in stage-1
<samueldr>
and availableKernelModules makes all those listed there available in stage-1
<elvishjerricco>
samueldr: Wouldn't the blacklist just be something that the kernel looks for itself?
<samueldr>
that's what I'm looking for
<samueldr>
"how does the blacklist works?"
<samueldr>
though there's one obvious thing: if the blacklist isn't available in stage-1, it can't know it shouldn't load xhci_pci
<elvishjerricco>
Well, I think you're right that the blacklist is only for preventing the kernel from doing stuff automatically. So if we just never ask for xhci_pci in stage 1, then it should be fine
<samueldr>
>> The blacklist <module_name> command, however, does not prevent the module from being loaded manually, or from being loaded as a dependency for another kernel module that is not blacklisted.