<samueldr>
fails for a non-mainline qualcomm device
<samueldr>
and asus-dumo "fails", but really works
<samueldr>
except that an issue with the mainline driver for the display where it cannot suspend properly makes it so it's not useful
<samueldr>
annoyingly I got in touch with people that know more about it, and even ask for guidance about working around the issue and they stubbornly don't want to tell, rather they say it should be fixed the right way then explain things I'm sure make sense when you already know them
<samueldr>
:(
<samueldr>
what they don't want to tell is the dumb "just shut it all down and start it all back up" method which at least would allow me to use the thing
cole-h has joined #nixos-aarch64
<andi->
yeah, embedded :/
<andi->
I am trying my luck with modifying the device tree file now
<samueldr>
I have a good grasp of the high level issue
<samueldr>
it apparently works with "atomic" operations
<samueldr>
that's what chromeos was designed around
<samueldr>
but X11 and modesetting or legacy framebuffer (e.g. for the VT) won't
<samueldr>
once I have my gui toolkit working strictly with DRM, I'll disable legacy framebuffer and try it out then
<samueldr>
but it's still working around a mainline incomplete driver
<samueldr>
but since it's all happening in contorted logic around the quite vast drivers, I literally have no idea where to even start looking for a fix
<samueldr>
driver callbacks piled-up upon driver callbacks
<andi->
Oh well syntax error in my DT patch.. guess I should read up on the format
<samueldr>
can you also share the patch?
<samueldr>
though, the spect itself IIRC is good enough to get going
<samueldr>
but still, it lacks a glossary of terms IIRC
<samueldr>
yep, no quick glossary
<andi->
https://termbin.com/ohev I didn't even try to understand what I did there. Just went by what I saw in the file and improvised :D
<andi->
I read up that uart4 is on the same GPIOs and thus should be disabled..
<samueldr>
andi-: semicolons
<andi->
garr
<samueldr>
the dtc errors are not good
<andi->
FATAL ERROR: Unable to parse input tree
<andi->
what is not good about that?!?
<andi->
it at least didn't compile
<samueldr>
it's accurate
<samueldr>
imprecise
<samueldr>
btw, I often get caught by the same mistake
<andi->
well
<andi->
Lets see if it still boots with these changes and if that changes anything at all
<andi->
So, the other day you said the flattend dt (FDT) would be passed on by u-boot. So actually I shouldn't patch (just) the kernel but also uboot?
<samueldr>
nah, u-boot will load the dtb file you're building for the generation
<samueldr>
which by default is the one built with the kernel
<samueldr>
andi-: grep for jedec,spi-nor in rk339 prefixed .dts files I guess
<samueldr>
rk3399-rockpro64.dtsi and rk3399-roc-pc.dtsi are likely trustable on that topic
<samueldr>
(I looked at what's on the tip of torvald's master branch)
<andi->
Yeah I am looking at 5.10 right now which is probably close enough
<samueldr>
I would assume
<andi->
all of them seem to use spi1 instead of spi0 what I found on some random armbian trash forum
<samueldr>
armbian often use the vendor kernels as a variant
<samueldr>
it could matter here
neverredneverred has joined #nixos-aarch64
<andi->
ok, yeah I recall some of these patches in their repos from the lkml
<samueldr>
in my experience SBCs rarely stray too far from the reference design, which is helpful to get those things figured out
<neverredneverred>
Howdy folks! I'm pretty new to nix and I'm trying to generate an image for the pinephone. I don't have an arm machine to use, so I downloaded a pre-built image as suggested here https://mobile.nixos.org/getting-started.html. I also created the local.nix file. I'm lost at this point. The pinephone page says to run this command to build the image:
<neverredneverred>
"nix-build --argstr device pine64-pinephone -A build.disk-image". This reports the warning "trace: WARNING: evaluation includes ./local.nix", and it seems to attempt a cross-compiled build. How do I get nix to build the image from the rootfs I downloaded?
<neverredneverred>
I don't have NixOS installed, just home-manager on a mac.
<neverredneverred>
The actual error reported is "error: attribute 'selectBySystem' missing, at ~/tmp/nix_build_test/mobile-nixos/modules/system-target.nix:23:16"
<samueldr>
hi
<samueldr>
since you have a prebuilt image, you don't have to build anything
<samueldr>
you can `dd` the .img file to an sd image
<samueldr>
note that mobile nixos is not as ready as many other mobile distros
<samueldr>
neverredneverred: it's the device that is different, there are two pinephone variants, one with 2GB of ram, another with 3GB of ram
<samueldr>
(well, more than two pinephone variants, but for what matters here, two)
<neverredneverred>
ohhh. I have "UBports Edition"
<samueldr>
I don't think they had a 3GB variant at that time, not 100% sure
<samueldr>
anyway, the worst that would happen is that it wouldn't boot
<neverredneverred>
yea, I'll find out in a bit. writing the image right now.
<neverredneverred>
2.5 MB/s...
<neverredneverred>
do you use nix for many of your personal devices and computing environments?
<neverredneverred>
I've grown tired of my systems being throwaway one-time setups and easily broken by rolling release.
<samueldr>
all of my computers run on Nix, server too, and hopefully at some point during the next year my phone does too
<neverredneverred>
any regrets? or words of advice?
<andi->
do it.
<andi->
You'll never want to go back.
<andi->
After >3years of daily usage I am probably unable to use anything else..
<simpson>
I regret not doing it sooner. I could have survived nixpkgs being, say, 5yrs younger, and I would have been able to contribute more.
<samueldr>
same
<samueldr>
it's one of those things where even if you don't have the time, it is worth it in the end to make the time to get it
<andi->
I lost soo much time with my Gentoo before this.. and the Arch in the mean time..
<neverredneverred>
yea, I've been working on reducing undesired cognitive overhead of life, in a general sense, and my computing environment is now the biggest drag. So many powerful FOSS tools out there, but the maintenance burden is too damn high.
<neverredneverred>
everything having a place <-> declarative system configuration
<andi->
53min into the build.. I made another typo
<samueldr>
yay
<samueldr>
I know the feeling
<neverredneverred>
so it booted into ubuntu touch O.o. Is there a default passcode for the "pinephone" user?
<neverredneverred>
nvm, it must have booted from the eMMC
<samueldr>
I don't recall, going from memory, but I think you'll need to do things to get a working image without building it
<samueldr>
because we don't build a "full" image for any platforms on hydra
<samueldr>
(for the time being)
h0m1 has quit [Ping timeout: 264 seconds]
h0m1 has joined #nixos-aarch64
<neverredneverred>
maybe I should just find an arm machine in the ClOuD
<andi->
tell me once you found one that is affordable
<gchristensen>
AWS's aren't *SO* bad but no KVM unless you do $expensive/hr
<andi->
wasn't the AWS charge for the biggest VM the same as for the metal version?
<andi->
2.5$/h or so
<andi->
a1.metal 0.466/h
<andi->
a1.4xlarge 0.4656/h
<gchristensen>
yeah you need the .metal one to get kvm
<neverredneverred>
I have a raspberrypi2 but had a bad time trying to get nix on it
<andi->
personal advice: do not bother
<andi->
waste of time.
<neverredneverred>
the installer was trying to build everything. I let it run for 3 days
<andi->
haha
<gchristensen>
yeah totally a waste of time if you don't have a big ol' machine to do alllllll the lifting
<andi->
How many pots of coffee did you heat?
<neverredneverred>
its cold here, but the little pi didn't help much
<samueldr>
raspberry pi 2 is 32 bit ARM
<samueldr>
which is part of why you had a bad time
justanotheruser has quit [Ping timeout: 272 seconds]
<neverredneverred>
ngl, nix has not been nice to me. Its just broken right now on one of my systems. I can't install a derivation for a personal python package because it says some /nix/store is missing. Even after doing nix-collect-garbage and having no references to it in /nix, when I try to nix-shell into my derivation, it complains.
<neverredneverred>
when its basic with home-manager and I just specify packages that work already from nixpkgs, lifes great.
justanotheruser has joined #nixos-aarch64
<neverredneverred>
How long does it usually take for an EC2 instance to come up? 20 minutes and the instance still isn't responding to pings.
<neverredneverred>
ok now I'm getting somewhere. running a build...
justan0theruser has joined #nixos-aarch64
justanotheruser has quit [Ping timeout: 272 seconds]
<neverredneverred>
hmmm, getting an error that there is no space left, but df shows plenty of space left.cp: cannot create regular file 'source/README': No space left on devicecp: cannot create regular file 'source/README.md': No space left on devicedo not know how to unpack source archive /nix/store/zb2r1afdb87rsz4w9idk9n0zvf1byy6y-sourcebuilder for
<neverredneverred>
'/nix/store/4vhbyjb5sfy4ikk8cqpxf5p767xhnwgl-linux-5.9.0.drv' failed with exit code 1note: build failure may have been caused by lack of free disk spaceerror: build of '/nix/store/4vhbyjb5sfy4ikk8cqpxf5p767xhnwgl-linux-5.9.0.drv' failed
<clever>
neverredneverred: singleuser or multiuser nix?
lgcl has joined #nixos-aarch64
<neverredneverred>
singleuser
<patagonicus>
neverredneverred: if you trust a random person on IRC, I could give you an image I have running on my pinephone. Not very customized, but it does boot to Gnome desktop with an onscreen keyboard (but one without stuff like Ctrl, so limited usability). Having access via the serial port really helps.
<patagonicus>
Ah, thinking about it, I don't have that image yet, I just nixos-rebuild switch'ed from one without gnome. But I could build the image.
<neverredneverred>
nah, I want to be able to build images. reproducibility would put my mind at ease
<patagonicus>
Totally understand that. That's why I build everything myself on my armv7 machines. :D
<patagonicus>
I started of withe regular (non-demo) system image and then build an image based on the Getting Started guide with that + adding wpa_supplicant. Then I could bootstrap from there over the serial port (log in, set up wifi, switch to SSH, configure more stuff), but haven't gotten far.
<clever>
note to self, dont use arm32 strace on arm64 binaries
<clever>
it names every syscall the wrong way
<neverredneverred>
samueldr, I think you were right about tmpfs. It was ~350MB
<neverredneverred>
I bumped up the `/run/user/1000` tmpfs to 5G, and all of the other tmpfs have 1.7G, and I'm getting the same No space left on device error. I wonder if it's not enough RAM... theres only 2.3 GiB available
<neverredneverred>
maybe thats what I get for being cheap on an AWS time thats gunna cost me less than a dollar lol
<patagonicus>
I'm not sure 5GB is enough. I think it copies everything for the root fs and then creates the image for it, so it's going to need at least twice the size of the root image. And I think it compresses it afterwards, so that's probably ~2.5x the size of the actual root fs in tmp space.
<neverredneverred>
well, the command I used to resize it doesn't care if I resize it to 20 petabytes, do you know, maybe that command didn't do anything...
<neverredneverred>
so* you know
<neverredneverred>
`sudo mount -o remount,size=20000000G /run/user/1000`
<neverredneverred>
I wish nix would tell me where it was trying to unpack files to
<neverredneverred>
"cp: cannot create regular file 'source/drivers/media/dvb-frontends/dvb_dummy_fe.h': No space left on device"
FRidh has joined #nixos-aarch64
<neverredneverred>
`/tmp` has 26G available
<patagonicus>
What about /nix/store?
zupo has joined #nixos-aarch64
<neverredneverred>
ok, I ran `watch df -h` and saw `/run/user/1000` increase to about `2GB` when running the build. similarly, available ram starts dropping, and then its free when the build fails. Pretty sure its not enough ram
<neverredneverred>
`/nix/store` is on /, which has 26GB space free.
<clever>
neverredneverred: sounds like its using $TEMPDIR and your in a single-user setup
<clever>
if you change TEMPDIR, nix will put the temp files elsewhere
<clever>
ah, and you did answer above
<patagonicus>
Yeah, I'd guess that tmpfs is limited by RAM + swap available. :)
<patagonicus>
You could add zram into the mix, but that's probably going to make stuff slower. Easier to move nix' tmp dir to disk.
<sphalerite>
disk is a loooooot slower than compression.
<clever>
patagonicus: a tmpfs defaults to 50% of ram i believe
<clever>
but you can still set it above, like neverredneverred did above
<clever>
i try to set the size to something sane, based on ram&swap, so it wont push things too hard
<neverredneverred>
AWS is kinda nice. Super easy to just spawn a new instance.
<patagonicus>
Yeah, but you can't store more in it than ram + swap as far as I know. Even if it lets you set the size bigger than that.
<neverredneverred>
I'm gunna sleep now though, nix transcendence will have to wait
<clever>
patagonicus: exactly, but pushing too close to that edge can harm performance massively
<clever>
patagonicus: and the OOM killer cant kill a tmpfs
<clever>
so you may not have enough ram to recover the system
ky0ko has joined #nixos-aarch64
neverredneverred has quit [Remote host closed the connection]
FRidh has quit [Read error: Connection reset by peer]
FRidh has joined #nixos-aarch64
lgcl has quit [Quit: ERC (IRC client for Emacs 27.1)]
lgcl has joined #nixos-aarch64
ib07 has joined #nixos-aarch64
zupo_ has joined #nixos-aarch64
zupo has quit [Ping timeout: 246 seconds]
tilpner_ has joined #nixos-aarch64
tilpner has quit [Ping timeout: 256 seconds]
tilpner_ is now known as tilpner
lgcl has quit [Quit: ERC (IRC client for Emacs 27.1)]
lgcl has joined #nixos-aarch64
zupo_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
monk has left #nixos-aarch64 ["Error from remote client"]
alpernebbi has joined #nixos-aarch64
monk has joined #nixos-aarch64
monk has left #nixos-aarch64 ["Error from remote client"]
monk has joined #nixos-aarch64
blitzclone[m] has left #nixos-aarch64 ["User left"]
veleiro has joined #nixos-aarch64
monk has left #nixos-aarch64 ["Disconnected: Replaced by new connection"]
monk has joined #nixos-aarch64
monk has left #nixos-aarch64 ["Error from remote client"]
<sphalerite>
holy crap, is the emmc in my chromebook suddenly working reliably!??
<sphalerite>
ok not 100% reliably by the looks of it, but it's taking far fewer attempts for it to mount… I'm going to shut down and wait a while and test a few more times to see if it's not just a fluke
<sphalerite>
But I'd be very pleased if this was indeed due to 5.9.11 → 5.9.12
<Ke>
which chromebook?
<sphalerite>
gru-bob
monk has joined #nixos-aarch64
<sphalerite>
never mind…
ib07 has joined #nixos-aarch64
zupo has joined #nixos-aarch64
monk has left #nixos-aarch64 ["Error from remote client"]
neverredneverred has joined #nixos-aarch64
<neverredneverred>
So I built an image for the pinephone, but it fails to boot with a message about not being able to find a disk by label "NIXOS_SYSTEM". The output from the build command listed the partitions in the image generated, and the labels include "misc", "persist", "boot", and "".
<neverredneverred>
whats the magic incantation to add a label to the disk partition thats missing it?
<neverredneverred>
weird, e2label /dev/sde4 prints "NIXOS_SYSTEM"
lgcl has quit [Quit: ERC (IRC client for Emacs 27.1)]
<samueldr>
if it's by-label, it's by filesystem label, so it should be fine... assuming you're writing the whole disk image
<neverredneverred>
yea, I wrote the whole image to the sd card.
<samueldr>
though seeing you get the error on the pinephone (I assume) it means that the issue around the 3GB variant is not one of your worries
<samueldr>
with it it flat out wouldn't boot
<samueldr>
I assume you don't have the serial cable to get more output, right?
<neverredneverred>
yea, thats the error I get on the pinephone.
<samueldr>
can you share the full exact error you get?
<samueldr>
I've been thinking over the past few days it would be helpful to print all of the log from the system in another page on that error applet, and I think you're basically proving me right
<samueldr>
you can try running fsck from a computer
<samueldr>
last time I saw that error (literally yesterday) I flashed an incomplete image, it was truncated in my case
<samueldr>
did you `sync` the device, or use equivalent `dd` flags, after writing to it?
<samueldr>
(device -> sd card)
rajivr has quit [Quit: Connection closed for inactivity]
<neverredneverred>
yea, I ran sync
<neverredneverred>
and dd oflag=sync
FRidh has quit [Quit: Konversation terminated!]
<samueldr>
hmm, odd
<samueldr>
though it's pretty definitive the issue is that it failed to fsck
<neverredneverred>
Yea, it reports an error. "NIXOS_SYSTEM contains a file system with errors". Is clearing the journal inode fine? It prompts tat it has data.
<samueldr>
no idea, at this point it's not an issue specific to mobile nixos
<neverredneverred>
I validated the sha256 sums after scping the result image, so nothing went wrong there.
<samueldr>
one thing I'd look at is first validate the .img file you~~ ok
<samueldr>
haha
<neverredneverred>
:)
<samueldr>
next would be writing fresh on the sd card
<samueldr>
there is a method to `sha256sum` the sd card's version, which I don't have on hand
<neverredneverred>
what if I didn't wipe the sd card first?
<samueldr>
shouldn't be an issue
<samueldr>
it's still a block storage that reacts the same as others... when it is working right
<neverredneverred>
I'll clear the inode, try to reboot. They try a fresh flash.
<neverredneverred>
Now the prompt goes from verifying disk to continuing to stage-2. Then I see flashing green/orange led (kernel panic?). Then a reboot.
<samueldr>
sounds like something in stage-2 is wrong, probably from that issue
<samueldr>
I would personally never trust a fresh flash that fialed fsck
<neverredneverred>
I didn't provide the oflag=direct option, and I performed the flash from a mac. trying on a linux machine with that oflag.
<neverredneverred>
fresh flash, ran fsck and it reports "Superblock has an invalid journal (inode 8)"
<neverredneverred>
I ran the build directly from master of mobile-nixos
<neverredneverred>
Hmm, shouldn't there be an fsck check right after the build?
<neverredneverred>
ugh, I should've kept the build output.
<Ke>
direct is not the one you want but conv=fsync
<Ke>
well at least on linux
<Ke>
though I guess oflag=direct with bs=4M or something may be sane
<Ke>
but you still need fsync
<sphalerite>
that's what I usually use
<sphalerite>
(direct+big blocks)
<sphalerite>
Why do you still need the fsync?
<Ke>
because just sending the io does not flush it
<Ke>
on an sd card you won't likely lose much, but it's possible
<samueldr>
be careful about flags
<samueldr>
conv and flags are different
<sphalerite>
dd will sync at the end before exiting thoguh, no?
<samueldr>
sync on each of them does not mean the same thing
<sphalerite>
`cp foo.img /dev/sda && sync` ftw
<samueldr>
conv=sync -> pad every input block with NULs to ibs-size
<samueldr>
*flag=sync -> use synchronized I/O for data [and] for metadata
<sphalerite>
samueldr: but there's also conv=fsync :)
<samueldr>
yeah
<samueldr>
read your man page
<samueldr>
especially if you're not on Linux
<samueldr>
since it's likely to differe there too
<neverredneverred>
Not sure if this is relevant, but in the "# Add the PMBR" section of the build, the `cgpt` command invocation for NIXOS_SYSTEM doesn't have a "-l" argument or a "-u" argument, the others do.
<sphalerite>
that sets the partition label
<sphalerite>
which would be relevant if it were using /dev/disk/by-partlabel
<samueldr>
GPT has the concept of *partition* labels, which are different from *filesystem* labels
<samueldr>
both can be set
<neverredneverred>
gotcha
<samueldr>
but more importantly: make sure you don't mix-up (in your mind) conv symbols and flag symbols with dd
<samueldr>
and make sure you read the man page to know what they do and decide which dd cargo-culting school you'll follor
<samueldr>
follow*
<sphalerite>
actually what I usually use nowadays is pv
<sphalerite>
(and sync at the end to be sure)
<samueldr>
I use a mix of pv and dd in a script
<samueldr>
because I prefer direct I/O from dd
<samueldr>
otherwise pv finishes but still has to sync
<andi->
another day another attempt at writing a DT file...
<samueldr>
andi-: picking up from the few hints I gave you yesterday?
<andi->
yeah
<andi->
I feel like I changed nothing but whitespaces..
<andi->
but lets see maybe it compiles this time
<Ke>
I just don't know almost any use case
<Ke>
maybe if you are logging something bizarre with dd
<Ke>
so I won't rush them
<Ke>
I also wait a small amount of time, because a lot of this hw is somewhat faulty
<samueldr>
my use case: getting a truthful ETA
<samueldr>
pv being limited by dd, which in turn is limited by the hw, means that pv's ETA is good
<neverredneverred>
Could the issue be related to resizing of the partition? The first fsck error reported is "Superblock has_journal flag is clear, but a journal is present." After I clear that, I get a "Resize inode not valid."
<sphalerite>
btw angerman I think the helios64 just has a bit of existential panic at the beginning of its operational life… then it just sort of settles down and gets used to it. :p
<neverredneverred>
Should I just chalk this up to a botched build?
<samueldr>
neverredneverred: the fsck happens before the filesystem itself is resized, so I don't think it would be an issue
<samueldr>
sphalerite: I'm curious how mine will exhibit existential issues
veleiro has quit [Ping timeout: 264 seconds]
<neverredneverred>
well, I'm at the limits of my knowledge. Is this image salvageable? If so, what should I try next? I tried `cp && sync`.
<samueldr>
sync . syncs the current directory
<samueldr>
one thing you might want to do is verify the sd card is not broken in some way
<samueldr>
I'm not sure if f3 would help detect that
<andi->
in order news today the kernel just built.. gonna reboot the box now
<andi->
Might be required for the kernel to probe/support it..
<andi->
I can test both
<samueldr>
I would assume it's not needed, considering it already probes the bytes
<samueldr>
and I also assume these are drop-in replacement parts, where you don't need to produce a different revision of the board to use another chip
<andi->
I don't get how someone can let the patches rot on the ML after submitting them...
<sphalerite>
samueldr: kernel panics :)
alpernebbi has quit [Quit: alpernebbi]
<andi->
garr, since when does Nix no longer write .drv files on remote builders?!?
bennofs[m] has joined #nixos-aarch64
<neverredneverred>
So when I wrote the image to the sdcard, It repored X bytes copied, Y records in, X bytes copied and Y records out. Now, when I dd from the disk to an image, it reports Z bytes copied, Y records in, A bytes copied and Y records out. Why would a different number of bytes be copied from the card than would be written to the file?
<andi->
because the card has a different size?
<andi->
You are likely reading the whole card while you are just writing the image
<andi->
so while reading you have to constrain the size to the actual image size
<neverredneverred>
nah, I'm specifying a block size and count. The numbers are close to what was written
<neverredneverred>
When I wrote the image to the card, It wrote 1178634240 bytes, 140 records with bs=8M. When I write the card back to the image with dd count=140 bs=8M, It copies 1157627904 bytes from the card, and writes 1174405120 bytes to the file. The partial block copy is inconsistent with what was written to the card, but even then, idk why it would be
<neverredneverred>
different bytes copied vs written.
veleiro has joined #nixos-aarch64
<andi->
samueldr: the kernelModules parameter is not required as the spi-nor module is buitlin
<andi->
sadly adopting said patch to the current kernel didn't really do the trick
<samueldr>
probably was different for the pinecube build daniel//rf[m] was using
<andi->
have to check again what I might have done wrong
<andi->
yeah
<andi->
hah, I forgot the critical part in the patch to actually include the new struct..
<andi->
the proposed upstream patch is also missing that..
neverredneverred has quit [Ping timeout: 245 seconds]
<andi->
Has anyone looked at power savings onthe rk3399 chips? I having a bad time googling that topic
<neverredneverred>
anyone know about the dd issue I posted above?
zupo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<samueldr>
neverredneverred: last block was partial when written
<samueldr>
but reading, it was complete
shad has joined #nixos-aarch64
<andi->
samueldr: should we also add the spi binaries to the qemu builds in Nixpkgs?
<neverredneverred>
thank samueldr
<samueldr>
andi-: u-boot?
<samueldr>
andi-: probably
<andi->
yeah, u-boot
<samueldr>
I had that far on the backburner
<samueldr>
it needed to work for allwinner too
<neverredneverred>
I tried something else. I tried to mount the image directly, using an offset to specify the partition. It worked for mounting the boot partition, but when I tried to mount the NIXOS_SYSTEM, I get the following error: "mount: /mnt/pine_img: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error."
<neverredneverred>
The command I ran was "sudo mount -o loop,offset=X pine.img /mnt/pine_img"
<samueldr>
so I had to take my pinebook pro, roc-pc, and pine a64-lts and try to get things going in the best way
<andi->
so my mtd device seems to work just fine. I managed to write Hello World into it and read it again..
<andi->
Now finalizing the upstream kernel patch so it doesn't get lost again.
<samueldr>
(btw, I'm not saying don't do anything about it, simply that it'd be good to figure out if there's some level of abstraction)
<neverredneverred>
alright, I'm gunna call it quits. Thanks for the help folks.