Evan Carlin

So long SourceHut and Netlify

2026-09-03T00:00:00-06:00

Intro

For the past six years I've been a happy user of both SourceHut (my code forge) and Netlify (my blog host). A few days ago SourceHut changed their terms of service to no longer accept the "use of LLMs or other generative AI tools to produce or assist with the production of source code, assets, tickets, emails, and so on".

First off, I want to commend SourceHut for taking a stance that aligns with their values. I initially switched to SourceHut because it aligned with some of my values. I like the simplicity of their site, and paying for a service instead of trading my data for something "free". It takes commitment to put one's foot down and turn away potential customers. I'm happy they took their stance and I wish them all the best. While (I hope) I'm not drunk on the LLM craze, I do use LLMs to author some of my code and to help me solve problems. So, SourceHut's change set me looking for an alternative.

I received the email about SourceHut's TOS change while I was in the airport flying home from a work trip. I was in Las Vegas, and maybe it was all of the annoying screens and bright lights of the slot machines that pushed me to want to self-host on a dusty old server. Whatever it was, on my way home from the airport I swung by Colorado State University's Surplus Property store to see what they had in stock. It is like stepping back in time in there. Lots of old doohickeys with one foot in the grave. If I ever need a VGA cable again, I know where to go. I opted for an HP Z240 SFF. Onboard, this beast has an Intel i7-6700, 8GB of DDR4 (at current prices that was probably the entire cost of the machine), and a lovely 250GB spinning hard drive. $70 and it was mine. I got a nice nostalgic feeling booting it up, like firing up my childhood home computer. A great tactile power button, a nice whir to the fans, and the screech of the HDD. I've been using Mac laptops for too long...

To make matters better (worse?), I also decided it was time to switch to a proper config management tool. Something to provision machines so that when this box eventually died I'd be able to easily switch over to something new. Ansible, Salt, and the like have never really grabbed me, but I've read over the years about NixOS and it seemed like an interesting approach. So, with a weekend ahead of me, some fresh metal, and the helpful/infuriating/clever/moronic hand of Claude, I set off.

Porting router to NixOS

My first port of call was converting my BABS (big ass bash scripts) that configured my router (a Beelink EQ14) to NixOS. The scripts were about as close to a 1:1 conversion as I could hope for. For example:

This Bash

install -Z -m 600 /dev/stdin /etc/dnsmasq.d/router.conf <<'EOF'
<snip>
# Don’t read /etc/resolv.conf because we configure our
# own name servers.
no-resolv
server=1.1.1.1
server=9.9.9.9
<snip>
EOF

became this NixOS equivalent

{
  services.dnsmasq = {
<snip>

      # Don't read /etc/resolv.conf because we configure our
      # own name servers.
      no-resolv = true;
      server = [
        "1.1.1.1"
        "9.9.9.9"
      ];
<snip>
  };
};

This went fairly smoothly. Regardless of whether it was Bash or NixOS, I still grind my teeth doing this work. Networking code in particular can "work" but be totally wrong and/or insecure. Likewise, when it doesn't work there isn't a stack trace pointing me to the issue. Just the inability to reach the server. I want to make my configuration repo public, but I'm afraid there will be some glaring bugs in it. Putting aside my epistemic anxiety, I came away really liking NixOS. The internet tells me folks balk at the syntax, which I found fairly pleasant to read and which LLMs were good at generating (after deleting an absolute mountain of comments to get at the actual code underneath...).

Porting blog and code forge

On to my new server. I first needed to figure out which code forge to use. Prior to SourceHut, I used both GitHub and GitLab. I would describe both of them as "meh", or "what I use at work". Recently GitHub has been circling the drain looking for BitBucket down in the p-trap. I chose Forgejo as my new code forge. I thought for a moment about self-hosting SourceHut, but in their TOS change blog post they themselves said Forgejo is "easier to deploy". Maybe a dig at LLM sloppers?;) Only so many oceans I can boil in a weekend...

My blog was very easy to port over. I have had no problems with Netlify, but I also don't really get much from them. Might as well ditch them along the way. My blog is just a static site generated by Pelican. So, it was a matter of pointing nginx at the files it outputs. Again, NixOS made this simple. Friendly settings like "enableACME" are easier than managing the systemd services myself, and their language reads a lot like the nginx conf files I'm used to. For example:

virtualHosts = lib.mkMerge [
    {
    "evan.carlin.com" = {
        enableACME = true;
        forceSSL = true;
        root = "/var/www/blog";
    };
    }
    <snip>
    ];

So long

A Saturday evening, all day Sunday, and I was back up and running. I have another hand-rolled box (Beelink SER8) still to switch to NixOS. This one runs Home Assistant and UniFi OS Server in VMs, so I think it will take a bit more noodling to get them moved over. We'll see how this journey goes. I also switched my yearly payment to SourceHut into a donation split between Forgejo and NixOS. I told my wife recently that I wanted to simplify my life. Too many things vying for my time. I don't think this is what she or I had in mind.

Happy Hacking.

McMansion Programs

2026-05-25T00:00:00-06:00

I had some cleaning up to do this morning so I tuned in to the Oxide and Friends podcast. I chose the episode "Are LLMs Insufficiently Lazy". I haven't finished the episode (or my cleaning) because it led to a post by Bryan Cantrill called "The peril of laziness lost" (quick and worth a read).

One sentence that really struck a chord with me in that post is: "Left unchecked, LLMs will make systems larger, not better". Oh how true that is!

I've been recently calling it "industrial programming". While writing this post I thought of calling it the "supersize era". I then thought of "McMansion Programs". If I were an LLM I'd use all of these abstractions interchangeably throughout this post and probably dash in a few more along the way.

We seem to be fully immersed in a moment in software where we need more "stuff". Burn more tokens. Write more lines of code. Ship more features. IPO for a trillion dollars. Who cares if it's better.

What we seem to be lacking in all of this is taste. A quip I learned from my older sister which I've repeated many times is "money can't buy taste". McMansions are the USA's greatest call to this saying. Humongous homes. Sloppily built. Filled with junk. Expensive to maintain. They give off the appearance of great wealth. But, take another look or use any of the janky hardware in the home and the shine will be gone.

The programs LLMs build tend to fall in this style. Just because we used a lot of tokens building something doesn't make it any good. With LLMs we can build first and think later (or not at all). We can create more features and new additions faster than we can understand our own code/product and certainly faster than our customers can make sense of it. Instead of refining the features we do have and carefully adding new ones we are able to implement every idea that comes into our head.

With the aid of LLMs we have to be more careful. Review our code more deeply before opening a PR. Take another pass at honing it. The first pass we make as humans isn't good enough. Likewise, the first pass the LLM makes isn't good enough. In 1983 Stephen Johnson and Brian Kernighan said "first make it work, then make it right, and, finally, make it fast". In the same year Butler Lampson said "Plan to throw one away". With an LLM, making a version and throwing it away is now easier. But, it is just as important as ever to throw (at least) one away.

Extending the McMansion metaphor a little further: A curious overlap between McMansion Programs and McMansion Homes is that they were both aided by a new technology. In the case of McMansions Homes it was the truss connector plate. If you're curious about that sort of thing this video goes into more details on it and the industrialization of home building. I hope our programs aren't doomed to the same soulless fate as many American suburbs.

Shop Notes v3: Thank you doonut

2026-05-25T00:00:00-06:00

After managing to remove the brake rotor from the track driveshaft I thought my journey with this part of the snowmobile repair was over. I could now remove the driveshaft from the brake caliper which would let me remove the caliper. I could then push out the seized brake piston and be on my way. How naive.

What I should've realized was that if the brake rotor was so seized on the shaft that it broke my puller, then everything else would be equally seized on the shaft.

I won't go into all of the details, but I ended up cutting the brake caliper off of the shaft. I highly doubt this was the best path to take, but in my frustration it is where I ended up. What I was left with was one destroyed brake caliper (easy enough to replace) and a track driveshaft with 2 inner bearing races and 1 spacer completely seized on the shaft. At this point, I didn't want to also have to buy a new (used) track driveshaft so I slowed down a bit.

I could see no good way of removing the races from the shaft. They were completely seized on. Two nights of a PB B'laster soak made no dent. And there wasn't enough purchase anywhere to get a puller on. Off to do some research.

I came across this wonderful post which instructed me to cut a groove in the race. Cut it as deep as I could without going into the shaft. Then use a cold chisel and heavy hammer to split the bearing.

The post said "a real good whack". Mine took considerably more than one whack. But, eventually, the bearing split and I could then walk it off!

Closing thoughts

Many thanks to forums and the people who post in them. For years I've felt a growing sense of shame that I have consumed so much helpful information from folks on the internet and paid back almost nothing. Besides one YouTube video and a couple Stack Overflow answers when I was bored at my first job, I don't think I've ever posted on someone else's site. I'm not sure how or when that will change.

I hope Meta's new Forum app is a flop and people continue to post freely on forums that are indexable, viewable without an account, and not run by such a turd of a company.

Shop Notes v2: The cat strikes back

2026-05-10T00:00:00-06:00

Tool Foam

Working on my snowmobile again has me using my wrenches. My wrench drawer has been on my list for organizing. I bought a CNC at the start of this year and it has been very fun to organize the tools in a drawer in Fusion 360 and then have the CNC cut out their profile in some tool foam. On Saturday I made an insert for my wrenches.

Snowmobile

Now that the sled is running there are just a couple of loose ends to finish. Chief among them is fixing the brakes. They drag and definitely need a bleed.

To fix the drag my thinking was I'd need to pull the brake pistons and give them a once-over with some fine Scotch Brite. To do that I'd need to pull the caliper and to do that I'd need to pull the disc rotor.

The job started with a minor hiccup. I stripped one of the socket head bolts in the caliper. Thankfully by switching to a larger size hex key I was able to get the bolt out. Not often that trick actually works.

The disc rotor bolt was also somewhat of a challenge. It is an odd shape and requires a special socket. I have the socket but I had a hard time getting it to seat far enough on the bolt that I could put the torque I needed to on the wrench.

I made my way through, and once the outer half of the caliper and brake rotor bolt were off I tried yanking on the disc rotor. No movement. The rotor sits on splines on the track driveshaft. I figured it was rusted on like many of the other parts on the sled. It was getting late so I sprayed some PB Blaster on and gave it a night to soak.

In the morning I gave the rotor another yank. Nothing. I sprayed on some more PB Blaster and went on a hike. When I got back from my hike I was very hopeful the rotor would finally break free. This time I used a small pry bar. Still no movement. Bummer.

A search through the forums turned up a couple of posts with people facing the same problems. Many of the replies mentioned using pullers. Hmm, I think I've seen those at Horror Freight. Should I go buy them? I struggle in moments like this with what tool to buy. In general, tools that I know I will repeatedly use I don't mind spending some money on to get a well-made Western tool. But, for tools I rarely use I'm okay buying a cheap Chinese version. The problem is when a project is going sideways a good tool can straighten everything out. A bad tool can make it 100x worse ("the most expensive tool is a cheap tool"). I don't know that I'll use pullers enough to justify a nice tool. The folks at Garage Journal recommend the SnapOn (figures...) as well as the OTC or Proto. I've had very good luck with OTC tools and pretty good with Proto. But, both would take a few days to get here and be several times more expensive. I went back and forth but in the end decided to stay cheap today and I bought the 4-piece puller set from Harbor Freight.

It took some finagling to figure out how to get the jaws lined up and locked in. But once they were on I started to crank. The first couple times the jaws snapped off with a bang. In both cases the rotor itself had bent and the jaws then slipped off. I think initially I was using too small of a puller set. There was a balance between what jaws I could fit in the limited space behind the rotor and what jaws were big enough to get the job done.

I switched to the 6" jaws and a 3 foot breaker bar. I really started to put my weight on it. The wet noodles they use to make Pittsburgh products gave way.

Dang! Should've bought the nice pullers... Now I had a rotor that hadn't moved a millimeter and a puller stuck on with no easy way to get it loosened. I devolved into a primate and started to wail on the puller with a dead blow hammer. I believe this is the only time in my entire life that this did anything even remotely positive! After many blows I went to try and loosen the screw on the jaws and I could twist it with my fingers. The rotor must be coming off! I twisted off the puller and now there was enough space for the 8" jaws to fit behind the rotor.

The 8" jaws made quick work of the rotor now that it had mostly come off.

Lots of rust on that thing. I don't think I've ever had to work that hard to get a rusty piece off.

Some carnage from the puller.

That is 5 pieces of 1/8" plate steel that the lead screw of the puller was pressed against. It looks like they stopped a bullet.

A bent bolt from one of the puller arms. I'm very glad that didn't break and send pieces flying.

All's well that ends well. I'll be testing out Harbor Freight's return policy tomorrow morning.

Shop Notes v1: The cat purrrs

2026-05-04T00:00:00-06:00

The Players

I've been kicking around the idea of keeping a journal of things I do in the garage / around the house. My sister, Emily, keeps a journal of her woodworking journeys and while I was talking to her on the phone yesterday she encouraged me to start writing my own posts. With her encouragement and a successful day in the shop I figured now is as good of a time as any for my first post. Also, if you'd rather read good writing instead of my boring prose head over to her site.

The Setup

A fine feature of rural areas that get snow: rundown houses with even more rundown piles of decaying snowmobiles in the yard. In November of 2023 I had the great fortune of helping one of those people unload some of their junk. For $600 I picked up a barely running 2006 Arctic Cat M7 snowmobile.

The Hook

The new sled was in poor but serviceable condition. These sleds are known to be durable and there is decent availability of second-hand parts. So, I brought her home and tore in.

The rough plan was to clean up all of the 2-stroke oil splattered across the engine, install a new top end, cross my fingers, and be surfing pow by spring.

The Tale

The project started with a bang. I got the engine out quickly. Cleaned it up and started disassembling. I swapped cylinder jugs for new ones. Put in new pistons. Replaced some old hoses. Yada yada yada. Breaking things apart is easy.

The Wire

Assembly is where I hit the wall. The clutches both needed to be rebuilt. The shocks were shot. And the engine had many suprises. When you rebuild a 2-stroke engine, you get everything back together, plug up all of the holes in the engine, pump some air in, and then see if/where air leaks out. The goal is for little to no air to leak out. This engine was leaking like a sieve. I could hardly pinpoint a single place to get started. Out came the high-temp silicone sealant. I unbolted much of the engine and slathered it all over. The leaks slowed but the engine was still losing too much air. More silicone. Fewer leaks. Rinse and repeat. Eventually I narrowed the final (and worst) leak down to a crack in the "y-pipe". That is the part of the exhaust that comes straight off of the cylinders. Doh. That's where all of the oil soaking the engine was coming from!

The Shut Out

This started an Olympic-caliber project detour. The section of exhaust that was cracked is no longer manufactured. The ones available on eBay looked equally as rusty as mine but I bought one anyway. It was cracked in the same spot. I've wanted to learn how to weld and this seemed like a great time to do it. I could learn a new skill and fix my project.

Denver has a very cool tool library. You can rent tools, take classes, and meet like-minded folks. Weeks went by. The welding class came. I took it and could barely get two pieces of metal stuck together. It turns out welding is actually a skill that takes practice and not something I could pick up in an evening (who would've guessed!?). I decided I'd take the level 2 course to get more experience. More dismal welds. I then bought a welder and started practicing at home. ~~When I finally felt like I was ready~~. When I finally didn't feel like waiting anymore, I took my preschool-level welding skills to my exhaust. What I would come to learn is this exhaust is one of the more challenging welding tasks I could've given myself. The metal is exceptionally dirty. It is cast steel. The area I needed to weld was hard to reach. The weld needed to be airtight. And I bought a TIG welder. TIG welders really dislike dirty metal. For the non-welders, that is all a recipe for welding to not go well. Add in my impatience and I just blew hole after hole in the metal. What started as a minor crack turned into a yawning cavern. I eventually got some metal glommed on. But, it was definitely not air tight.

At this point I think winter had passed. I was sad about my welding journey and mostly left the snowmobile alone. Aside: A colleague recently asked me how I work on so many projects. My answer - by starting many and finishing few.

More time passed. I proposed to my girlfriend (now wife:)) and we moved to Fort Collins. The night before the movers came, I manically bolted things back together. Not before I had to unpack all of my tools I had neatly packed that day. If I was going to move this snowmobile it had to be put back together enough to be strapped to a trailer and driven 60 miles.

Upon arriving in Fort Collins, the snowmobile sat for another winter collecting dust. Too many new house projects to distract myself with.

The Sting

The weekend before last I decided enough time had passed. No more distractions. It was time to get the snowmobile running or sell it. I bit the bullet and found a company selling a new upgraded exhaust. I ordered it and a few other odds and ends and was off to the races. I really wasn't that far from getting this snowmobile running. All told it took me two Sundays to get everything back together.

Here it is on first start. Volume up.

Thanks to Em for the encouragement to write this. It brings me a great deal of happiness on a weekend when I'm out in the shop to think of her and know she's probably in her own garage lost in a project as well. And thanks to my neighbors for putting up with the noise and smoke. Don't show that video to the EPA.

Lessons learned
- Job 1 is "job done". My wife likes to describe most of my shop time as "low priority tasking." I did an ungodly amount of that in this project. I should've written a todo list every day and stuck to it.
- Don't mix projects. To save myself a couple hundred dollars on an exhaust pipe I ended up spending several hundred dollars on classes and welding equipment. All to end up right back where I started. I wanted to learn how to weld. But, that should've been its own activity. I think it set my welding journey back by how frustrated I got.
- A well-written shop manual is a work of art. The Arctic Cat shop manual I used had spectacular descriptions and a picture at every confusing step. I pray with all of my might that AI slop doesn't crush this industry. When embarking on a project, try to see if there is a good manual beforehand. It will make the whole thing a lot more fun.
- Most important: Take a thousand pictures. As the years passed, I invariably forgot how things went together. Shop manuals are great but they can't go into every little detail. Being able to look back from many angles is extremely helpful. I even tried my hand at filming.

Notes: Autodesk Fusion 360

2026-03-07T00:00:00-07:00

Cutting a toolpath in half

Nice to be able to cut toolpaths in half when working on odd 3D shapes. That way I can clamp half of the piece down really well on one half while milling the other half and then switch.

Make a construction plane offset from the surface you want to mill
Draw a sketch on that plane around the area you want to mill
In manufacture create your toolpath (adaptive, parallel, etc)
In the geometry tab set the machining boundary to your sketch. Under Model select the faces to mill (KEY: uncheck "Include Setup Model". Under Avoid/Machine Surfaces you can select surfaces to avoid if you find your toolpath running into anything weird.

Ref: https://forums.autodesk.com/t5/fusion-manufacture-forum/parallel-tool-path-is-plunging-on-the-end-of-a-face-selection/td-p/10035789#M71684

Inserting a canvas and tracing

This is useful for things like tracing a wrench to build a tool foam drawer

Insert > Canvas
Document explorer > Canvases > canvas you inserted > calibrate
In the edit menu of the canvas you can flip it horizontally/vertically
Trace a fit point spline around it. For tool foam I've found being a tiny bit wide of the tool is best (too tight then it is hard to get in and out)

Moving a canvas and spline sketch

Once I've inserted a canvas and drawn a spline it can be nice to move them together to adjust their positioning.

Document explorer > canvases > canvas you inserted > edit > adjust x/y
edit sketch > draw box around all points > press m key (for move) > move x/y same amount as canvas

Variables

Create: s (hotkey to open shortcuts menu) -> Change Parameters -> Click + button in dialog box to create new use

Use: When you need an amount just type the name of the var. Can be used in expressions too (2+outside_diameter)

Dogbones

Install this lib to make it easy to place dogbones for mortise/tenon in wood. https://github.com/DVE2000/Dogbone

s -> dogbone. You can manually select/deselect dogbones.

Mortise and Tenons

https://forge.carlin.com/e-carlin/F360MortiseAndTenon

s -> Mortise & Tenon

French Cleats

https://forge.carlin.com/e-carlin/F360FrenchCleat

s -> French Cleat Hook

Midplane in a body

Select two opposite faces on the body (like the two ends of the board).

Aligning design for manufacture

Arrange requires a commercial license. This is how to do it manually on the free personal license.

1. Convert bodies to components

Browser → expand Bodies → Shift-select all bodies
Right-click → Create Components from Bodies

2. Create a Manufacturing Model

Switch to MANUFACTURE
SETUP → Create Manufacturing Model
Right-click Manufacturing Model 1 → Edit Manufacturing Model

3. Align each part flat on XY

MODIFY → Align
Select the part's large flat face → select Origin → XY Plane
Flip if it lands below the plane → OK
Repeat for each part
Check from FRONT view: all parts sit on the ground line

4. Space the parts apart

TOP view → MODIFY → Move/Copy
Gap between parts: ≥ 2× bit diameter
Accept Capture Position if prompted
Verify gaps with INSPECT → Measure
Finish Edit

5. Create the Setup

SETUP → New Setup → Milling
Model: Manufacturing Model 1
WCS: Z up, origin at stock box point (top corner)
Stock: Fixed size box = sheet size × thickness

Easy Links to Source Code with Emacs

2026-01-22T00:00:00-07:00

This little bit of elisp is probably my most used command. I use it a dozen times a day for sending links to colleagues. When run with my cursor over a line of code it will generate a GitHub url linking straight to the line of code.

Someday I'll make it smart enough to read the git remote and pull the url from there so it can work with non-GitHub repos.

(global-set-key (kbd "C-c cg")
                'ec-generate-github-file-url)
(defun ec-generate-github-file-url ()
  "Generate a GitHub URL for the current buffer file and line number."
  (interactive)
  (if buffer-file-name
      (progn
        (kill-new
         (format "https://github.com/%s/blob/%s/%s#L%d"
                 (replace-regexp-in-string
                  "https://github.com/\\(.*?\\)\\(.git\\)?$" "\\1"
                  (string-trim (shell-command-to-string
                                "git config --get remote.origin.url")))
                 (string-trim (shell-command-to-string "git rev-parse HEAD"))
                 (file-relative-name buffer-file-name
                                     (vc-git-root buffer-file-name))
                 (line-number-at-pos)))
        (message "Copied GitHub URL"))
    (message "Not visiting a file.")))

LLM's Can Delete Code Too

2026-01-22T00:00:00-07:00

In the past month I’ve opened 2 PRs that deleted 2 confusing features my team was fighting against. In total it was about 1,400 lines of code. It turns out both of these features were completely dead code and the battle was in vain. An LLM was no use in finding out that the code was dead. It led all of us down confusing paths, spinning tales about what work we needed to do to support these features. But once I realized the code was dead, I could prompt an agent to help me clean up the code.

Pre-LLM, one of the more tedious parts of deleting code is finding all of the code that can be deleted after each round of deletion is through. I delete some code, now I need to go find all of the code that was only used by the deleted code, delete it, recurse. I found prompting an agent to help with this dramatically sped up the process.

I’m not positive why I trust an LLM to find all related dead code when it couldn’t even tell me the code was dead in the first place. But I have a sense (and some double-checking told me) that it seemed to be doing a good enough job.

Best case is that dead code isn’t left lingering around codebases. And LLMs struggle to find it (as far as I can tell). But once a human identifies it, an LLM is pretty decent at cleaning up the mess. About the least it can do with all of the verbose brain-dead code it seems to create.

Stripe is Good

2025-02-28T00:00:00-07:00

It is always easier to tear down than to create. I don’t consider myself a negative person, but I’ve done some complaining on this blog. I’d like to change course on that.

Today, I was implementing Stripe for the second time (once before on Moss and now on Sirepo). I have to say that each time has been a pleasure.

Good Docs

Like, seriously good docs. Like someone deeply thought about them. And then someone else did a really thorough review. And then someone else diligently kept them up to date. And then someone listened to developers and added more docs for specific things that were causing people confusion.

Documentation like this is not easy to make. It is a whole job function and can’t just be tacked on after the fact. I’m really grateful for their work.

A prime example is the first docs you’ll probably come across: the quickstart. They have major frameworks/languages ready to go. They also have raw HTML as an escape hatch (we use AngularJS, so I started with the basic HTML docs and adapted). These docs are thoughtfully written. They cover enough of the initial use case to be useful. It is a fine line in quickstarts to balance useless toy “hello world” examples vs. information overload, and they nailed it.

CLI to Make Testing Easy

At Moss, we haven’t yet implemented handling webhook events. On Sirepo, we are going to want to handle them out of the gate. When I was building the plan for implementing Stripe in Sirepo, I set aside extra time for developing/testing the webhook. With other tools, testing webhooks can be a pain. They don’t have a good way to trigger events in development to easily test. Not the case with Stripe.

First, one needs to install the CLI. They have that covered in spades. They handle all major OSs/distros (and then some). And it is just a painless process. Getting the RPM for Fedora and installing it took me 30 seconds.

The CLI has a local listener, so you can easily forward real webhooks to your development server running on localhost. No ngrok or socat needed here. Just a simple tool that does exactly what so many systems like it need.

Finally, you can trigger any event. They make this dead simple (also a consequence of their good API design).

I had set aside half a day for hacking up some testing of the different webhook events we are going to need to handle. In the end, I spent only about an hour implementing and testing. That is a huge gain and is thanks to the work Stripe has done to make developers’ lives easier.

Bugs

I was cutting a release one night for Moss and ran into a bug caused by the Stripe Python SDK. Stripe was quick to fix it. My only slight comment is that for a regression like that, one should add a test. But we can’t all be perfect all the time. :)

Thank you, Stripe! You’ve saved me time and made your product easy to integrate. Please take my money!

Cloud Sync/Storage

2025-01-22T00:00:00-07:00

For the past three years, my family and I have changed cloud sync/storage providers each year. We started with Google Drive/Photos, then moved to Dropbox, and most recently to OneDrive. Don’t ask me why we’ve changed so many times. Maybe we’re cheap? Maybe we’re dumb? Probably both. Each provider has its pros and cons, but OneDrive has been the worst so far. I’m thinking I’ll switch back to Dropbox, which seemed to be the least evil.

As part of this, I thought about the features I need from a cloud storage provider. I’m writing them down so that next year, when I get annoyed (hopefully not!), I can remember what I actually need and see what’s out there.

Sync and backup, duh

This is the basics. I need to be able to sync with different phones and computers and count on the service to not lose my data. I don’t want all data in all places at all times, but I want to be able to pull files onto a device as I need them.

Mobile app with photo backup

I need an iOS app that backs up all photos and videos I take. I never want to think about this feature. It better just work day in and day out without me fiddling. I also want some sort of logic to decide how these photos are named. Ideally, they should be named using the date and time they were taken.

View pictures

I want to be able to view all my pictures in the web and iOS UIs. I want this to be fast. I don’t know what it is, but some services are just terribly slow when loading thumbnails.

CRUD with text files

I leave small text files scattered throughout folders I back up. For example, I have a folder where I store information about a snowmobile I’m bringing back to life. I have a shop_log.txt where I record reminders about the work I did, a todo.txt for work I need to do, and a to_buy.txt for parts I need to order. The UI for working with these files can be extremely simple, but I need to be able to access and edit them on both the web and iOS.

Google Docs/Sheets integration

For better or worse, I use Google Docs and Sheets when I have to create those types of documents. I’d like some sort of integration between the cloud storage provider and Google. Ideally, I want to be able to search for files on the storage provider, click on them, and have them open in Docs or Sheets. From there, I can do my work, and it should be automatically saved back to the storage provider.

Ability to ignore files/directories

I need to be able to ignore certain file types and directories. For example, I have multiple node_modules directories under ~/OneDrive. OneDrive diligently backs them up—even though I have no desire for that. To make matters worse, when looking at pictures in OneDrive, it shows me thousands of images from node_modules.

Nice to have: Photos AI

I should really organize my photos, but I never make the time. So, I like having the ability to use AI searches to have some shot at finding photos.

I think that’s all. I’ll update as I come across other things.

Emacs

2025-01-07T00:00:00-07:00

Below is a running list of things I've learned about Emacs that I want to remember but I'm unlikely to recall off the top of my head.

Kill buffers

C-x C-b # open helm-buffers-list
D # mark buffers for deletion
x # execute deletion

M-x proced

Like top but in Emacs

HomeNotes: Electricity

2025-01-04T00:00:00-07:00

Outlets

Black (hot) to gold, white to silver
Line is incoming power. Load is outgoing power. For example, imagine a kitchen counter wall with many outlets. The line power comes from the electrical panel (or one of the other outlets) and then the load power goes to power the next outlet in the line. When the wires reach the new outlet then they go into the line power for it. Recurse.

Code

GFCI protection is needed on each circuit in a kitchen. So, if you have multiple outlets you can wire the first one to be GFCI and that protects the whole series of outlets on that circuit.
Outlet orientation is notdefined. But, generally hot side (smaller opening) is on the right hand side when outlet is vertical.

GitHub URL UX

2024-11-17T00:00:00-07:00

I was adding a GitHub URL to a presentation to reference the source code of what I was talking about. The URL was:

https://github.com/slaclab/slactwin/blob/3c84ca7d4b3b31c21d07a95cf77766507d4e1c97/slactwin/pkcli/db.py

That's pretty long for text on a slide and took up more real estate than I liked. A hyperlink would work, but I wanted people to see that the link was from GitHub and hopefully that would prompt them to click on it and look at the source code. Git (and by extension GitHub) works with shortened commit hashes. Usually, just the first few characters of a hash are enough to uniquely identify a commit. Looking at the repo, it seems like GitHub is using the first 7 characters. So, I tried shortening the URL:

https://github.com/slaclab/slactwin/blob/3c84ca7/slactwin/pkcli/db.py

Voila! That works! That will fit nicely on my slide. I find GitHub, in general, is kind to its users regarding URLs. For example, you can go to /issues/pr_number, and it will redirect to /pulls/pr_number:

https://github.com/radiasoft/sirepo/issues/7292
# redirects because 7292 is actually a PR
https://github.com/radiasoft/sirepo/pull/7292

Often in issues, emails, Slack, etc., people will put something like #1234. Just by looking at the number, it is impossible to know whether it is an issue or a PR. So, GitHub lets you just say it is one or the other, and it will do the work of redirecting if you guessed wrong. One caveat is that (to my eyes) there is confusion in their URLs regarding words being plural:

# pull is singular
https://github.com/radiasoft/sirepo/pull/7292
# issues is plural
https://github.com/radiasoft/sirepo/issues/7292
# pull -> pulls searches author:<number>
https://github.com/radiasoft/sirepo/pulls/7292
# issues -> issue is a 404
https://github.com/radiasoft/sirepo/issue/7292

That's a bit odd. I wonder why? Also, in case you didn't know, when you want to provide a link to a line of code, you can select the line number next to the link, and GitHub will link directly to that line (with it highlighted). Before copying the link, you can press y, which will change /blob/master in the URL to /blob/<commit_sha>. Including the commit SHA in links ensures future readers go to exactly the line of code you were looking at when you created the link. Without this, who knows what line 62 in foo.py on master will be pointing to 3 years from now.

P.S.

While writing this, I learned git has configuration for the number it uses when abbreviating hashes. Nice.

TypeScript (+tsx) mode for Emacs.

2024-10-07T00:00:00-06:00

It has been a while since I did much development with React. Last time I had a good setup I was using VSCode. But, I wanted to do some work with it so I took some time to get a good Emacs setup.

The React docs now mention to use a tool like Next.js on top of React (some great fodder for another article...). So, I setup Next.js and the default config is TypeScript. Of course, Emacs doesn't have great support for TypeScript out of the box. I briefly felt the pang of wanting to switch back to VSCode. But, I dug in and decide to make it work in Emacs. Below are my notes on how you too can get it setup.

I've done the following procedure successfully on an M2 Macbook. An Intel Mac probably won't require anything different. For Linux you'll have to use your OS's package manager (I use homebrew below) but the rest should work the same. No clue about other systems.

Compile Emacs with Tree Sitter

Mastering Emacs has a wonderful article on how to get Emacs with Tree Sitter support as well as some background on Tree Sitter. I copied quite a bit of my install directly from that article.

Emacs has Tree Sitter support but it isn't enabled by default (as of writing). So, you'll likely need to compile Emacs from source with Tree Sitter enabled:

#!/bin/bash
set -eou pipefail

brew install wget
cd ~
declare v='29.4' # In 29 emacs has tree sitter support but need to be compiled with it
if [[ ! -f "emacs-$v.tar.gz" || ! -d "emacs-$v" ]]; then
    wget "https://ftp.gnu.org/pub/gnu/emacs/emacs-$v.tar.gz"
    tar -zxf "emacs-$v.tar.gz"
fi
cd "emacs-$v"
# Install adapted from https://www.adventuresinwhy.com/post/compiling-emacs-with-tree-sitter/
brew install \
    autoconf \
    gcc \
    giflib \
    gnutls \
    imagemagick \
    jansson \
    jpeg \
    libgccjit \
    libpng \
    librsvg \
    libtiff \
    texinfo \
    tree-sitter
CC=gcc-12 ./autogen.sh
# --with-tree-sitter is all you really need. I like the rest so I can view images.
# --with-native-comilation=aot used to really speed up Emacs but the effects seem
# to diminish with every release. Still can't hurt.
CPPFLAGS="-I/opt/homebrew/opt/jpeg/include" LDFLAGS="-L/opt/homebrew/opt/jpeg/lib" ./configure \
    --with-gif \
    --with-imagemagick \
    --with-jpeg \
    --with-native-compilation=aot \
    --with-png \
    --with-rsvg \
    --with-tiff \
    --with-tree-sitter \
    --with-x-toolkit=gtk3 \
    --with-xwidgets
make install
mv nextstep/Emacs.app /Applications

Okay. You should now be able to run the Emacs app.

Adding tsx-mode.el

I'm using tsx-mode.el to add a major mode for TypeScript in Emacs. To install it you'll need to:

cd emacs.d # Whever you keep your .el files
# Add to your init.el.
cat >> init.el <<'EOF'
;; Enables downloading of grammars for different languages
(setq treesit-language-source-alist
   '((bash "https://github.com/tree-sitter/tree-sitter-bash")
     (cmake "https://github.com/uyha/tree-sitter-cmake")
     (css "https://github.com/tree-sitter/tree-sitter-css")
     (elisp "https://github.com/Wilfred/tree-sitter-elisp")
     (go "https://github.com/tree-sitter/tree-sitter-go")
     (html "https://github.com/tree-sitter/tree-sitter-html")
     (javascript "https://github.com/tree-sitter/tree-sitter-javascript" "master" "src")
     (json "https://github.com/tree-sitter/tree-sitter-json")
     (make "https://github.com/alemuller/tree-sitter-make")
     (markdown "https://github.com/ikatyang/tree-sitter-markdown")
     (python "https://github.com/tree-sitter/tree-sitter-python")
     (toml "https://github.com/tree-sitter/tree-sitter-toml")
     (tsx "https://github.com/tree-sitter/tree-sitter-typescript" "master" "tsx/src")
     (typescript "https://github.com/tree-sitter/tree-sitter-typescript" "master" "typescript/src")
     (yaml "https://github.com/ikatyang/tree-sitter-yaml")))
(require 'tsx-mode)
(add-to-list 'auto-mode-alist '("\\.[jt]s[x]?\\'" . tsx-mode))
EOF
wget https://raw.githubusercontent.com/orzechowskid/tsx-mode.el/refs/heads/main/tsx-mode.el
wget https://raw.githubusercontent.com/orzechowskid/tree-sitter-css-in-js/refs/heads/main/css-in-js-mode.el
M-x package-install coverlay origami corfu # Deps of tsx-mode
npm install -g typescript-language-server # Language server eglot will use
M-x treesit-install-language-grammar typescript # Install the TypeScript grammar

Profit

Not too bad when the instructions are set out like this. But, it took me an evening to figure out how to string all of the pieces together. You should now be able to open a .tsx file and have tsx-mode provide a nice major mode for working with the file.

Kubernetes: A first pass

2024-08-23T00:00:00-06:00

Prologue

What follows is really more of a journal entry than a blog post. It is a start at a collection of my thoughts on using Kubernetes (K8s). I'm planting this stake in the ground so I can see how my thoughts change over time.

Background on my world with regards to K8s

K8s has been the talk of the town among the physics/software community I've been working in. Many large institutions are switching to it to manage as many of their applications as they can. CERN being the largest and most well known of them. So, I've been looking forward to my chance to use it.

At RadiaSoft we deploy everything inside of docker containers. But, we've chosen to not use K8s to manage containers. In as few words as possible our reasoning is: K8s doesn't provide enough abstractions at a high enough level to actually solve our problem. We'd have to write a piece of software on top of it to speak a langauge above it for our apps. And we'd still need a configuration management tool like Ansible to actually configure our metal. So, we'd add a dependency (K8s) and not really be any closer to having our problems solved. Instead we wrote our own tool RSConf which speaks a level of abstraction that matches our domain. It is akin to ansible - we use it to configure our machines. It sets up docker and systemd (among many other things). Then those two (along with our custom job supervisor) do mostly what K8s would do for us.

Inside of Sirepo, our web-app for runing physics simulations, is where the job_supervisor lives. The job supervisor manages our "agents" which is where the actual simulation/computation occurs. In production we use agents running in their own docker container. For development (and quick prod setups) we run local agents which just run as a unix process. In prod we also run "sbatch agents" which are unix processes run on login nodes at super computers. Those agents then manage job submissions through the SLURM workload manager. So, we run our "top-level" services using docker/systemd and then the job supervisor has flexibility about the types of agents it creates and it does the management of them (ex restarting if they die).

So, that's where I stood going into using K8s. We have an app and it could be deployed in K8s instead of with systemd. And maybe parts of the job supervisor could even be replaced with K8s. Or at the very least, it could "speak K8s" so agents would be started as K8s pods. I was curious how everything worked and excited to get the chance to see.

My first real use of K8s

We've been working with a national lab who is making the move to K8s. They've setup a cluster and have a couple demo applications deployed. We're working for them on a contract to build a better dashboard for visualizing simulation results. Eventually we will run a "digital twin" of their system. This would be a simulated system that would run alongside the real life system and could be used by operators to help optimize the functioning of the real system. So, I have been tasked with deploying Sirepo inside of their K8s cluster.

Sirepo consists of multiple front-end servers, one job supervisor, and n number of agents that run the simulations for users. The database is split between sqlite and postgres. A few services but nothing that crazy.

Time to get coding...

The make make problem

The first thing I noticed about K8s is a common problem. I know it as the "make make problem". A phrase I learned from Rob. Tools like Autoconf and CMake exemplify the make make problem. They are abstractions on top of an underlying abstraction (Makefiles) to build a piece of software. When Makefiles (and deployment targets) become too numerous/large/cumbersome then people reach for tools like CMake which creates a higher level abstraction on top of the Makefiles. This recursion of making programs that make other programs to eventually get the actual program working is in some ways what software is all about, abstraction. But, in other ways it is a sign that the low level thing (ex Makefiles) don't offer enough abstraction to get the job done so something else needs to be used to make them useful.

To me K8s looks like a bunch of Makefiles. I'm being glib when I say that. K8s is operating at quite a high level of abstraction. But, the individual yaml files are the Makefiles. For Sirepo this resulted in: A namespace, two config maps, two secrets, a persistent volume, a persistent volume claim, one storage class, 2 deployments, 2 services, and one ingress. As I wrote out each file and started to see how everying in K8s tied together I could already feel myself wanting some tooling that generated the yaml files. Things likes having variables (ex port numbers) be shared between the files would be nice. Also, I just described only our alpha system. A copy of each one of those files (with minor differences) would be needed for beta and prod. I needed something to generate all of this. A "make make" is born.

In itself make make isn't a terrible problem. It is common and can be solved with programming. It happens whenever a generic tool like K8s is used. The creators can't possibly cover all use cases so they give a hammer and nails with some instructions and tell you to build the house. The problem with K8s is they don't give you a hammer and nails. They give you and entire consturction company but you're only allowed to communicate with the workers via smoke signals.

Aside: DevOps tooling

There seems to be an illusion around a lot of DevOps tooling that it will be useful without much work. I don't have specific citations I can point to but this is more a feeling I get reading articles and talking to peers. For example, no one would install Python and Flask and expect to have a working web app. Many hours of work are going to have to be put in using those tools to build an app. But, there is some idea one can install a tool like Ansible, write up a couple yaml files, and voila - all infrastructure is now managed and perfectly happy. The DevOps tools themselves lend themselves to this illusion. The fact that so many of them have a yaml interface adds to the idea that you won't have to "program" them.

YAML is not a programming language

Back to my point about smoke signals above: YAML is smoke signals. It is a severly lacking way to communicate anything with even remote complexity. I don't have beef with YAML itself - json, HCL, toml, it doesn't matter to me. All of them suffer from the same problem: they are configuration languages and not programming languages. The fact is, that setting up a K8s cluster and deploying an application involves some configuration but it also involves some programming. I think we are kidding our selves, and by extension making our lives harder, by thinking that it is only a configuration problem. All of the programming details can't be hidden by the tooling makers. A toolmaker must have a great deal of confidence that their tool solves every last problem if they are only going to give YAML as the language to interface with the tool.

In "Code Complete" Steve McConell advises one to program into your langauge not in your langague. By that I think he means to create the missing pieces of the programming langague that you need to bring the abstraction level up to solve your problem. The simplest example I remember from the book is to create an assert function if your language doesn't have one and you need it.

For K8s I want (at the very least) a "make maker" so I can program into K8s and avoid a lot of repetition. But, YAML is all I have to build this with. With YAML I am forever bound to the abstraction level offered by the tool. In the case of K8s that means I am doomed to problems like copying port numbers around because YAML (and K8s underneath it) doesn't really support variables. To take a real example: Our client is securing their system by whitelisting IPs. They are just copying the list of IPs between each ingress resource they deploy. They don't have something on top of all ingress resources to manage the list. It goes without saying that is quite fragile. If K8s had a programming interface instead of a configuration interface this problem would be more easily resolved.

YAML has constructs like variables but I strongly believe it is a misfeature and a code smell to start using them. Once you need variables then you need conditionals. And then loops. And then you just want a full programming language not a configuration language.

I don't think I'm the only one with this problem. Tools like cdk8s solves this exact problem. They offer a programming interface instead of a configuration interface. There are also tools like Helm and Kustomize that solve it in other ways but are still hamstrung by a YAML interface. But, when I need to use cdk8s to start creating the abstractions I need to actually make K8s useful I'm meeting my brethren in the land of JavaScript fatigue. I need tools for my tools and that 2:30 feeling is setting in. K8s better be offering me something really powerful to warrant this russian doll stacking of tools. For Sirepo I don't think it offers enough to warrant using it.

Conclusion

The make make / yaml problem doesn't make K8s bad per se. But, it just means that once you install K8s and write your first few yaml files the story has just begun. You will probably need (or at least I would certainly want) another layer of abstraction on top of it. Another layer with programatic control is key. That way I could program "into" K8s to build abstractions for my domain. At that point I think it would be a good place to pump the brakes and see if you need K8s at all. Does it really solve your problems? Could you get by with less? At 2am when the pager goes off the fewer tools I have between me and my code the better. But, maybe you do need it. Google certainly did. That's why they wrote it. But, few people are working on Google scale problems.

Epilogue

It is always easier to tear down than create. I've just taken my swings at K8s. I went from never using K8s to having our app running in about a day. Any tool that can do that I think is getting some of the abstractions right. Kubectl is cohesive and well designed. The introspection one can have into the environment is really nice. And there is a lot of polish on K8s. Polish that is nearly impossible to achieve outside of a large tool worked on by many people. There is a huge library of informative talks, blog posts, tutorials, and docs. So, all of those resources give a lot of support to move quickly and get help when you're stuck.

In an effort to do some of our own building we're considering adding support for Kubernetes to RSConf. So with just the right bits of YAML and Python we can program into Kubernetes.

LinkedIn Verification

2024-07-15T00:00:00-06:00

LinkedIn verification is an annoying anti-pattern. First, I needed to download their app. Why? Why can't I do this on my computer? I'm on an iPhone so usually the security controls are okay but I really hate downloading apps. They have in the past done things like upload people's entire contacts after being installed.

But okay I bit the bullet. Now I need to verify myself with Clear? UGH! Again, why? Now I have to read Clear's privacy policy to see what they are doing with my data? I see them at the airport and I think they have something to do with collecting biometric information to verify my identity. Are they selling it to other people? Have they ever been hacked? In the future will they ever be hacked? I'm a little gobsmacked I have to do this to use a social network.

Is verification like this necessary? I don't work at LinkedIn so I don't know what went into this decision. But, I'd really appreciate at least some acknowlegement of why they need verification at all and why this type of verification was selected. Instead, they just tell me that verified users get more traction on their account. Great, thanks for pressing me to do this.

I was talking with someone else about this and they said they tried to verify but it didn't work. So, we have an onerous system and it doesn't work. Not great.

It is a bummer when companies use their weight to press on users. LinkedIn knows people need to work. So, they know people are desperate and willing to jump through hoops to have the best shot at getting a job. So, they implement this wacko verification system and leave people in a lurch. Get a job or give up my data? Not a trade I want to make. The steps aren't worth it for me for now. But, maybe in the future I'll be more desperate...

SSH key basics

2024-07-15T00:00:00-06:00

At work we're migrating from CentOS7 to Alma Linux 9. As part of that we are separating out some pieces of our infrastructure and generally cleaning up cruft. To separate out parts of our infrastructure we are going to setup a new build server at a cloud provider. So, we'll need to migrate some of our SSH infrastructure to get everything working.

I was working on a plan for the migration with Rob and we were going over the SSH bits. Every time I need to setup SSH, which I've done quite a few times, I forget the different keys and what each one does. What is known hosts? What goes in authorized keys? Where is the public and private key? I'm embarassed to even be writing that out. It isn't very complicated. But whatever the cache invalidation strategy is in my head it keeps removing SSH knowledge before the next time I need it. So, this is a very basic article of the bare minimum of knowledge needed to work with SSH keys. Hopefully I can at least remember I wrote this article and refer back to it next time I need this information.

Public/Private Keypair

When people say "SSH keys" this is usually what they are reffereing to. The keys clients use to authenticate.

Private Key (ex. `~/.ssh/id_<algo>`)

The private key lives on a client's machine and is usually protected by a password. This key is a secret. It is able to decrypt messages encrypted by the public key.

Public Key (ex .`~/.ssh/id_<algo>.pub`)

This key lives on the server one wants to authenticate with. It is not a secret.

Other words to remember: - public key authentication: Another way to describe this auth mechanism. - Assymetric encryption: One half public and one half private.

Authorized keys (`~/.ssh/authorized_keys`)

This file contains the public keys of all clients that can authenticate with that user/server (they live under a users $HOME).

Host Keys (ex. `/etc/ssh/ssh_host_<algo>` and `/etc/ssh/ssh_host_<algo>.pub`)

The host key identifies a server. It should be unique to every server. When setting up an SSH connection the server exchanges this key with the client. The client validates this key against the fingerprint it has for it in ~/.ssh/known_hosts. If it is different the user is notified. If it is the first time the client is connecting then the user must have some alternate means of validating that the key is correct.

Known hosts (ex. `~/.ssh/known_hosts`)

Contains a fingerprint of the public keys of all hosts the client has authenticated with.

There is also an ip/hostname that is stored/checked (maybe).

Baisc steps of setting up an SSH connection

Sever and client negotiate encryption for the session and setup symmetric encryption keys they will use to secure the conneciton.
As part of step 1 the server provides it's public host key which the client checks against known_hosts.
The server and client then do a dance using the ssh keys, authorized_keys, and session key to authenticate the client.
Use the encrypted tunnel!

Application CLIs

2024-06-30T00:00:00-06:00

This article made its way to the front page of Hacker News last week. One of the points the author makes is on miscellaneous tooling. Specifically, the benefit of having a place to put utility shell scripts from the get-go of the project.

Benefits of utility shell scripts

I agree with the author. Having a structure for writing utilities makes it easier to write and share them among the team. These can be for things like fundamental features (ex. starting the application) to rarely run scripts used just for development. Creating a structure for the whole team to share them makes everyone's life easier.

Codifying the utility shell script concept (pkclis)

I think there is something even better than just having a collection of shell scripts. Something I'm calling "application CLIs" for lack of a better term.

At work, we have a framework we use to initialize all new Python projects. It is called projex. This is a scaffolding tool like yeoman generator. It sets out the initial bones of the application. Adds a README, LICENSE, GitHub workflows, test and source directories, etc.

One of the important features it creates, that I think is an evolution beyond just shell scripts, is a concept called "pkcli". Pkclis are CLIs that one can write in Python (the language of our app) and through a small bit of infrastructure become CLIs that anyone who has installed the application can use. These are the "application CLIs" for our projects. Projex itself is actually a pkcli.

Benefit: In the native language

Shell scripts certainly have their place. Even though we have application CLIs, we still write shell scripts. But, something you can't do with shell scripts is write them in the native language of your application (assuming your app isn't written in a shell programming language).

By writing CLIs in the language of the project we can use all of the features of the project. For example, we have an application CLI that deletes users from our system. Someone could certainly write this utility as a shell script. But, it's trivial to write this in the language of the application by just using the APIs already in our application for managing users. For example, we don't have to invent a new way of finding the database in a shell script. Our application needs the database so we already have an API for finding it. In addition, if we want to add a user interface for deleting users we can share the code between the two modes of interaction. This creates a more explicit coupling between the application and the CLI. This, in turn, makes testing and future changes easier to support and more likely to be correct.

Benefit: Fewer languages

Writing CLIs in the language of your application comes along with the normal benefits of using fewer languages. Bill mentioned in his article that he learned how to write shell scripts from a colleague who set up the shell scripts in his project. Everyone on our team knows some shell programming language (Bash for us). But, they know more Python. By writing CLIs in the language of our application we reduce the amount of things people need to know. Likewise, we get to use all of our existing infrastructure for free. We can do things like test our application CLIs alongside our regular application tests. We can use the same formatter and linter our application uses. We can use the same documentation generator to build docs from comments in the code. We can also use the features of our IDEs to navigate and refactor code in just one language. By writing CLIs in the language of the application we reduce the barrier to entry for writing CLIs and get to share in all of the infrastructure/tooling we have established in our application.

Benefit: Availability

Shell scripts can be hard to distribute. How do you update your $PATH so all scripts in all of your projects are available globally on all systems where you might want to run them? At what level of portability do you write them to make sure different systems can execute them? These are solvable problems but application CLIs solve them without much thought.

Using Python's pyproject.toml we can easily define CLIs that then become globally available. Anywhere we're running our application, which in my experience is everywhere we want to run the application CLI, already has the runtime installed. For us that means Python and all of the needed dependencies. In terms of portability, Python handles this for us. Anywhere that can run Python can run our CLIs.

Benefit: Dependencies

The shell programming language I'm most familiar with is Bash. So, this may only be relevant to it. But, it doesn't have great support for managing dependencies. This is related to availability above. For example, if one wants to add a testing framework to their Bash scripts they may use bats. It provides a myriad of ways to install. None of them are particularly friendly. We could have documentation that tells users what they need to install. Or roll our own Bash dependency management system that we'd have to make portable to different systems. Neither docs nor our own system are great solutions. Python's dependency situation is a mess. But, at the very least it provides files that the built-in tools in Python know how to use to manage dependencies. So, someone new to our application just has to get the application installed and then they can use our app and all of our CLIs. By using the language of our application we get good enough dependency management built in.

Benefit: Better namespacing

Another benefit is namespacing. One must be careful in shell scripts to create namespaces without conflicts. In our application CLIs, this comes easier. The first part of all of our application CLIs is the name of the project. Then the second part is the name of the module. And finally, the third part is the name of the function in the module. So sirepo admin delete_user is in the project sirepo, in the module admin.py, and runs the delete_user function. This can be done in shell scripts. Just prepend the application/module name to the name of every function. But, it can devolve into long/messy names and make integrating different projects a challenge.

Benefit: Better arg handling

I'm not a Bash wizard, so take this with a grain of salt, but I find parsing args in Bash to be a pain. The top hit for "how to parse args in Bash" is a classic Bash answer. Conflicting info. Confusion over portability. Many caveats. New ways that supposedly fix all of the problems. Again, all solvable problems. But, one has to create infrastructure and knowledge among the team to make it happen. With our pkclis, the details are handled for free (mostly) by a well-supported third-party library. From the perspective of someone writing an application CLI, they just have to understand args and kwargs in a fairly friendly way. Anything that is a *arg becomes a positional argument. Anything that is a **kwarg becomes an optional flag that you can supply a value for. We haven't run into much friction with people learning that.

As always, there is no free lunch. We've run into problems with our arg parsing. But, the fix was in one place and fairly easy to figure out.

Shell scripts are still used

Even with application CLIs, there is still a place for shell scripts. For example, we use a shell script that configures and starts our application. Setting up an environment and managing certain parts of a Linux system are just easier in a shell script. But, our startup shell script calls our application CLIs to actually start the application. The application CLIs provide a nice interface to bridge the gap between the shell scripts and the application. In other cases, we only use shell scripts. Sometimes it is just the best tool for the job.

Conclusion

Application CLIs are not unique to the company I work for. WordPress has wp-cli, Django has django-admin, and Ruby has a comprehensive suite of CLIs. Rob, the CTO of my company, brought the idea of application CLIs from a previous framework he built. But, I think the gospel of application CLIs could still be spread further.

There's nothing inherently wrong with shell scripts. In the right hands, they can be just as powerful, offering their own unique benefits. But I've found success with application CLIs and will bring them into the codebases I work on in the future.

A spin on Windows

2024-04-20T00:00:00-06:00

I haven't spent much time using Windows. My personal laptop is a Dell XPS running Ubuntu and at work my team uses Macs (ssh'ing into Fedora VMs for development). The longest stretch of time I spent using Windows was eight months or so at my first job. I was given a laptop running Windows 10 and it took me 8 months to convince them to give me a Mac. Even then I spent most of my time in the Windows Subsystem for Linux. With just this little taste of Windows, my feeling was I'd like to avoid it if possible. For web browsing and other general computing it is fine. But, for any development it is a world I'd rather avoid. Not so much for the OS which I thought was usable. But, because I didn't want to get totally immersed in the Windows world.

I follow along with DHH's blog*. He has been fighting with Apple trying to get his company's calendar app released. As part of his battle he's made the commendable decision to try to "put his money where his mouth is" and to get off of Apple products. As part of that process he has tried out Windows and has had good things to say about it. That made me a tiny bit curious to try out Windows again now that I have some more years under my belt. Maybe I'd like it too?

This past month I had a chance to do that. We are setting up JumpCloud for MDM at our company so I've been trying out provisioning Windows devices (some folks at the company use Windows). I haven't done any development on the machines but just some basic clicking around. After this little bit of time I can say unequivocally that I won't be switching any time soon. Windows is one giant ad and my experience as a user suffered from it.

Here is my experience: When I opened my Windows machine (running Windows 11 Pro) I was greeted with the normal setup your computer flow. Connect to the internet. Enter my name. Then came the first bit of ad junk. I had to enter Microsoft account credentials to get to the next step. There are ways around this but they aren't just a click away so I conceded and set up an account. Then I was presented with a "Free offer for Microsoft 365". No, thanks. Then it was "Free offer for Xbox game pass". I'll pass. Finally, into the desktop. In Windows 10 I remember the Windows start button was on the bottom left so I instinctively clicked there. I was greeted with some half broken mess of widgets showing me news articles and stock prices. How delightful. Looks like the Windows start button is now center aligned (copying Mac dock?). I clicked it and was able to open Edge. But, I had to click through three windows before I could actually use the browser. Each one of them asked me some form of what data I wanted to share with god knows who. The final one gave me a chuckle. It read "do you want to make your Windows experience better?". Yes! That would be great. But, I'm sure that by "better" they mean better for them and worse for me. So, I clicked no. It would be hard for my experience so far to get worse so I figured "no" couldn't be that disastrous. Finally I was just about to sign into the JumpCloud console when a dialog box popped up asking me if I wanted to secure my computer with McAfee. I hadn't seen that since the library computers in middle school. Once again, no thanks. Now, I was able to use the browser. But, the McAfee popup kept appearing as well as a few guest spots from Xbox Game Pass.

After navigating the sea of ads/popups I was left dismayed. I imagine there is a way to turn them all off but I wish off was the default. I know Apple is no shining star but at least I can click "setup later" and "later" is on my watch, not theirs. I have to remove things like AppleTV from the dock when I get a new machine but they don't find their way back. And there are no popups from companies like McAfee following me around. I encourage DHH to continue his search but I won't be leaving my Mac for Windows anytime soon.

*I could write another post on his blog and the ideas he shares. I have no plans to but just know that just because I read it doesn't mean I agree with him.

P.S. While updating my site with this post I was greeted with this ›Error: request to https://api.netlify.com/<snip> failed, reason: Hostname/IP does not match certificate's altnames: Host: api.netlify.com. is not in the cert's ›altnames: DNS:*.safezone.mcafee.com, DNS:safezone.mcafee.com My router from CenturyLink has Mcafee garbage too. I've turned it off no less than a dozen times...

Carbide Scraper

2024-02-16T00:00:00-07:00

I've been rebuilding the the motor on my snowmobile. The only part of the process I dread is dealing with old gaskets. After 18 years of use the gaskets on this motor were totally baked on. Common advice is to carefully use a razor blade to remove the old gasket material. But, under the gaskets is soft aluminum. If you knick the aluminum while you're getting the gasket off you will create a surface that the new gasket will have a hard time sealing. I have never removed a gasket without knicking the surface. Some people reccomend a tool like a scouring pad (by hand or on a rotary tool). But, this will fling gasket material and little metal shards from the scouring pad all over the motor. Possibly into the crank of the motor where the debris will wreak havoc on all of the spinning parts.

After desperately searching I came across a tool that has revolutionized the process for me. A carbide scraper. The always reliable mechanic porn site Garage Journal is what tipped me off to this tool. I want to buy an alphorn and yell about carbide scrapers from the mountaintops. They are amazing! It is shocking to me that any other tool is ever reccomended.

The scraper has a perfectly flat blade that doesn't nick the surface. You hardly need to apply any pressure and the gasket is easily scraped away.

I bought the cheapo version of the tool from O'reily. Next time I'll buy the USA made Super Scraper. But, if you're in a time crunch like I was the Titan scraper works wonders.

The tool worked so well I spent the afternoon going over every gasket surface I could find. If you ever find yourself removing gaskets (or I'm guessing many other stuck on substances) don't even think about reaching for a razor blade. Do yourself a favor and get a carbide scraper.

Bangle.js Hackable Smartwatch

2023-12-28T00:00:00-07:00

Two years ago I did the Grand Traverse ski race. It is a 40 mile race between Crested Butte and Aspen through the Elk Mountains. I wouldn't exactly say I was racing. More just hoping to survive to the finish line. As part of the preperation for the race I followed an 80/20 Endurance 50 mile ultramarathon training plan.

This plan involves workouts that look something like:

5:00 in Zone 1
5:00 in Zone 2
5 x (0:30 in Zone 3/0:20 in Zone 4/0:10 in Zone 5)
5:00 in Zone 1
5 x (0:30 in Zone 3/0:20 in Zone 4/0:10 in Zone 5)
5:00 in Zone 1
5 x (0:30 in Zone 3/0:20 in Zone 4/0:10 in Zone 5)
10:00 in Zone 1

That is one of the more involved workouts. The specifics aren't important but what is important is there are 50 steps in there and some of the steps are as short as 10 seconds. So to follow along one needs some sort of system. I ended up doing the workouts just by bouncing between the Training Peaks app where the workout is stored, Strava where I could see my heart rate, and the iOS timer app. This would result in me staring at my phone the whole run which isn't very fun. So, near the end of my plan I went in search of a better system.

I thought the simplest solution would be to buy a Garmin smartwatch (or similar). They know about workouts like this and you can download the workout straight to the watch. But, I didn't really want to spend a few hundred dollars on a watch that would sit in a drawer after my five month traning plan was done. Also, from the people I've talked to the Garmin software is nothing amazing.

During my search for something better I stumbled upon a pretty cool solution. The Bangle.js. The Bangle.js is "the world's first open source hackable smartwatch." You write code in JavaScript for it and it only costs about $90. Perfect!

I finished my traning plan (and the race) before I had time to actually program the watch. It then sat in a drawer for two years before I got the idea to do the Grand Traverse again. So, recently I pulled out the old training plan and dusted off my watch.

Below is some information on the app I built to manage my training plan. I think the content is relevant to many people who want to program on the Bangle.js even if the app you want is for something else.

First some eye candy.

That is the main face of the watch while running my app. It has time left in the current stage, current heart rate, target heart rate range, and the name of the target heart rate zone. It isn't that amazing but getting there involved some twists and turns.

If you prefer reading code over prose here is a link to my app. There is much to improve but it works for now.

Basic development

The simplest way to program the Bangle.js is to do so in the web IDE. That is the way they tell you to program in the docs. But, I like programming in Emacs so a web IDE isn't going to fly.

Instead I connected to the watch using the espruino cli. Connection is fairly simple, you can do espruino -d Bangle --watch app.js. -d Bangle connects to any nearby bluetooth device with "Bangle" in the name. --watch app.js uploads my app and watches for changes. Using this cli I was able to program in Emacs and on every save my code was uploaded to the watch.

Bluetooth connection

Dealing with bluetooth is always a pain. From just being a consumer trying to get my headphones connected to my laptop to actually programming it on the watch I loathe interacting with bluetooth. The APIs for dealing with bluetooth on the Bangle.js are decent but bluetooth is just a finicky protocol no matter what.

I need bluetooth connectivity in my app to connect to an external heart rate monitor and to receive push notifications. The Bangle.js comes with a heart rate monitor under the watch face but it is pretty innacurate with any movement. So, I wanted to connect to a heart rate monitor I wear on my chest.

This is the main bit of bluetooth code in my app. Like most bluetooth code once you get it working it isn't that many lines but getting there takes some time. Also, it is all wrapped in an infinite retry because bleutooth connections never work on the first try and tend to drop after a while.

One issue with programming with bluetooth on the Bangle.js is that you can only connect to one peripheral and one "central" device at a time. So, I can connect to the debugger and to my heart rate monitor but nothing else. You'll see how this becomes a problem in the next section.

Getting the workout from my iPhone to the watch

I needed some way of loading my workout onto the watch. While developing I just had it hardcoded in the app. But, in the "production" version I settled on a nice Rube Goldberg contraption.

I first download the day's workout from the iOS Training Peaks app on my phone. This comes in a json file. I then use an iOS shortcuts flow that I wrote which let's me select the file, parse it, and send a push notification with the parsed data. My watch is paired with my phone so it receives the push notification and gets the workout from the body of the notification.

All in all, this seems to work fine. Getting the connection between my phone and the watch working and creating a small enough noticication so it wouldn't get clipped by iOS proved a bit challenging. I think the iOS push notification size limit is 4KB but I'm not really sure. I was bumping against whatever the limit was until I made the message as small as possible. This involved doing point and click programming in the iOS shortcuts app which was a chore but I was happy to have some way of writing a little program on my phone without having to resort to writing a full app.

The hardest part of this entire process is there is no good way to debug. As I said above you can only be connected to one peripheral and one "central" device at a time. My heart rate monitor counts as a peripheral. So, for the "central" connection I had to decide between being connected to the debugger or to my phone to receive a push notification. This resulted in numerous cycles of connecting to the deugger, uploading my code, disconnecting, fighting to connect my phone and the watch, sending the notification, having the app not work, and having no clue why.

I found a few solutions that may help future Bangle.js users.

Write to a file instead of writing to stdout. You can write to a file quite easily so I would do that as a means to figure out what was going on.
Similar to the first hint you can write all uncaught exceptions to a file. I was quite relieved when I found that burried down in the docs.
Use the Light Blue app for connecting to the watch. Once the watch is paired the iPhone will connect to it again in the future. But, for whatever reason BLE devices aren't listed when trying to pair for the first time.
There is more advice in the README.

After more time than I care to admit of carefully reading code and fixing bugs I still wasn't able to get my code working. But, the above tools finally gave me enough of a view into the errors that I could finish the app.

Final thoughts

I'm really happy with the Bangle.js. It is a super impressive bit of tech and I think it is almost entirely built by one guy, Gordon Williams. He is a very active coder and is very active on the forums. He's built quite an ecosystem.

If you're curious about hacking on a smart watch I highly reccomend the Bangle.js. I have the first version of the device but there is a newer version out with better specs. I hope to upgrade one of these days.

If you happen to already have a Bangle.js you can get my app from here. It has my heart rate zones hard-coded but I hope to fix that one of these days. You'll also need the ios integration app, my iOS shortcut and a workout loaded in Training Peaks. I haven't put any thought into whether or not this app will work for anyone besides me so buyer beware!

E Street Garage

2023-12-04T00:00:00-07:00

I'm on the cusp between being a Millenial and Gen-Z. I remember 9/11 which I think is a reasonable distinction for being a Millenial but it is one of my earlier memories. Today I let my Gen-Z shine by becoming a content creator. I posted my first video to YouTube*. But the truth is, if I had to put my videos on a generational scale from Charlie Chaplin to Troye Sivan I would put them squarely in "hey boomer, turn your phone sideways when you're filming."

My desire to create a YouTube channel is twofold. First, I've received thousands of hours of eduction from strangers through YouTube. I feel it is time for me to give back in whatever small weird way I can. Second, I'm already taking videos of most of the things I work on. I have an exceptional talent for being able to take things apart and not being able to put them back together. Recording videos helps me figure out what things looked like as they came apart. So, between feeling a debt to the person with 3 views on their video detailing how to replace the cabin air filter on a 2004 Honda Odyssey and already having videos of me doing equally obscure work I figured it was time to make a channel of my own.

These videos aren't intended to be well shot, lit, edited, or entertaining to watch. They are supposed to be easy for me to create and help someone who is in a jam trying to finish a project.

Here is my first video. It shows how to replace the secondary clutch rollers on a 2006 Arctic Cat M7 snowmobile.

*I'll be honest I don't even know if Gen-Z uses YouTube. For all I know I'm just solidifying my boomer creds.

Moving in vi

2023-06-28T00:00:00-06:00

I recently paired with a colleague who is learning vi. While watching him navigate code, I noticed he almost exclusively used the arrow keys. When I started using vi, I read numerous times that one should avoid using the arrow keys. So, I relayed this advice to him. He, quite rightly, asked "why?". Of course, I should've considered why I was giving this advice before blurting it out. I dregdged up some memory of the articles I had read and explained that it is faster to keep one's fingers on the home row than to move to the arrow keys. However, after making this point, I realized that wasn't the complete explanation.

While it might be faster to keep your fingers on the home row, there is a deeper reason to avoid the arrow keys. Utilizing the arrow keys (and hjkl, for that matter) is the slowest possible way to navigate in vi. Moreover, observing him work with the arrow keys made me realize that this approach often kept him in insert mode rather than inserting text and instinctively switching back to normal mode. Being in an "insert mode mindset" meant that he wasn't thinking about other, faster ways to navigate code. Techniques like searching, moving by word, finding a character, etc., are all only possible in normal mode.

While writing this article, I did a little research to see what others had to say about why the arrow keys should be avoided. Most articles I read stated that it's slow to move from the home row to the arrow keys and that moving one char/line at a time is tedious. However, some people only mentioned the slowness of moving away from the home row (for example). But, the issue of moving away from the home row is not that significant. Although it's slower, it's not nearly as time-consuming as navigating one char/line at a time.

I also wondered why recalling information about moving more than one character or line at a time took me a while. My guess is that when I read the vi articles instructing me to avoid the arrow keys, I had yet to master the more powerful navigation techniques. So, while I understood the speed advantage of not leaving the home row, I lacked the muscle memory to make other forms of navigation faster. As a result, this piece of information didn't stick in my mind.

I must admit that, to this day, I am no navigation master myself. I don't use the arrow keys, but I commit a related sin. I increase the key repeat speed on my machine so that navigating by hjkl doesn't feel so sluggish. Although I use some combinations other than hjkl to move, I'm confident the key repeat crutch is holding me back. Perhaps it's time to disable it...

Using ChatGPT

2023-03-31T00:00:00-06:00

When GitHub Copilot came out, I joined the technology preview hoping it would help me be more productive at work. The result was extremely underwhelming. The code it produced was buggy and didn't meet any of the coding standards we use. It seemed more risk than reward, with the subtle bugs it would introduce. So, after a few hours of using it, I gave up.

Then, ChatGPT came out. It seemed like a similar hype cycle. But, I am genuinely interested in finding ways to be more productive. I'm also curious if one of these tools is going to take my job or become my boss. So, I messed around with it some. I thought it was better than Copilot but still not amazing. It seemed way too eager to give me wrong answers, apologize for them, and then give me the same wrong answers. So, like Copilot, I gave up after a few hours.

This morning, I read this article about one person's path to increased productivity with ChatGPT. The specific use case outlined for ChatGPT didn't strike me as particularly interesting. If I'm being totally honest, it seemed like most of the hello world examples I see. Cool that it helped this one person write some little script. But, not really going to move the needle for me getting my job done. For no apparent reason, though, this article lit a fairly large spark in me to try again to see if I could make use of ChatGPT.

Today at work, I spent a lot of time on this PR. We are trying to get NFS mounts working in a Fedora 36 VM running on a MacOS host. The meat of the work has been taking place in this file. It is a shell script to generate a Vagrantfile (a file which defines the configuration for a VM), which will create the VM and then do some provisioning of the machine. There are lots of little details to hold in one's head and many different cases all competing for my attention.

To add to the confusion, it uses bash parameter expansion heavily. For the life of me, I can't remember what ${param:-1} vs ${param-1} vs ${param:+1} etc., all mean. Every time I interact with parameter expansion, I need to visit the docs to remind myself what each one of the expansions does, and I usually then end up writing a little note to summarize everything.

Today, I decided to plug my questions into ChatGPT. I said:

Can you explain bash parameter expansion? Specifically,
${parameter:-word} ${parameter:-+word}

You'll notice right off the bat, I have a typo. The last example should be ${parameter:+word}. ChatGPT didn't mind and gave me a very useful answer.

Bash parameter expansion is a powerful feature in shell scripting that allows
you to manipulate and transform variables or parameters in various ways. It
helps in enhancing the flexibility of scripts by providing ways to use default
values, substitutions, string manipulation, and more.

In your question, you've mentioned two types of parameter expansions:

    ${parameter:-word}
    ${parameter:+word}

Let's dive into each of these:

    ${parameter:-word} - This is called the default value parameter expansion.
    It expands to the value of 'parameter' if it is set and not null; otherwise,
    it expands to the value of 'word'. This is useful when you want to provide
    a default value for a variable that may not be set or may be empty.

Example:

name="John"
greeting="Hello, ${name:-User}!"
echo $greeting

This will output "Hello, John!" since the 'name' variable is set to "John".
If 'name' were unset or empty, it would output "Hello, User!".

    ${parameter:+word} - This is called the alternate value parameter
    expansion. It expands to the value of 'word' if 'parameter' is set and not
    null; otherwise, it expands to nothing. This is useful when you want to
    conditionally insert a value or perform an operation based on the existence of a variable.

Example:

name="John"
greeting="Hello${name:+, $name}!"
echo $greeting

This will output "Hello, John!" since the 'name' variable is set to "John". If
'name' were unset or empty, it would output "Hello!".

These parameter expansions are just a couple of examples of the many ways you
can manipulate variables in Bash scripts. They help in creating more robust and
versatile scripts by accounting for different situations and handling variables
more effectively.

From 1000 feet above, the docs and ChatGPT look to give the same answer. But some of the subtle wording in the ChatGPT response finally helped me hold in my head what each expansion does. What really did it for me was "this is called the default value parameter expansion" and "this is called the alternate parameter expansion". For whatever reason, "default" vs "alternate" caused things to click.

I was pretty impressed with ChatGPT at this point! I spent the rest of the day forgetting how expansion worked or running into a slightly different expansion and getting ChatGPT to help me. Like a perfect little helper, it held context all day long as I asked it questions.

The funniest part was, like a coworker helping, it even reminded me that I was dumb and it had already answered the question before:

In the given expression ${vagrant_dev_no_nfs_src:+1}, the parameter expansion
used is the alternate value parameter expansion, which we have covered earlier
as ${parameter:+word}.

So, while not earth-shattering and not doing my whole job for me, I actually found some real value in using ChatGPT to help me along. It won't take my job, but I definitely plan on having a tab with it running going forward.

P.S. If you're from the future and ChatGPT doesn't exist, send me a note so I can reminisce.

P.P.S. In a recent article, I was told by a friend that I could use more commas. I don't have the faintest idea where to put a comma when I write. But, I fed it through ChatGPT and it produced an article with at least 2x the number of commas, so from now on, all of my articles will have more comma baby.

ImageMagick

2023-03-27T00:00:00-06:00

I've written about ImageMagick before. It is a very useful tool with a great interface. It seems that whatever I want to do to an image is just one command away.

I used ImageMagick again yesterday when I wrote my first post with images. I needed to resize the images and combine two together. In an effort to not have to go searching every time I want to do some image manipulation I'm going to keep this living cheatsheet of commands I've found useful.

Convert HEIC to JPG

See my previous post.

Resize an image

convert original.jpg -resize 1000x1000! resized.jpg

Combine two images top-to-bottom

convert -append top.jpg bottom.jpg combo.jpg

Combine two images side-by-side

convert +append left.jpg right.jpg combo.jpg

Make a slideshow

mogrify -resize 800x600 -background black -gravity center -extent 800x600 *jpg # Make images the same size mogrify -auto-orient *jpg # Make the images all the same orientation convert -delay 800 -loop 0 *jpg slideshow.gif # Make a gif slideshow ffmpeg -i slideshow.gif slideshow.mp4 # Convert gif to mp4

Strip identifying info

magick mogrify -strip *.jpg

Dirt Bike Fuel Injector Cleaner

2023-03-26T00:00:00-06:00

This winter, I set up my 2019 KTM 500 EXC-F as a snowbike. Riding and working on it has been considerably more fun than I could have imagined.

The past few times out, it has been bogging around 90% throttle and above. I'm new to working on dirt bikes, so I don't really know the cause. But, I've read that bogging near wide-open throttle can be an indication of fuel starvation. I have already replaced the fuel filter in the gas tank on this bike, which is a common weak point. So, today I decided to clean the fuel filter outside of the tank and clean the fuel injector.

Reading online, it seems that the two cleaning strategies are either to send your fuel injector to a professional to have it cleaned or to buy a tool to do it yourself. Part of the joy of owning this bike has been working on it, so I went the DIY route.

Normally, I don't mind spending money for the right tool for the job. "Buy once, cry once" is cliché but works. Although, I have been known to visit Harbor Freight to limit the number of tears I shed. From my brief research, the go-to (and maybe only?) tool for cleaning dirt bike injectors is the Motion-Pro Fuel Injector Cleaner Kit. It retails for about $160, and I would have to wait for it to be shipped to my house. Blugh, expensive for what amounts to a 9-volt battery, switch, and a holder to direct the cleaner into the fuel injector. I was hoping to solve this problem today. So, I went the DIY-DIY route and built one myself.

A perfectly hideous homebrew tool. The 22-18 AWG butt splice connectors slipped right over the prongs in the injector. They don't fit snugly, but they hang on well enough to get the job done. I used a 9-volt battery because that is what the Motion-Pro tool uses. They say not to hold the injector open for more than 3 seconds, presumably to not burn it out. The injector is used to being opened and closed rapidly, not being held open. The fuel system operates around 50 psi, so I set the regulator on my compressor to that. When power is applied to the injector, it opens and you can blow cleaner/air through it. I backflushed first and then blew out the direction fuel travels.

The tool worked like a charm, and I was able to flush some gunk out of the injector. In the process, I realized my fuel hoses were crumbling and definitely introducing little bits of rubber hose to the fuel. Of course, all downstream of the two fuel filters in the system.

I'll probably add a connector for the 9-volt and a switch to make things easier. But, I will miss the leopard duct tape.

There you go. If you're cheap, impatient, or both, like me, this will get the job done for a fraction of the cost of the professional tool.

For the programmers who read this article: That tool is a slapped-together shell script. I wouldn't hand it to someone else and tell them to use it, but it worked for me, and if you're careful, maybe it will for you too.

Docker top

2023-01-27T00:00:00-07:00

We run an internal GPU server at work. The scientists share the resource and access them through Jupyter servers. Sometimes we run into issues where GPUs report being in use even though users aren't actively using them. This is caused by Jupyter kernels left idling that once upon a time were using the GPU.

Running nvidia-smi will tell you the PID of each process and what GPU it is using. You can then feed that into the script below which will tell you which container is running that PID.

Someday I'll figure out if there is a fix to idling kernels holding resources. But for now this will do.

#!/bin/bash
set -eou pipefail

pid=$1
for c in $(docker ps --format '{{.Names}}'); do
    if docker container top "$c" '-o pid' | grep "$pid" > /dev/null 2>&1; then
        echo "$c"
        break
    fi
done

Converting HEIC image to JPG

2022-11-08T00:00:00-07:00

This isn't new information. I learned how to do it from here. But, I'm recording this because I have trouble finding that link. And maybe someone else will come across this and it will be useful.

#! /bin/bash
set -eou pipefail
shopt -s nocaseglob

b=heic-bkp
mkdir "$b"
for f in *.heic; do
  convert "$f" "$(basename $f .HEIC)".jpg
  mv "$f" "$b"
done

Linux threads and PIDs

2022-09-17T00:00:00-06:00

At our weekly software team meeting we were discussing our progress towards removing Flask in favor of Tornado. As part of the dicussion we were talking about how we need to be careful in Tornado because no "work" can happen in the main server process. In Flask work could happen because multiple threads were running. So if one thread was blocked doing some work other threads could still answer requests. Tornado is single threaded with cooperative multitasking so that won't work. Tasks need to cooperatively monitor other processes doing the actual work. In passing the CTO mentioned that each thread has a unique PID (or at least, that's what I heard him say). That surprised me. I assumed that threads shared a PID.

I did a bit of research and it looks like things have changed over time. Specifically pre 2003 (<2.6 kernel) the thread implementation was called LinuxThreads. Each LinuxThread had a separate PID. In >=2.6 the thread implementation changed to Native Posix Thread Library (NPTL). This is an implementation of threads that aligns more closely with the POSIX threads standard. In pthreads they share the same PID.

To add to the confusion the definition of a PID is hard to nail down. It seems POSIX and Linux have different meanings for the term PID. In addition, the kernel meaning of a PID and user meaning of a PID aren't the same. See the notes section of the getpid man page for more discussion. In programs like top it may display one of many different identifiers under the PID column (see section 3.a.19 PID).

To add further to the confusion it seems threads are mostly implemented in user space so libc implementations besides glibc (ex musl) may do things differently.

Going one level deeper: The proc filesystem (procfs)

2022-09-16T00:00:00-06:00

At work our CTO has started doing "Software Life Stories". The idea is he will bring in someone who has had a career as a programmer to talk to us over lunch. For our first installment he brought in his friend Tom Cargill.

Tom has been a programmer for longer than I've been alive and had a wealth of information to share. One question I asked him was if he could point to specific things he did that made him a better programmer One of the answers he gave was that he understood what he was working on at one level deeper than the current problem he was solving. For example, he said while working on debuggers he understood the machine code beneath the language he was writing a debugger for.

As part of Tom's career he worked at the Bell Labs. While there he helped invent the proc filesystem. He mentioned this in passing saying that it helped him build a debgger. I had heard of proc before but didn't know anything about it. A few days later I was checking in on the health of our servers and applications. The script we use to do that had identified a problem. To understand what might be going on I first needed to understand what the script was doing. While reading I saw "/proc/$pid/cwd". Shoot there's proc again. Time to take Tom's advice and go one level deeper to understand what the heck it is. Or at the very least scratch the surface of the next level.

This article seems like a good overview of procfs. Take that from someone who doesn't know anything about procfs so everything in there could be a lie. The high-level view is it dumps kernel datastrcutures to files so applications can learn more about what is going on in the system. So, the "/proc/$pid/$cwd" above is getting the current working directory of $pid. Interesting! Systems that know about themselves and you can ask questions of are one's I like working on.

I messed around some with the examples in the article. This one jumped out to me:

~$ cat
# In another terminal window
~$ pgrep -x cat
187843
~$ echo foo > /proc/187843/fd/0
# foo will appear in the other terminal window

Cool. So, we can write to fd/0 (STDIN) of the process. Since that process is cat it writes out to STDOUT whatever is input on STDIN. I wonder if we can read from STDOUT?

~$ tail -f /proc/187843/fd/1
# In another terminal window
~$ echo foo > /proc/187843/fd/0
# crickets in the window running tail...

Hmm, nothing is ever quite as easy as I hope it will be. I don't understand why I can't just listen to STDOUT but it has something to do with the fact that cat is connected to a tty. In an effort to not go too deep I'll save that rabbit hole for another day.

As I said earlier Tom mentioned that that he helped to invent this filesystem. I looked up the paper about the invention of /proc. It is shockingly readable and short for a technical paper so I definitely think it is worth a read.

If you happen to be trying to run the examples on your own machine you may have found that /proc doesn't exist. On some flavors of Unix there is no /proc. This got me curious about the reasons why. It seems that it can be unsafe. FreeBSD had it at one point but removed it. There is much hand waving aobut the reasons why it was removed. My takeway from that is that to build anything mission critical on top of proc I'll need to understand it a bit more and some of the race conditions that can crop up when accessing the files in there*.

That's all for today. I'll call it 5% of a full level deeper.

* While looking into procfs I came across a really neat YouTube video of how file descriptors can be used to prevent race conditions when accessing files. I rarely watch YouTube videos about coding but that video is worth a watch and makes me want to see more of his videos.

All the small things

2022-02-05T00:00:00-07:00

As I cruised the internet last night I came across this Hacker News thread about rewriting an entire application. At work we've been having talks about rewriting our frontend code. It is written in AngularJS which is awful and reached end of life on December 31, 2021

In the thread a commenter mentions the talk "All The Small Things" by Sandi Metz. I figured I'd do what I normally do: start the talk, get bored 2 minutes in, and then move on to other content. I ended up watching the talk the whole way through and then waking up in the morning and writing this post I enjoyed it so much. Sandi is an excellent speaker and very clearly describes the concepts she's talking about. I highly recommend watching the talk.

Seams

The talk focuses quite a bit on object oriented design. At work we don't use many objects. But, I think the concepts the talk focuses on through objects are concepts we use. These concepts have proven to be life savers when complexity needs to be added to our system (ex dynamic dispatch).

One concept Sandi brings up is “seams”. My interpretation of this is they are places where you can get a handle on what needs to happen, dispatch, and then go to one small area of the code to handle the complexity for just the thing you care about. The opposite is what the code in the talk starts as, complicated branching where all of the complexity for everything is mixed together.

Hourglass

I've come across this concept before and thought of it like an hourglass. For example, at work all functions that begin with def api_ are our API endpoints. In our Flask server (server.py and job_api.py) I think of them as the top of the hourglass. They are varied and do many different things. But if you squint all of these API endpoints share some common functionality. They need security (who can access it?), they have common input (how do we parse the request to convert json to a Python dict?), and they need common resources (is the database initialized?).

At the small constricted neck of the hourglass we answer these questions. Importantly, we answer them abstractly for all API endpoints and we do it in only one place (uri_router.py). I think this is the seam.

The API requests are then dispatched (there is even a _dispatch method in uri_router) to their respective function. At that function a complex but small and isolated bit of code unique to that API happens.

Using seams/hourglass to fix problems

I've put these concepts to use before. We once had an issue where we had a bug in threaded code because we were clobbering flask.g (a thread safe datastructure). Here is the PR that fixed the bug. It is pretty hairy but we fixed many concepts along the way. The important part of that PR with regards to seams/hourglass is that we were able to add with sirepo.auth.process_request() to the _dispatch of uri_router to fix all APIs. process_request is tricky code. It handles threaded (flask) and non-threaed (tornado) environments and sets up our auth (cookie) and database (sqlite) for every request. By having all API requests go through _dispatch we had a handle (a seam or the neck of the hourglass) where we could do this.

Had we not had this seam I think the fix would have been harder to identify. Answering a question like "how do I do this for all APIs?" would have been overwhelming. Likewise, it may have lead to a solution like adding a new decorator to every def api_* which spreads more complexity throughout the codebase and introduces more areas for error (ex. forgetting to add the decorator to a new API).

How to write code with seams

Sadly, I'm going to leave you empty in one critical way as the talk left me. How does one identify code without seams, find what the seams are, and add them? I think I'm ok at adding seams. Maybe I can identify code without them (lot's of complicated logic mixed together). But, I'm not confident I know how to find what the seems are. For example, I could use uri_router to add complexity (using the seams) but I didn't write that code initially.

My hope is that by having this new mental model, "seams", maybe I'll be able to add them in the future. If you know of a good way to identify what the seams should be in an application please reach out.

Successful use of git reflog

2022-01-20T00:00:00-07:00

git reflog is one of those commands that I've tried to use many times with no success. Usually something goes horribly wrong with one of git's more advanced commands (ex rebase, cherry-pick, reset), I then search for something to recover my mistake, someone online says it is so easy with a simple reflog, I try it, make my problem worse, and eventually give up. But, a few days ago I had success!

Play-by-play

I opened a pull request and was resolving comments left on it. I fixed a group of related comments, made a commit for the changes, and started to run the tests. While the tests were running I went on to fix other comments. I stacked on two more commits. I then went to the tests and saw that they failed. My changes in the first commit since opening the pull request weren't quite right.

Here is what my commit history looked like

$ git log --abbrev-commit --pretty=oneline | head -n 5
64f97aac7 move checking for mpi to sim_data
5e95a50e3 simplify
a565074a0 mpi abort is needed; share between common-header.py.jinja and sirepo.mpi
688fd1c92 Fix #4038: Always call python parameters.py from slurm

688fd1c92 was the original commit where I opened the pull request. a565074a0 was the first commit resolving pull request comments. 5e95a50e3 and 64f97aac7 were more fixing of pull request comments.

So, I wanted to make some changes but they are relevant to 688fd1c92 since that was the commit I was running the tests for. My normal workflow for this sort of thing is to:

# make some changes
$ git stash
$ git rebase -i master
# select edit for the commit I want to edit (in this case 688fd1c92)
$ git stash pop
$ git commit --all --amend --no-edit
$ git rebase continue

I did just that and here is what I saw:

$ git status
On branch 4038-nersc-prepare-slurm
Your branch and 'origin/4038-nersc-prepare-slurm' have diverged,
and have 4 and 4 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

Shoot! I did git rebase -i master. Rebasing on master means that my original commit for the pull request, 688fd1c92, was modified from the rebase because I set the base to be master. I had broken the golden rule of rebase - never rebase on a public branch.

The correct syntax would've been git rebase -i a565074a0^. Which means rebase on the commit one before the commit I want to edit.

Ugh, now to recover from my mistake. Thankfully I hadn't pushed anything so recovery shouldn't be too bad.

Off to Google how to recover. I searched for "how to undo a git rebase". The first answer, as expected, says to use git reflog.

Here is what I saw:

$ git reflog | head
c152af107 HEAD@{0}: rebase (finish): returning to refs/heads/4038-nersc-prepare-slurm
64f97aac7 HEAD@{1}: rebase (pick): move checking for mpi to sim_data
c7e8b67af HEAD@{2}: rebase (pick): simplify
df5260920 HEAD@{3}: commit (amend): mpi abort is needed; share between common-header.py.jinja and sirepo.mpi
a56507a40 HEAD@{4}: rebase: fast-forward
688fd1c92 HEAD@{5}: rebase (start): checkout master
9fc121349 HEAD@{6}: reset: moving to HEAD
9fc121349 HEAD@{7}: commit: move checking for mpi to sim_data
5e95a50e3 HEAD@{8}: commit: simplify
a56507a40 HEAD@{9}: commit: mpi abort is needed; share between common-header.py.jinja and sirepo.mpi
688fd1c92 HEAD@{10}: commit: Fix #4038: Always call python parameters.py from slurm

Alright, I must be getting better as a programmer because this looks 10x less initmidating than the last time I remember looking at it. I see that for HEAD@{5} it says rebase (start). I probably want to move to the commit just before that. So I tried that:

$ git checkout HEAD@{6}
$ git diff

I did the git diff to make sure that things look ok. They do so I think this is where I want to be. Let's do it for real:

$ git switch -
$ git reset --hard HEAD@{6}
$ git status
On branch 4038-nersc-prepare-slurm
Your branch is behind 'origin/4038-nersc-prepare-slurm' by 2 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

A git diff confirmed that I was back to where I was before the botched rebase! The only bummer is that I lost the work that I wanted to apply to 688fd1c92. I'm sure there is a way I could've saved it but no big deal. It was easy to write again.

Editing a commit the right way

Now that git reflog saved my skin I wanted to edit the commit with my changes. Here is what I did:

# make the changes
$ git stash
$ git rebase -i 688fd1c92^ # ^ means one commit before 688fd1c92
# select edit for 688fd1c92
$ git stash pop
$ git commit --all --amend --no-edit
$ git rebase continue
$ git status
On branch 4038-nersc-prepare-slurm
Your branch is ahead of 'origin/4038-nersc-prepare-slurm' by 3 commits.
  (use "git push" to publish your local commits)

Nice, 3 commits and not in conflict with the upstream. Thanks git reflog!

What Does This Code do?

2021-12-24T00:00:00-07:00

A colleague opened a pull requests with this code in it a some months ago ago.

self.nextId = function() {
    // a uuid generator found on the interwebs
    return ([1e7]+-1e3+-4e3+-8e3+-1e11).replace(/[018]/g, c =>
        (c ^ window.crypto.getRandomValues(new Uint8Array(1))[0] & 15 >> c / 4).toString(16)
    );
};

I reviewed the code and left this comment

The word next is misleading because to me that implies some ordering and the
uuids generated are random. Also, is there a reason we need true uuids instead
of doing something like sirepo-lattice.js? If there is a reason I think adding a
comment about it in the code would be helpful.

While the comment was fine, I hope, I did not actually understand this code which is a problem since I reviewed it. So, here is my attempt to go back and figure out what the code actually does.

This is just a stream of consciousness of me solving the problem. It bounces around. For better or worse that is how I think.

Breaking down the code

There is quite a bit of this code I don't understand so let's dive right in.

I think 1e7 is scientific notation.

> 1e7
10000000

The REPL confirms that it is.

Why is 1e7 an array and what does it mean to add the other values to it?

> [1] + 2
'12'
> x = [1] + 2
'12'
> typeof x
'string'

WAT?? (If you do nothing else with this article please stop and watch this talk.)

JS is always full of surprises. Some days I kick myself wishing I could remember all of the gotchas. Other days I'm just happy if I remember to zip up my pants after going to the bathroom. On those days I can not be bothered by remembering details like this. What I do try and remember is that JS has strangeness like this. When doing any operation to two things of different types it is a good idea to slow down and make sure I get it right.

> [1e7]+-1e3+-4e3+-8e3+-1e11
'10000000-1000-4000-8000-100000000000'

Ok, we have a string of some numbers separated by dashes. Looks something like a UUID.

At this point I get distracted and think "what happens when I just search 'js uuid generate'."

The first result is, as expected, a link to StackOverflow.

The top answer mentions the UUID module

In the simple case (all we need) that module just calls crypto.randomUUID(). Why can't we just call that? Off to caniuse.com.

Ugh, we don't have the strictest requirements for supporting old browsers but if the latest Safari (15.1) does not support it then it won't work for us.

This makes me wonder how JS implementations implement it. A quick search yields the W3C spec. The description of the spec does not look much like the code we are using. I then go to the Node.js docs. Why don't the docs link to the source code? That is something ruby-doc.org really got right. Anyways, the code is not that hard to find. Again, this looks like the spec and not much like our code.

From reading the spec and looking at our code I think our code is probably taking some short cuts. Reading through the SO comments this seems to be the case. Again, I don't feel it is that big of a deal because the likelihood of a collision seems pretty small and fairly harmless (the UUID's are for one simulation for one user).

Back to the code...

'10000000-1000-4000-8000-100000000000'
> ([1e7]+-1e3+-4e3+-8e3+-1e11).replace(/[018]/g, c => console.log(c));
1
0
0
<snip>
> ([1e7]+-1e3+-4e3+-8e3+-1e11).replace(/[018]/g, c => 'e');
'eeeeeeee-eeee-4eee-eeee-eeeeeeeeeeee'

This is replacing '0', '1', and '8' with the result of the callback.

Now I need to understand the structure of a UUID.

The '4' that is not replaced is the version of the UUID spec. Version 4 means random.

Why is the '8' getting replaced though? That is the "variant" and has some meaning I don't yet fully understand. Maybe the c ^ ... does some mask operation that includes it?

There are still some missing pieces but I'm starting to see that we are mostly just replacing values in the original string with randomly generated values.

Now, let's understand some about the random generation.

> crypto.getRandomValues(new Uint8Array(1))[0]
Uncaught TypeError: crypto.getRandomValues is not a function

Gr888. Maybe my version of node is out of date? Yes, but also there is this handy comment in the code saying I can just use randomFillSync

> crypto.randomFillSync(new Uint8Array(1))[0]
124
> crypto.randomFillSync(new Uint8Array(1))[0]
91
> crypto.randomFillSync(new Uint8Array(2))
Uint8Array(2) [ 39, 53 ]

Alright, we are getting one 8 bit unsigned integer. But, now I'm starting to get scared. Putting together this whole line is going to be tricky.

(c ^ window.crypto.getRandomValues(new Uint8Array(1))[0] & 15 >> c / 4).toString(16)

There is more bitwise operations going on than I am comfortable with. I need to remember operator precedence rules (does the >> happen before or after the & or does it even matter)? And, I need to understand why we do each operation as well as what the magic numbers 15, 4, and 16 all mean.

Back to the REPL.

> c = 1
1
> c ^ crypto.randomFillSync()[0]
9
> a = new Uint8Array(1)
Uint8Array(1) [ 0 ]
> a[0] = 34
34
> x = a[0]
34
> c ^ x
35
> 0 ^ x
34
> x = 254
254
> 1 ^ x
255
> 0 ^ x
254
> 1 ^ 1
0
> a = new Uint8Array(1)
Uint8Array(1) [ 0 ]
> a[0] = 260
260
> a[0]
4
<snip>

I did a lot more messing around. I realized I was mistaking ^ with |. Clearly I don't do much bit twiddling. ^ is XOR.

I am still not quite sure what this step is doing but I think it does not matter for the 1's or 0's. It has to do with the 8.

Looking at other SO answers this seems to be the case. That answer does some masking in the 8 case but not in the other cases.

Things are still unclear and I realize I could be thrown off by operator precedence. It turns out I am getting it wrong

So the division happens first

> 1/4
0.25
> 0/4
0
> 8/4
2

Then the bit shift

> 15 >> 1 / 4
15
> 15 >> 0 / 4
15
> 15 >> 8 / 4
3

Then the bitwise AND

15 and 3 are the two possible values for the result of the &.

0b1111 means that any bit that is set in the number being & with will be present in the result. So, once again the 0 vs 1 is confirmed to not mean anything because the original number is just being passed through.

0b0011 means that the leading two bits will be 0 in the output of the &. Why is this necessary? Let's look at the ^

8 is 0b1000. Then we are doing a ^ with some number that is in the form 0b00xx (the x's mean could be 1 or 0 since we did an & with a 1 in that position). So this results in some number of the form Ob10xx.

Remember that the 8 is the "variant" bit in the UUID? Looking at the Wikipedia page for UUID's I see that a variant of the form 0b10xx is a variant that conforms to DCE 1.1, ISO/IEC 11578:1996. That is the variant for the spec we are after!

The final bit of code is the toString(16). I know that UUID's are made up of hex characters and that hex is base 16. Looking at the JS docs I see that specifying a number in toString sets the radix (aka the base). So, setting it to 16 makes it a base 16 (hex) string.

Holy moly! There were moments in there I wasn't sure I was going to be able to figure all of this out. Putting it all together, the code creates a string of the right shape as a UUID with a few special characters (4 and 8). Everything besides the special characters are replaced with a random character in the hex range. The 4 is not replaced at all. The 8 is replaced with a value that conforms to 0b10xx. That's all that's needed to create the UUID we are after.

Thoughts on the code

Now looking at this code it makes sense (yay:)) but honestly it looks to me like code golf.

Using scientific notation (ex 1e7) is a cool way to make something with 8 characters but throws the reader off in a couple of important ways. First, the number is going to be converted to a string so it is confusing to use a number. Second, the number is made up of a 1 followed by n 0's. When I read code I'm looking for patterns. This pattern is actually totally not important. The 1's vs 0's make no difference in the final result (only the field with the leading 4 and 8 mattered).

Using the syntax [1e7]+-1e3 is using bizarre (IMHO) type rules in JS. You get a string, and the - isn't for creating a negative number but for the hyphen separating the parts of the UUID.

c ^ window.crypto.getRandomValues(new Uint8Array(1))[0] & 15 >> c / 4 is a fairly hard to understand line too. It should be read from right to left (the opposite of the way code is normally read) because of operator precedence. Operator precedence is something developers need to know but also isn't something they will know off of the top of their head. Especially for operators that don't appear anywhere else in our code (^, &, >>). I think too many ideas are happening in one line. The terse vs readable balance has swung too far in the terse direction. In addition, there are the magic numbers of 15 and 4. They make sense once you break the code down but having a function/named const I think would be better.

Conclusion

Ultimately, I think the code is too terse but maybe that is a matter of taste and it is an exercise in futility to make it clearer. It is pretty unclear to the untrained eye what it does. But, I never know if the eye should become more trained (read: I should become a better programmer) or if the code should be re-written to be clearer.

The code is in a small part of our codebase, isn't mission critical, can be recovered from if it is buggy, and is constrained neatly within a function call. So, it is probably ok as it is but I still can't help but try and re-write it.

Here is my attempt

// Generate a version 4 (random) variant DCE 1.1, ISO/IEC 11578:1996 UUID
// Adapted from https://stackoverflow.com/a/2117523
// TODO(e-carlin): When more browsers support crypto.randomUUID() use it instead
self.randomUuid = function() {
    function randomHexValues(count, prefix) {
        return (prefix || '') + [...Array(count)].map(
            _ => toHexString(randomUint4())
        ).join('');
    }

    function randomUint4() {
        return window.crypto.getRandomValues(new Uint8Array(1))[0] >> 4;
    }

    function toHexString(uint4) {
        return uint4.toString(16);
    }

    function variantHexValue() {
        // Variant must begin with 0b10xx
        // See https://en.wikipedia.org/wiki/Universally_unique_identifier#Variants
        let v = randomUint4();
        v = v | 0b1000;
        v = v & 0b1011;
        return toHexString(v) + randomHexValues(3);
    }

    const VERSION = '4';

    return [
        randomHexValues(8),
        randomHexValues(4),
        randomHexValues(3, VERSION),
        variantHexValue(),
        randomHexValues(12),
    ].join('-')
};

This is much less concise. But, since our code rarely handles things like this I think being less concise here is better. By being less concise there are many breadcrumbs sprinkled throughout that I think help explain what is going on. In addition, I think it is possible it will make developers smarter by reading it. If you don't know about UUID's (I didn't before writing this) then you at least learn they have this specific layout and there are versions and variants embedded in their structure. Ultimately, when browsers support randomUUID we'll be using that and leave UUID generation up to the pro's.

Now, off to forget everything I just learned about UUID's, operator precedence, and bitwise operators.

Git Branch Naming

2021-08-27T00:00:00-06:00

The CTO of my company recently reviewed a set of dependent PR's I submitted (here, here, and here). They had a particularly long and annoying branch name radiasoft/devops/issuue/232-raydata-initial-app**. The branch name was annoying enough that he sent an email to our software team saying: he would prefer if we use kebab-case in our names and we shouldn't include the organization name (ex radaisoft) in the branch name. His reasoning for kebab case is personal preference and the organization name is unnecessary since we rarely submit PR's outside of our organization. My feeling when someone sends and email like this is that the answer should almost always be "yes."

Issues like branch naming are the thorny issues in software where finding the right answer is nearly impossible. But, the wrong answer is to spend time thinking about it and for folks to not come to an agreement. Having a regular naming pattern that everyone agrees on allows tooling to be built around it and frees minds to think about other things that matter more. Amazon has a value of "Have backbone; Disagree and Commit" that I think that applies well in this case. Maybe by saying "yes" so quickly I didn't have a backbone. But, all of us committing to using the same strategy is what I think is important.

But, if I was the kind of person that liked to die on hills of no value then I would say I disagree.

The long name with the organization is annoying but it is only annoying once, the first time I type it in. We use GitHub and their UI has nice "copy to clipboard" buttons in most places where a branch name is displayed. In my shell I have the function gb which displays all of my local branches and lets me enter a number to select one. So, after the initial git checkout -b I never fully type out the name again.

Using organization/issue/number-description maps closely to the syntax for tagging an issue in GitHub (organization/issue#number). That means less syntax for me to remember and fewer characters to change when I copy the branch name into a commit message to tag an issue.

Using organization/issue/number-description also maps closely to URL's. It would have been nice if GitHub had chosen issues/number for tagging because then branch names would map perfectly to their URL's. But, still I only have to pluralize issue and remove the description (easy with C-k in the browser) and I can quickly navigate to the issue I'm working on.

Maybe one day when I'm in charge I'll use my strategy but for now it's kebab case with no organization name for me.

** In case you didn't catch it I misspelled issuue. I've started using flyspell in Emacs but it doesn't play nice with code. If someone has a spellchecker they use in their editor that actually works please send it to me.

Create More, Consume Less

2021-08-25T00:00:00-06:00

A few days ago I read an article titled Consume Less, Create More. It reminded me of the article I read in college that was one of the inspirations for this site.

I haven’t posted an article since May 16, 2020.

In that time I’ve written one article that covers my thoughts on Bruce Springsteen’s entire discography. I don’t know if I’ll ever share it but it was fun to write.

I’ve written down many ideas for articles I might write but I get scared wondering if they are good enough which prevents me from writing even a single word. For example, while writing this I’ve stared at the screen for ten minutes trying to decide if the title should be “Create More Consume Less” or “Consume Less Create More.” Spending time worry about details like that is preventing me from gaining any of the insights I'm hoping to find from writing.

So, I’m going to start writing more. These articles are going to be bad. I’m going to proofread very little and I’m not going to have someone else look at them before I publish them. I’m trading quality for quantity and I have a feeling that quality will naturally arise overtime. My goal is to publish one article per week.

I’m going to write about things I’m not qualified to talk about so please take everything you read with a grain of salt and reach out when you see inaccuracies.

Shortened URLs in email are evil

2020-05-16T00:00:00-06:00

Shortened URLs in emails are bad for security, encourage tracking, and are completely unnecessary.

Huzzah, take that URL shorteners!

A few days ago, a recruiter reached out to me. All of the links in the email were shortened URLs. The URLs redirected to the domain zen.sr. My initial reaction was "Ooh, definitely won't be clicking on those links!"

I've never seen an sr domain, I hadn't heard of the company, and I'm pretty suspicious of clicking on any links in emails. Gmail does a pretty good job of filtering out the phishing emails that come my way; and the few that do manage to slip through tend to be blatant spam. I read over the recruiter’s email again. It didn't seem very spammy. It was well-written and the content looked pretty standard for an email from a recruiter. I suppose that all blanket recruiter emails that are general enough that they could be sent to any engineer for any position are spam by definition but I'll save that argument for a later time. Now I was curious: Was this a real recruiting email or was it some well-disguised spam?

First, I looked at the sender email. I figured that if they worked for the company they were hiring for their email would have the same domain. One point for this being a real email; the email was joe@acme.com. I opened up a private Firefox window and did a search for acme.com. There was a hit and everything seemed pretty normal. I went to the website and clicked around. If this was a phishing scam, it was pretty elaborate. Looked like a real company to me. At this point, I decided that the company seemed real. But I was bugged by the shortened URLs!

So, I decided to do some investigation into the shortening service behind zen.sr. The first search result was exactly what I was looking for: A help center article from gem.com. According to Gem's website, they are "The Platform For Modern Recruiting." Okay, the recruiting email almost certainly wasn’t spam. The help center article is pretty funny. It says that they shorten URLs to zen.sr so they can track clicks. Yep, I had a feeling. The funny part is the last sentence of the article. It says, "As a best practice, we recommend turning it on, with the exception of a few roles that tend to care more about privacy (e.g. security engineers)." Ha! Security engineers?? So you should turn off tracking for people who know they are being tracked but tough luck for everyone else? Ugh.

This sent me off on a little hunt to learn more about shortened URLs and their history. The first site I landed on had all of the info I wanted. The prose bugged me a bit (I wasn't in the best mood at this point) but it was a treasure trove of information on URL shorteners. The tl;dr is that they have been around since 2001. They were originally invented so a guy could post links on a unicycle forum (I love the web). Since then shorteners have had a checkered history. At first, people were upset that they masked the true destination of links. A few years later, they started to show up in any plain text context where long strings were an issue (ex. Twitter) -- this is arguably the only place where URL shorteners are somewhat useful. In 2008, as so much of the web started to shift into a tracking and advertising machine, so too did URL shorteners with bit.ly's launch of the first link analytics tracking service.

After the history lesson, I looped back to the recruiter's email. Why did they need to use the shortening service? The email was HTML and the link was in an <a> tag so the shortening aspect of it was unnecessary. That leaves one reason for the shortened URL: Tracking. This type of tracking is invasive. What are the metrics that are gained by tracking? They already know how many people respond to their emails. Isn't that enough? You can run a/b experiments and the like just based on that. Also, are the metrics really that sound? Certain email clients will click on links to assess their security. Likewise, when I held down on the link on my iPhone to copy it, it opened it in a little modal. I wouldn't count either of those as real clicks. Does Gem not count them too? I'm sure my imagination is letting me down here but this tracking just seems unnecessary.

At the very least, obscure links in emails are unsafe. Every publication under the sun has written about not trusting links in emails. Informed users have been trained to not click on links in email. They may have even tried to fight invasive email services like Superhuman. But with invasive email practices (as with so much else in tech) there is an asymmetry between the people building the tech and those who use it. Steve Jobs discouraged his kids from using iPads. The same ones he was trying to sell to schools. And most people don’t have the time to dodge tracking at every turn. If we normalize obscure links in emails then we’ll never get the cat back in the bag (though perhaps the cat is already out of the bag with the prevalence of shortened URLs on Twitter).

Link shorteners in email are evil. At the very least, services like Gem should disable them by default. Their docs encourage users to disable them for “security engineers.” If they are willing to disable them for the population in the know then they should extend that decency to everyone. This email was from a recruiter who wrote me out of the blue. I’m not looking for a job and I don’t advertise that I am anywhere. If someone is going to write to me then they shouldn’t track me while doing it. This is like a door to door salesman snapping your photo as soon as you open the door. There is an information asymmetry that means the sender gets to know more about me before I even know what their business is.

P.S. If you're thinking to yourself that it is annoying to read a post that bemoans tracking written by someone who has worked for the chief tracker (rhymes with “Schmoogle”) and uses Gmail for their email; all I can say is that isn't lost on me and you are certainly right. Happily, there are people hard at work trying to reduce harmful email practices and I encourage you to check them out.

P.P.S. This website uses analytics. You can read more about them and get a link to the public dashboard by visiting my about this site.

The Interactive Subshell: Emacs’ Killer Feature

2020-04-07T00:00:00-06:00

I've been using Emacs exclusively for a couple months now.* Like any good tool, there are things I like and things I don't like. I'll save diving into all of those upsides and downsides for another post. In this post, I’ll talk about Emacs' killer feature: the interactive subshell. Perhaps the "real" killer feature is something more meta like Emacs' programmability but that wouldn't have made for such a juicy title would it?

*If you’ve never heard of Emacs and you are curious, prepare to spend the next hours, to weeks, to your lifetime learning and using one of the most extensible text editors. Once you’ve played around with it come back and read this.

Disclaimers

Before I begin, you should know that:

I have a bias towards interacting with my computer through the keyboard as much as possible. The full power of the interactive subshell can only be harnessed with the keyboard. So, this article will probably be most valuable to people who also prefer to use their keyboard, but even if you prefer to use your mouse you may still find something worthwhile here. And who knows, maybe you'll be so inspired by the interactive subshell that it will be your hook into using your keyboard more.
I don't have very much experience programming on an OS besides macOS or Linux. I used the Windows Subsystem for Linux briefly but that's about as far out as I've ventured. I think this article has a bit of a Linux slant but I believe the message holds true regardless of what OS you use as long as you use a terminal.

Now that I've checked off the mouse vs keyboard and OS boxes on my flamewar bingo card, let's get into the good stuff.

What is an interactive subshell and why should you care?

The "interactive" in "interactive subshell" means that all of the text in the terminal can be interacted with just like the text in any other file. Everything you can do with text (delete, move, select, etc) you can do with the input and output of the shell.

Interactive means you just grep'ed for something and because your regex fu is so bad you don't know how to craft one that doesn't include a bunch of useless results — so you just delete those lines from the output.

Interactive means you just cat'ed a log file and you're sifting through the output and see two lines that you want to compare side by side. You just delete all of the lines in between.

Interactive means your test suite is running and you want to search for a keyword in the output but you forgot to clear the output of the last run so you just select the old output and delete it. No need to worry about possibly finding the keyword in the old run.

Interactive means you have an ENVIRONMENT_VARIABLE_THAT_IS_UNBELIEVABLY_LONG and you want to unset it. If you wanted to echo $ the variable then your shell would let you tab complete it but it can't tab complete with the word unset in front of it. In an interactive subshell, you can use the autocomplete of Emacs to tab-complete the variable name.

Interactive means you are trying to find a file and you can't remember the name so you run a few find . -name "filename". Each time you run the command you want to di" (delete all text in-between "", would delete filename). But oh wait set -o vi in Bash doesn't support that feature even though your editor does.

I could go on but I'm ranting and I didn't anticipate ranting in just my second blog post. Here’s what matters: Interactive means you are in control. You can do whatever you want with the text in the shell as if you had written each line by hand yourself.

Why should you care? Well, I'll describe why I care and maybe you'll agree. Like many programmers, I interact with my shell (Bash) regularly. There are countless tales on the web of why shells matter and how they are a powerful way of interacting with the os. I won't get into the shell/GUI debate (add it to your flamewar bingo;)) but I use my shell and I like my experience interacting with it to be comfortable. I want to interact with the shell input and output however I please. I’ve described a few scenarios where I want to manipulate the shell input and output and I don't think my terminal should get in the way of me doing that.

In addition to manipulating the text as I please, using all of the features I use elsewhere in my editor within my terminal reduces cognitive load. If I want to use vi keybindings in my editor why should I get some watered-down version of them in my terminal? If I want to autocomplete, I should be offered suggestions of strings found elsewhere in my program. In both places I'm just editing text so I'd like my workflow to be the same. My terminal should trust me to edit the content however I want.

IDE's and Terminals

VSCode, the JetBrains suite of IDE's, Eclipse and many other editors of the day have powerful text editing features. You can efficiently manipulate text; moving it here and there and customizing your editing experience late into the night. All of these editors support plugins to allow for vi or Emacs keybindings which further increases your ability to manipulate text without leaving your keyboard. What all of these editors lack (at least in my experience, please prove me wrong) is a terminal where you can work on the text as you do in the rest of the editor. They all have a terminal but the terminals offer no special text editing features. For example, in exactly zero of these terminals can you delete a line of output.

NeoVim and Vim 8 have a half-baked terminal mode where you can navigate text (hjkl, search, highlight, etc) but you can't manipulate (delete, autocomplete, etc) text using the same commands you use in the editor. Vim also offers a mode where you can take the command, go into a temporary buffer, edit the command with normal text editing capabilities, and then drop back in the terminal pasting the command in. This is clunky; you lose the context of everything else going on in your terminal, and it only works for the shell input, not the output.

In addition to the editors, the terminal emulator apps (iTerm, GNOME terminal, etc) suffer from the same lack of editability. All of them offer some amount of text manipulation for editing just the command but not anything else. Usually, they default to having Emacs keybindings and if you're so inclined you can set -o vi and hjkl to your heart's content. While the ability to edit the command is nice (mandatory!) the inability to edit anything else is suffocating. I'm a vi keybinding user (talking about how good Emacs is, gasp) so I can't speak to the Emacs keybindings but if they are like the vi ones then they are half-implemented at best.

For example, in iTerm with vi keybindings enabled gUw (make uppercase from cursor to the end of the word) doesn't work. The specific command isn’t so important, but what is important is that it’s muscle memory. It slows me down when I have to stop and say "ohh I'm in the terminal now and the terminal says I can't do that" (said in a whiny kid voice). Other things like copy and paste keybindings also don't work; I can't move text elsewhere in the terminal (or in another split pane). This can be done with NeoVim and it can be done using Tmux but it doesn’t feel natural, in my experience. In NeoVim I'm always forgetting whether I'm in insert mode, normal mode from set -o vi, or normal mode in NeoVim. Tmux uses different keys altogether; so more muscle memory where my wires get crossed. Ok, that's enough bloviating on why the terminal you use is awful and why my setup is just so much better:). Check off Emacs vs vi on your bingo cards (sheets? I've never actually played bingo...) and let's move on to why Emacs solves all of the issues elaborated so far.

Emacs Shells

If you’ve made it this far hopefully I’ve persuaded you a bit that your terminal isn't interactive but it should be. Thankfully Emacs has us covered. As with seemingly all things Emacs, there are multiple shells and modes of interaction. Any shell could conceivably be made interactive. The shell itself isn't what is important but the way its input and output are presented is. I describe a mixture of shells and terminals below. Try to ignore that I sort of conflate the two. The point is these are the default shell/terminal combos that come with Emacs and let you do the shell/terminal things you might expect.

There is Eshell, which is a shell implemented entirely in Elisp. The Eshell is presented in an interactive manner which is excellent but the rest of the shell is not my cup of tea. I don't know Elisp so I think I'm not the target audience. The skills learned using it are not very portable to other shells.

There is the Emacs terminal emulator which, as the name implies, is a terminal emulator running in Emacs. You can run whatever shell program you want and you can edit the text as you wish. This is all great but the term mode has one downfall. It is modal (A vi user bashing on something being modal? I can't get my story straight). There is line mode and there is char mode. Line mode is the fully interactive mode where you can manipulate the text displayed. Char mode is like any other non-interactive terminal emulator. I have no need for char mode so I find switching between the two annoying and pointless.

Finally, there is the Emacs shell mode. Shell mode is like the Winston Churchill quote about democracy, it is the worst shell experience except for all of the other ones. It is slow, C-c frequently doesn't work (I think that is an artifact of using Evil Mode), and just generally feels kind of clunky (I know, very scientific). Also, it doesn't support ANSI escape sequences. If you're like the guy I work for, you'll go all the way to building a website to remind people that in shell mode you $TERM=dumb and they should play nice. Programs like top won’t work at all. These defects are just minor inconveniences and the upside, interactivity, far outweighs them. If you want an interactive shell then this mode is what you want.

Interactivity or bust

An interactive subshell might be something you have to play with to grasp. If you're unconvinced by my yelling then fire up Emacs and type M-x shell. That's Emacs speak for "hold down the alt/option key (Meta) and x." Then release, type in shell, and hit enter. That will drop you into the Emacs shell. Try using it for a day and see what you think. At first, it will feel clunky. This feeling doesn't really wear off but at some point you'll be so addicted to editing the text in your shell that you won't even care.

NY style pizza is better than Chicago deep-dish. That's five across and I win flamewar bingo. Have a great day.

It's just code

2020-03-27T00:00:00-06:00

Finding my way to the first post

Ahh, my first post. A daunting task for someone who has spent their entire life avoiding writing at any cost. I suppose this is a make or break moment where I decide if I can learn to enjoy writing. Lately, when my mind has the time to drift I have been mulling over what I should choose as the topic.

My first thought was something simple, personal, and harmless. I figured writing about my personal code setup (editor, shell, etc) seemed like a fine choice. Not very exciting but would get me in the groove. These types of posts are like candy; I can't say no to them but they're not what I need. But recently I reached what feels like a new milestone on my programming journey and I figured describing what happened would be much more interesting. So here it is.

Becoming an apprentice

In September 2019 I started a new job. I felt like I wasn't improving as a programmer in my previous job and I dreaded going to work every day. I love programming and working so that was a bad sign. Thankfully, I had a friend working at an interesting company and he was open to working with me. I wanted the job because the company is small. They're solving hard problems - physics simulations. Each of the three other software engineers has been programming for longer than I've been alive. And most important, they seemed willing to take time to pass on their collective knowledge to me. I had a hunch that there would be plenty to learn. Turns out that hunch was dead-on.

I work closely with the CTO, Rob Nagler. He has imparted quite a lot of wisdom so far. I'm uncertain I’m qualified to make this judgment (you'll see in this post how little I know about programming) but I have a strong suspicion Rob is a world-class programmer. I have had the pleasure of sitting 5 feet from him every day for the past six months. We talk about programming and I watch him solve problems day after day. Watching a master at work has helped me improve my skills at a faster pace than ever before. Reading works by smart people or being on a team with them is one thing. Being able to sit with a craftsman, watch them do their work, and get snippets of their thinking is so much more powerful.

It’s just code

One difference between Rob and myself is our fluency in reading code. We work on open source code and use many open source tools so there is a great deal of code to read. Before working at Radiasoft I can say (with some degree of shame from my present vantage point) that I had never really read code. If you're a professional programmer a statement like that probably makes your skin crawl. Looking in the rearview, it is somewhat amazing to me that I ever solved a problem with how little code I read.

I've worked on projects with other people and read the code they check-in. I've started new projects with existing codebases and read the absolute bare minimum of the code to do what I need to do. I've read snippets of code on Stack Overflow (duh) and blogs. I've even poked around open source projects I was thinking of using. But I've never really truly read code. I've never skipped the documentation and gone straight to the source code. I've never read another project's source code and used it to inform my approach to a similar problem. I've never encountered a bug in an open source project and fixed it myself (except for this small change). Rob does this kind of reading every day.

When I run into a problem I spend time fumbling through documentation, bug reports, mailing lists, and Stack Overflow trying to find answers. Most of the time this solution works well enough, but it can be slow, lead to awkward solutions, and sometimes just doesn't work. What I now realize is that programmers better than myself have another tool to use. Reading the source code.

In the past 6 months, there have been a few times where I have been truly stumped, tapped Rob on the shoulder to ask for help, and seen him go straight to the source code to find the answer. He's done this with the Tornado web server, the asyncssh Python library, the cPython source code, and our own codebase. All of these are large codebases with plenty of places to get lost. The first few times I saw Rob do this I was blown away. In seconds, he figured out where to look and in minutes he'd found the relevant bits of code and the answer. He knew that if he read the code, he’d find the answer. When I ask him how he does it he says, “it’s just code.”

I struggle to have this level of confidence. When I read code I tend to give up quickly, telling myself that the other person is clearly some sort of genius and my mind is too feeble to understand their masterful work. I usually start at a call site or object that I think should be relevant. I then go down long rabbit holes that are orthogonal to my problem and overly specific. For example, I’ll find myself reading through the ASGI spec and how the server implements it even though I just want to know why the incoming request connection was closed abruptly. I try to really understand every last bit of code because it is the only way I can make sense of anything (not everything, anything). This is usually the moment I get frustrated that I can’t find what I need and start to think it simply can’t be found. Rob, on the other hand, churns through other codebases. He moves along knowing that it is just code and with some time and determination he can understand it.

Now that phrase — it's just code — is a mantra for me when I read new code. Every time I read code it is slightly easier than the last time: It’s a muscle that you can build like any other.

Faith in the abstractions

In addition to reading the code (and remembering that it’s just code), Rob has taught me another trick: "Have faith in the abstractions."

Recently I added an administrative page to our website. It allows a user to see their running and pending jobs. I had just finished re-writing our entire job execution system (with a great deal of help from Rob) so I knew the back-end of the code well. I was less in tune with the front-end.

I needed to provide users with a link to each currently running job in the list. I wasn't sure how to do this so I set out in standard fashion: I hunted around the website for somewhere that provides a link to a job and tried to find the relevant source code, which I could then reuse for my purposes.

There was only one other place with a link and it was on the page of the simulation itself so we just used $window.location.href. Not going to work for what I need because I need to link to jobs that are not on the current page. So armed with "it's just code," into the code I went.

I knew that we have a concept of local and global routes so I figured if I looked near those words I'd find something. Sure enough, we have a function called formatUrlLocal. The code calls some other code that isn't dead simple. I'd need to spend some time with it to really understand what it was doing and how I might use it. I had the sense that the code was going to need to change (which was wrong) so I felt like I really had to grok it.

What I should have done at this point is looked for other uses of the code and just used it the same way myself. Instead, I was frustrated that I couldn't find what seemed like a piece of code that had to exist. I asked Rob for guidance. Specifically, I asked, "Are there any places on the site where we have links to simulations. I'm not sure how to construct the link but it seems like something we would already have code for."

Thankfully he's patient and happy to help. As one might expect, he navigated directly to the formatUrlLocal code I was looking at and said, "Here, use this." It took him the amount of time it took his fingers to search to find code it had taken me 15 minutes to get to (and that I didn’t quite know how to use even once I found it).

I asked Rob how he got to the code so quickly and how he knew it was going to help. His answer, "I have faith in abstractions." If you're adept at reading code you may be thinking, “Duh, you can't understand everything so look for the abstractions that may be relevant, try using them, and when necessary dive in and understand them.” But if you're like I was you may be thinking, “Wow I have no clue how to use that advice.”

This is the kind of advice that makes no sense until you use it and then becomes totally obvious. Maybe it’s akin to telling someone to counter-steer when their car is sliding. It just isn't obvious how it works until you can do it. Sure enough, the code worked. It said it formatted URLs and it did just that. There was an abstraction and it worked. I didn't need to understand it top to bottom but I needed to have faith. I needed to try using it.

Now when I encounter problems I have new tools. First, I tell myself: it's just code and dive right into the source code. As I look at the code, I remind myself to have faith in the abstractions. The latter strategy is much harder. There are words and styles that are suggestive of abstractions that will lead you to the answer but it takes time and experience to discern the worthwhile ones from the irrelevant ones. And over time, you hopefully learn when the abstractions aren’t quite right. That’s when it’s time to go deeper. I can't say I really have any sense for this yet. But what I do have is the awareness that the answer is out there, in code, waiting for me to read it.

These are just the first steps of my journey towards really reading code. I hope that, as my journey progresses, I will be able to write many more posts with actionable insight on how to become more fluent. For now, the only advice I have is to reiterate Rob's insights: It’s just code and have faith in the abstractions. On top of this, I have a small piece of advice of my own: If you're not reading code, start today. It gets easier every time.