This article made its way to the front page of Hacker News last week. One of the points the author makes is on miscellaneous tooling. Specifically, the benefit of having a place to put utility shell scripts from the get-go of the project.

Benefits of utility shell scripts

I agree with the author. Having a structure for writing utilities makes it easier to write and share them among the team. These can be for things like fundamental features (ex. starting the application) to rarely run scripts used just for development. Creating a structure for the whole team to share them makes everyone's life easier.

Codifying the utility shell script concept (pkclis)

I think there is something even better than just having a collection of shell scripts. Something I'm calling "application CLIs" for lack of a better term.

At work, we have a framework we use to initialize all new Python projects. It is called projex. This is a scaffolding tool like yeoman generator. It sets out the initial bones of the application. Adds a README, LICENSE, GitHub workflows, test and source directories, etc.

One of the important features it creates, that I think is an evolution beyond just shell scripts, is a concept called "pkcli". Pkclis are CLIs that one can write in Python (the language of our app) and through a small bit of infrastructure become CLIs that anyone who has installed the application can use. These are the "application CLIs" for our projects. Projex itself is actually a pkcli.

Benefit: In the native language

Shell scripts certainly have their place. Even though we have application CLIs, we still write shell scripts. But, something you can't do with shell scripts is write them in the native language of your application (assuming your app isn't written in a shell programming language).

By writing CLIs in the language of the project we can use all of the features of the project. For example, we have an application CLI that deletes users from our system. Someone could certainly write this utility as a shell script. But, it's trivial to write this in the language of the application by just using the APIs already in our application for managing users. For example, we don't have to invent a new way of finding the database in a shell script. Our application needs the database so we already have an API for finding it. In addition, if we want to add a user interface for deleting users we can share the code between the two modes of interaction. This creates a more explicit coupling between the application and the CLI. This, in turn, makes testing and future changes easier to support and more likely to be correct.

Benefit: Fewer languages

Writing CLIs in the language of your application comes along with the normal benefits of using fewer languages. Bill mentioned in his article that he learned how to write shell scripts from a colleague who set up the shell scripts in his project. Everyone on our team knows some shell programming language (Bash for us). But, they know more Python. By writing CLIs in the language of our application we reduce the amount of things people need to know. Likewise, we get to use all of our existing infrastructure for free. We can do things like test our application CLIs alongside our regular application tests. We can use the same formatter and linter our application uses. We can use the same documentation generator to build docs from comments in the code. We can also use the features of our IDEs to navigate and refactor code in just one language. By writing CLIs in the language of the application we reduce the barrier to entry for writing CLIs and get to share in all of the infrastructure/tooling we have established in our application.

Benefit: Availability

Shell scripts can be hard to distribute. How do you update your $PATH so all scripts in all of your projects are available globally on all systems where you might want to run them? At what level of portability do you write them to make sure different systems can execute them? These are solvable problems but application CLIs solve them without much thought.

Using Python's pyproject.toml we can easily define CLIs that then become globally available. Anywhere we're running our application, which in my experience is everywhere we want to run the application CLI, already has the runtime installed. For us that means Python and all of the needed dependencies. In terms of portability, Python handles this for us. Anywhere that can run Python can run our CLIs.

Benefit: Dependencies

The shell programming language I'm most familiar with is Bash. So, this may only be relevant to it. But, it doesn't have great support for managing dependencies. This is related to availability above. For example, if one wants to add a testing framework to their Bash scripts they may use bats. It provides a myriad of ways to install. None of them are particularly friendly. We could have documentation that tells users what they need to install. Or roll our own Bash dependency management system that we'd have to make portable to different systems. Neither docs nor our own system are great solutions. Python's dependency situation is a mess. But, at the very least it provides files that the built-in tools in Python know how to use to manage dependencies. So, someone new to our application just has to get the application installed and then they can use our app and all of our CLIs. By using the language of our application we get good enough dependency management built in.

Benefit: Better namespacing

Another benefit is namespacing. One must be careful in shell scripts to create namespaces without conflicts. In our application CLIs, this comes easier. The first part of all of our application CLIs is the name of the project. Then the second part is the name of the module. And finally, the third part is the name of the function in the module. So sirepo admin delete_user is in the project sirepo, in the module admin.py, and runs the delete_user function. This can be done in shell scripts. Just prepend the application/module name to the name of every function. But, it can devolve into long/messy names and make integrating different projects a challenge.

Benefit: Better arg handling

I'm not a Bash wizard, so take this with a grain of salt, but I find parsing args in Bash to be a pain. The top hit for "how to parse args in Bash" is a classic Bash answer. Conflicting info. Confusion over portability. Many caveats. New ways that supposedly fix all of the problems. Again, all solvable problems. But, one has to create infrastructure and knowledge among the team to make it happen. With our pkclis, the details are handled for free (mostly) by a well-supported third-party library. From the perspective of someone writing an application CLI, they just have to understand args and kwargs in a fairly friendly way. Anything that is a *arg becomes a positional argument. Anything that is a **kwarg becomes an optional flag that you can supply a value for. We haven't run into much friction with people learning that.

As always, there is no free lunch. We've run into problems with our arg parsing. But, the fix was in one place and fairly easy to figure out.

Shell scripts are still used

Even with application CLIs, there is still a place for shell scripts. For example, we use a shell script that configures and starts our application. Setting up an environment and managing certain parts of a Linux system are just easier in a shell script. But, our startup shell script calls our application CLIs to actually start the application. The application CLIs provide a nice interface to bridge the gap between the shell scripts and the application. In other cases, we only use shell scripts. Sometimes it is just the best tool for the job.

Conclusion

Application CLIs are not unique to the company I work for. WordPress has wp-cli, Django has django-admin, and Ruby has a comprehensive suite of CLIs. Rob, the CTO of my company, brought the idea of application CLIs from a previous framework he built. But, I think the gospel of application CLIs could still be spread further.

There's nothing inherently wrong with shell scripts. In the right hands, they can be just as powerful, offering their own unique benefits. But I've found success with application CLIs and will bring them into the codebases I work on in the future.