I think my understanding of programming languages — ex: what role do they serve in tech engineering work, what my favorite language is — is something that continues to evolve, and has done so once more recently. Here is where I started from and where I stand:
Meandering through languages
When first learning to program, I remember learning that BASIC was the beginner’s language. The Commodore 64 was how I played computer games, but it could allow you to write programs in BASIC. I only learned, though, when I signed up for an elective in elementary school. It was the successor class to an elective where all we did was play MECC games on Apple II computers (Number Munchers and Oregon Trail, especially). Over the summer, when I tried to take that knowledge to a regular PC with MS GW-BASIC, I wasn’t able to figure out how to draw to the screen, change colors, etc.
When I took a class in the following year about Programming in Logo, it was a breath of fresh air. The syntax was easy, it was seemingly all about drawing pictures, so the feedback loop was immediate, and the notion of functions and even recursion all kind of made sense. At the end of the semester, there was a chance to try out for a programming competition. There was a prompt that describes what the program should do, and some sample input and output. Since I didn’t have a Mac at home with a Logo interpreter, I decided to write it in BASIC, and I wrote a bunch of terrible code that didn’t catch corner cases. I got really frustrated — it challenged the notion that I had built up through Logo that I was capable of expressing exactly what I wanted. And that’s when I wanted to give up on programming altogether. It was too hard, too frustrating, requires unreasonable perfection, and therefore not enjoyable.
(In hindsight, I’m not sure I ever understood BASIC subroutines, even though Logo functions made perfect sense. And only many years later did I learn enough to be able to explain the level of complexity that BASIC requires of the programmer to deal with dynamic variables (and therefore, the state of memory, to some degree), and contextual control flow via line numbers and goto and gosub and return. Logo, as a functional (Lisp) language, was far simpler in its evaluation and execution model.)
I still kind of thought programming was interesting, and thankfully, I took Pascal a few years later. Turbo Pascal was capable of stylizing text with colors, made it easy to receive input, and manipulate text. That was enough to make an interactive text-based menu program that was interesting to me. And that led to learning a little of C++, and a little more C++, before learning a smattering of Java, C++, and Perl in undergrad. C++ seemed “adult” – this is what serious programmers used to write cool things, but it seemed complicated. Java was easier and more regular, but the design patterns & OOP seemed obtuse and clunky at times. All throughout, I didn’t have a clear favorite language. Whichever was “easiest” and allowed me to do the most would be my favorite, and since “easy” is a relative term, I was always enamored with whatever I was using at the time because I was in that headspace, and it was therefore easy to continue thinking with that mindset.
At one point in grad school, I started using Java to effectively write a simple CLI tool that would read in files in a particular directory structure and use those files as inputs to kick off other Java code with the actual complex algorithmic work. After learning Python, it made me think that I was wasting my time with Java — it was a lot of effort for little payoff, but Python allowed me to write small CLI tools with much less code and a tighter feedback loop thanks to the scripting nature and the interactive interpreter. Ruby was an improvement over Python in that regard — more regular syntax, and functional-esque constructs (collect/map and inject/reduce) that saved on code and made it a little more readable. It was only when I got to Python and Ruby that I really liked them a lot for how much easier they made my current task over any other language I had learned previously.
Different meta-views of programming languages
In undergrad, my Programming Languages class taught us different types of programming languages, explained how they are designed for different use cases / different types of problems, and showed us how to use them to solve toy problems in each problem domain. I still had to learn that lesson through experience later in grad school when splitting off the CLI interface to the Java tool and converting it from Java to Python.
The Programming Languages course also introduced the idea that anything expressible in one language is expressible in another, so there’s no limit to a language’s expressive power when compared to any other. That was my first level of “wisdom” — all languages are the “same”, in a way.
The next level of wisdom was that some languages really are just “better” than others. To me, Ruby was (and still is) more readable, more regular, more expressive than Python for general use cases. After learning Clojure, I realized that it took care of annoying file parsing tasks very concisely that I might have previously used Python or Ruby for, but it also handles larger scale programs well, both in representation of complex logic as well as deployment concerns. Clojure seemed strictly better than all those languages. And I continued to work in high-level application programming for a long time, for which that sentiment held. Paul Graham’s essay “Beating the Averages” that shows that different languages have different power, and that Lisp was at the top of the power continuum, was eye-opening. And the fact that Clojure can also make it into the browser as well as the backend while not constraining the backend runtime created a compelling story for web apps that really hasn’t been pulled off by any other language yet.
Another adage about engineering that I had heard throughout all this time was “use the right tool for the job”. Even outside of engineering, you can see photographers swap between different lenses for different settings. In technology, Mathematica is a nice program to get closed form solutions to relatively simple math expressions. Some languages / software environments are good for specialized purposes, like R for statistics or Matlab for matrix-based computations. If you’re trying to do logic programming, you want to use a language designed for representing facts and constraints and computing how they are satisfied.
Use the best designed programming language per category of use case
But recently, I have come back to that “right tool for the job” adage, with a slightly modified and opinionated interpretation Of course, we’re looking at programming languages as tools because they are. They facilitate human thoughts to be represented by computers and executed. But there is something to be said about the power index that the Blub Paradox explains. It helps explain why I had no regrets switching from Python to Ruby for simple scripting / CLI use cases. And how Clojure makes tricky parsing problems trivial (ex: I had a technical interview phone screen question that was basically to write code that parses the output of the command git log. Since I was allowed to use Clojure, the problem became trivial in a way that would have been tedious and error-prone in Python or Ruby.) And when learning Clojure, I would take Java code and port it into Clojure, only to find that the number of lines would shrink in a 2:1 or 3:1 or 5:1 ratio.
Ultimately, what I’m getting at is being more specific about what “job” means in “right tool for the job”. The examples above all fit into a category of “general purpose” programming where the algorithm / logic is important, requirements might change as you write the code and/or unpredictably in the future, and where squeezing out every bit of runtime performance isn’t the most important factor. Scripting languages and Java/Clojure rely on garbage collection, and Java’s latest iterations of GC have become impressive, according to others, allowing long-lived repetitive applications to not be too much worse than if it were written C/C++ (off by a few percent versus a constant factor? I don’t have a great intuition for the numbers). The point is that there are other factors that are also important to the business bottom line: developer time, developer velocity over time, maintenance costs, including the ability to elegantly handle major requirements changes without wholesale rewrites of your code. And the category that the use case falls into is what we should define as the “job” when it comes to programming languages.
Specifying the category where Rust is an exemplar
So in addition to the category that I’m calling “general purpose programming”, there are other categories. For example, as I mentioned before, Rust is great for library / low-level systems programming. Runtime performance, including control of memory allocation and reclamation, are paramount if you and/or a large ecosystem of others are to build applications using your code. Rust explicitly prioritizes runtime performance and control, and programmer safety in doing so, above all else. Ergonomics are important, but are still less important when they come into conflict with the first 2 priorities. Rust is designed well and has a clarity of design. With this in mind, it’s easy to see some parallels in the rapid adoption by C++ users of Java and the rapid adoption by C++ users of Rust. I would argue that these languages focus on 2 distinctly different categories of use cases — the general purpose, and the low-level systems programming. And this is why you really can’t compare Rust to Java, IMO, any further than that.
There is a little bit of extra complexity to Rust as a language, but it’s not a problem to me because the language is clear about making that tradeoff to allow that cost in order to get the benefits that Rust is optimized for — performance and developer safety. For example, with the notion of ownership, borrowing, and references, you have to learn about lifetimes. It’s impossible to avoid lifetimes if you’re writing more than a tiny amount of Rust in a real life situation. Lifetimes can be tricky at first, but they’re integral to properly dealing with issues of ownership/borrowing/references, and the compiler will let you know. The payoff is that Rust forces you at compile time to do many things that experienced C/C++ developers do to make their programs as efficient as possible — and in fact, C/C++ developers already will use terms like “ownership”, “move”, “borrow”, and “arenas” for the sophisticated things they do to manage memory.
IMO, Rust’s concept of immutability is also “memory-based”. It’s as “memory-based” as the difference between Rust’s Copy and Clone traits are tied up with memory ownership and references, so you can see how it makes you conscious of memory layout (ex: are the fields laid out in the stack frame or are they allocated on the heap?) Immutability in Rust also requires that references either allow 1 mutable writer or allow many read-only readers, and not both in the same lifetime.
To get a sense of why I describe Rust’s immutability as “memory-based”, contrast what it looks like to insert a value into a collection and then get it back out in Rust versus assoc’ing a value to a persistent collection in Clojure, where I describe the immutability as “value-based”. A call to HashMap::get takes a reference to a key, but gives you Option<&V> — an option of a reference to a value — enjoy! You don’t know if there is a non-null value for the key in the HashMap, and even if there is, the HashMap owns the value, so you only get a reference type. And if you complain, Rust people might say, “Do you really need a map, let alone one that wastes so much time doing hashing like a HashMap? Plus, the reference type allows you to avoid re-allocating. Surely you’re writing a system library that would be better off doing it a different way, like 2 arrays, or iterating differently.” But it is cumbersome at times to write code this way because a reference type cannot be compared to an owned type. Clojure makes different tradeoffs, where the language itself might be simpler, but the implementation of certain things like persistent data structures are more complex, and without the benefits like runtime speed that Rust is designed to optimize.
Another way that Rust is memory-based is that the difference between using one type or another is about whether you have to allocate memory. A Path struct instance is a read-only (no-allocation) value, but once you want to join a path segment, you end up with a PathBuf type value because you have allocated. It is akin to &str versus String, or what a slice is to a sequential collection type. And maybe you care about the corresponding difference between std::iter::Iterator and std::slice::Iter.
Clojure, of course, thinks of immutability as independent of memory / location: information is the accumulation of facts, a fact is a value at a point in time, values are immutable, and (I think) data is a collection of values which itself is immutable (a collection of immutable values should be seen as immutable too). Treating a database as an immutable value is a mind-bending concept, but apparently quite helpful in wrangling the complexity of database systems, and allowing the writing of unit tests for them. Again, you don’t the extra simplicity and power from Clojure-style immutability without some tradeoffs, especially when compared to Rust, that won’t fit the constraints of the kind of tasks for which Rust is the exemplar language.
However, you can compare Rust to C++ and say that Rust is a much better language for the use case category that it’s designed for. For Java, yes, Java is better for general programming than C++, but then you hear similar arguments trotted out by Scala users when they compare it to Java. They talk about Java being verbose, complex, and error-prone. But frankly, I think the same thing can be said about Scala when compared to Clojure. Scala’s hodgepodge of Haskell-inspired category theory-lite paradigms and constructs have led people to program themselves into a corner with Scala, get frustrated at their inflexible overly-typed code, and switch to a language and ecosystem that isn’t so rigid. Anecdotally, I’ve seen code shrink by a factor of 2:1 when going from Java to Scala, and again from 2:1 going from Scala to Clojure.
Examples of exemplars and use case categories
I think each use case category will end up with an exemplar language that does the “best” for that use case. Defining the use case category as well as “good”/“best” is fuzzy and fraught, but that’s ok. I think Rust will end up being the best language for low-level programming (it already is, but I think it will stay that way).
- I think Clojure is the best for general purpose programming. This isn’t to say that Scala’s Haskell influence is bad, but that it’s not suited for general purpose programming. And of course, if you’re doing web programming, Clojure + ClojureScript is a fully-featured full-stack language solution that is unmatched (old example from 2015), and whose data literals fix all the glaring holes in JSON to boot.
- That very high level of static typing that Haskell has can be helpful where you have a high-value situation in a closed environment, meaning that every bit of extra confidence you can give your code with rigidity via high-level static typing, combined with the comfort that requirements won’t really change and you don’t have to deal too much with the unpredictable messy nature of real-world input data (because you’re not interacting much with the outside world) means that Haskell could be a good choice, ex: for banking, avionics, health devices.
- Erlang still seems to be good for systems that can never be allowed to fail, at all costs (or so I hear… I’ve seen Akka in Scala, but not used Erlang personally)
- I suppose you have the domain-specific languages. Matlab for matrix operations, R for statistics, etc.
- The availability of necessary dependencies in a language or OS can dictate choices, of course. For example, because a lot of NLP/ML libraries exist in Python, Python becomes a good choice for “data science” and other data-adjacent exploratory programming.
- But you might notice that me even saying Python is good as a language for exploratory work is contradictory with my previous assertion that Clojure is the best general purpose language. This is true — a language that is unrivaled in power should certainly be able to obviate all other languages in this “general purpose” category, if what I say is true, right? And sure enough, a library exists to allow Python interop so that you can do all your NLP/ML work with Clojure’s creature comforts and simplicity.
- For declarative queries, SQL is the absolute worst! Datalog seems pretty nice — you get the ability to have functions and parameters like a real programming language. I haven’t used LINQ, but people like that well enough, and they say it’s clearly better than SQL.
But applying a programming language that isn’t “best-in-class” for a category of use cases is something that I would now be wary about, all things being equal. I think Scala is caught in between categories — it’s not really designed for any clear use case (because it wants to show that the marriage of OOP and Haskell-style Functional Programming is possible and beneficial), and you would be better off with Clojure or Haskell for the most likely use case categories, just as C++ users might be better off with Rust, Clojure, etc. for their most likely use case categories. For example, the idea of using Scala for frontend development, or using Rust for general-purpose web programming, are hard passes for me.
Language isn’t just syntax
And some people think that syntax is an important factor for a language. One person recently told me that they like the idea of Clojure, but instead they like F# better as a functional programming language because the syntax “isn’t weird” — it’s more familiar (in the style of C/Java/Python etc.) That again ignores the important points of Beating the Averages, so if you haven’t read it, go read it now and come back. Ok, so did you get the point about macros? The idea of using them to eliminate domain-specific boilerplate code whenever it crops up should say enough.
But then with macros, you have the ability to make a language library that implements logic programming, including syntactic similarities to languages dedicated to logic programming. This is just a small example how a language whose syntax can be bent into the shape of other problem domains and/or programming paradigm syntaxes is strictly better than a language that can’t. It’s a short-sided argument to say that you don’t like Lisp parentheses. Because if you did, you would be missing the deeper point about such a language… which has its level of expressivity precisely because of that parentheses prefix notation syntax. It allows something special and uniquely powerful. And more importantly, it should be clear how concerns like these are missing the point. There is a huge difference in simplicity between Lisp macros and AST-based macros — the latter is just not the same. Rich Hickey mentioned that macros are what allowed core.async to be written as a library (one that works in Clojure and CLJS), which is an important point because many languages like Scala and Python tried to implement “async” / “lightweight threads” and got bogged down because they required huge complex changes to the language compiler — it wasn’t something that could exist independently in user-land and used optionally. I’m guessing the same is true for transducers. I’m not saying that macro-writing macros should be a point of pride — macros should always be a last resort — but these are more striking examples of what is possible vs. what isn’t in an AST-based macro.
And as I’ve mentioned before, the ecosystem that Rich has created of plain data and a language full of functions that operate on those plain data structures has a huge multiplier effect of power for everybody. Another reason that works is the emphasis to the community contributors to strongly prefer creating libraries over frameworks, and to make those libraries interoperate with the Clojure plain data structures. That’s not something that a language spec alone can dictate — it requires an ecosystem with its own philosophy and culture — and regardless the end result of all this is phenomenal.
Parting Thoughts
The next time I hear someone enthusiastically be a cheerleader for a language to “take over the world”, I’ll just be ready to say, “I guess your world is a lot smaller than mine.”
And maybe if WASM’s Intermediate Representation (IR)’s S-expression syntax gets real trendy, and if the same happens for Rust’s immutability, macros, and functional programming style, it’ll be fun to see who is open-minded enough for a surprising revelation and who still isn’t.
Sequenced Thoughts from C/S 