The Chamber 🏰 of Tech Secrets is open, albeit a few hours late as I forgot to schedule the post. This week, we’ll dig into the concept of Abstractions and explore where they are useful and where software architecture practitioners might want to be more careful about them. Let’s go! Thank you for reading, as always! 🙏
Abstractions
The practice of software architecture often deals in the application of abstractions. Software systems are composites of many layers of abstractions interspersed with implementations of business logic that find their home somewhere in between.
Abstractions are a technique that can be used in software development or systems architecture that focus on hiding or abstracting certain features or functions with a goal of minimizing and managing complexities. Abstractions are present at all different levels, from software libraries to APIs to infrastructure.
Without abstractions, the tasks that are currently very simple would be time consuming and it would be difficult to build things of value.
Abstraction is really great for:
Hiding the inner workings of a system to make integration and interoperability simpler via interfaces instead of direct, point-to-point integration that requires server or database access and a working knowledge of a systems implementation.
Simplifying the knowledge required to complete a [complex] task. Consider an HTTP client that makes web requests simple or an API that enables processing a credit card transaction.
Force multiplying software delivery in organizations by breaking down a large system into loosely coupled architectural topologies with their own [DevOps] team structure.
Hiding functions that need to happen but are not the primary concern of a team. For example, routing traffic through Cloudflare to function as a WAF can ensure a team that the traffic arriving at their service has met certain conditions without having to worry about how those features are implemented.
The Gravity of Abstractions
Abstractions tend to have a gravitational pull. At first glance, they almost always look like a good idea. Implementers tend to think they are doing the rest of the community a favor by hiding something difficult or complex. There is a certain satisfaction when you define a new abstraction. However, it is very important to consider the second and third order effects of each abstraction you implement.
A good software architect (whether by profession or employing the activity as an engineer) learns to consider where abstractions can apply and be useful, but does not apply them whimsically. Abstractions, done wrong, lose their force multiplying impact and can make simple tasks more complex than they should be.
Not all abstractions are created equal
Lets consider some use cases where abstractions may be less valuable. These are not always, but could be, anti-patterns in many contexts.
Super SDKs: Super SDKs do all the things for a user, including implementing opaque business processes on their behalf. These SDKs generally seek to do too much and are difficult to understand, train on, maintain, and support going forward. Developers are hand-cuffed by issues since the [typically] proprietary code isn’t google searchable and can be hard to grasp (why did they do this, this way?). I initially pondered considering the AWS SDK as a super-SDK but decided it was not since it actually leaves much of the “business logic” to its user and basically just wraps APIs (feel free to disagree with me). We created a Super SDK for our Internet of Things clients at Chick-fil-A, which made it much harder for our client developers to succeed in deploying their applications at scale, and created a massive support burden that only our team could respond to.
Platforms that remove the superpowers of the underlying technologies they are built upon: Many platforms built on top of tools like Kubernetes (itself an abstraction for compute infrastructure) or similar hide valuable capabilities in the pursuit of making things “easy” for developers. Platforms built on top of K8s can be wonderful, but must at least match its native abilities in the key areas of concern in order to be useful. Many PaaS solutions have fallen into this trap, leaving developers frustrated. Another area of personal experience is in enterprise middleware tools, which attempt to abstract programming languages by creating configuration interfaces. These generally work great for happy-path scenarios but cripple execution elsewhere and take away the power of googling for answers.
Building abstractions that require complex new DSLs [or similar]: Any abstraction needs to be easier to understand than the thing it abstracts. If a new Domain Specific Languages (DSL) or similar is needed to leverage the abstraction, much of its power may be lost. A simple DSL may add value and justify the abstraction. Many of the frustrations with Kubernetes are due to the vast amounts of YAML developers must create and understand in order to carry out common tasks.
Creeping amorphous abstractions: Abstractions that force developers to adopt paradigms beyond the direct use of the abstraction itself creates lock-in (everything must be done this way) and lock-out (we can’t use any of those others things that may benefit us).
Some questions to ask yourself
Is my abstraction well-defined and easy to understand?
Does the lack of an abstraction tightly couple multiple teams together and force them to understand implementation choices (tech decisions, database schemas, etc.) made by other teams?
Does the abstraction force lock-in to any paradigms outside of the scope of the abstraction itself?
Does the abstraction lock-out the usage of other approaches to problem solving that are tangential to this problem space?
Will the abstraction make life simpler and easier for the developers and let them use the skills they bring from the market without learning proprietary new approaches?
Is the abstraction SOLID?
Does the use of an abstraction hide something that is unnecessarily complex for the average developer that works on our systems (or that might in the future)?
Can my developer community still use google / ChatGPT to find answers to their questions when using my abstraction (or am I comfortable maintaining lots of documentation and playing a support role long term)?
Does my abstraction comply with typical software engineering practices and industry standards?
Abstract away, or don’t
A well-designed system will meet requirements, be able to evolve in the future under new requirements, and be understandable by its maintainers. Abstractions can make this goal reachable, but they can also derail it when done haphazardly. Not all abstractions are good.
As a software developer or architect, I encourage reflecting on your own experiences of employing abstractions to consider where they have been a net positive, and where they have fallen short of their promise. Which abstractions have paid off with force-multiplying impact? Which abstractions have slowed progress and been challenging to work with?
The Chamber of Tech Secrets is closed. Thanks for reading! 🙏 If you enjoyed this post, please share it with your friends and co-workers. See you next week.