The Chamber š° of Tech Secrets is open. This week weāll explore the idea of Smart Tradeoffs š§ . Itās been a busy week here between work and personal undertakings. Iāll be heading out this afternoon for a quick round trip to Silicon Valley for the Embedded Vision Summit. Alright, letās dig in!
Making Smart Tradeoffs
I often find myself using the phrase āsmart tradeoffsā. What, exactly, is a smart tradeoff?
A tradeoff refers to a situation where achieving or gaining one thing requires sacrificing or giving up something else. -ChatGPT
We can then assume that a smart tradeoff is one in which we gain something good at the expense of something tolerable.
Making smart tradeoffs in systems architectures is often very complex. It is an art, not a science. Itās about building something good. That may mean āgood enoughā, āfunctionalā, or āthe most awesome thing everā.
Making smart tradeoffs is about building something āas good as requiredā¦ but no betterā. Weāll explore what a good solution looks like in principle and how we might think about the basis for making smart tradeoffs. Iāll use some observations about my experiences at Chick-fil-A to frame the discussion.
A good solution is built on a solid foundation
I am a big fan of documenting the principles that govern how I think and operate. I believe it is really important to think about what is important before you work on something that is important.
In my architecture practice, we attempt to do this with our Enterprise Architecture principles. I have written about them extensively but in short form, they are:
The goal is to be clear about how we think about approaching problems before the problem has a chance to present itself and begin to influence our thinking or warp it in a particular direction. Emotions, timelines, sets of biasesā¦ they all come into play when we start designing a system or writing code.
In the case of Chick-fil-A, our principles guide us to ensure we do not make dumb tradeoffs that compromise things that we already considered important before the problem presented itself. This is step one towards making smart tradeoffs.
A good solution picks the right sandpaper
At Chick-fil-A, we have regular department meetings. The cadence has varied over the years (monthly, quarterly) but the content is similar: business updates, outside speakers, project showcases, and little time with our CIO (now CDO) at the end.
Our previous CIO invested a lot of his time telling stories and using visuals. One that stands out was about picking the right sandpaper for the job. Some woodworking jobs justify multiple passes starting with coarse grains and slowly and methodically working down to very fine grained sandpaper. There might be more light sanding after a stain and polyurethane are applied. Some projects require very fine finishes.
Another project may be very different and justify much less sanding. Can you sand your framing 2x4ās in the same way you sand custom cabinetry or a console table? Of Course. Does it make sense to do so? Does the job require it? Does it add value?
This message about āgood enoughā was a common theme. Builders sometimes struggle to initially ship or stop improving software solutions because they want the finish to be perfect. After all, this software is āmy babyā. It is difficult to ship when there are still great things that can be done to make it better.
Itās critical to remember what project Iām working on and what degree of finish is necessary. A smart tradeoff is stopping at good when good is good enough. Sometimes I need to stop sanding and consider the project āgood enoughā.
A good solution solves problems step by step
While I was writing this post, I saw this quote from Matt Rickard:
Sometimes you have to think step by step. Executing a complete plan is hard enough, but it might be impossible ā unknown constraints, path dependence, or other exogenous factors. Unapproachable problems sometimes become excruciatingly obvious when a subproblem is solved.
A good solution is no more complex than it needs to be
Every serviceā¦ every new featureā¦ every line of codeā¦ is a liability. I am a firm believer in keeping a solution as simple as possible.
My favorite example of this is our Edge Computing architecture. We made a ton of decisions in order to keep our solution as simple as possible while still meeting the requirements that led us to build it in the first place.
Minimal usage of Kubernetes features: we could have gone crazy with Kubernetes and done lots of awesome things, but we chose to stick to the key features that helped us solve our core problem, such as scheduling containers to run across multiple nodes for resiliency.
Simple deployment techniques: we implemented simple tools and techniques for large scale deployments, such as GitOps. We use a one-to-one Git repo to K8s cluster relationship to keep things simple.
No SLAs on data persistence at the edge: Instead of building a complex architecture that reached some number of nines, we simply decided that our cluster could vaporize at any moment (it really could) and that all applications running needed to be prepared to handle that situation gracefully. That led to simple data exfiltration to cloud with rehydration patterns that were not very complex for application teams and kept the edge stack much simpler. It also lets us wipe nodes whenever needed and treat our edge deployments as cattle not pets.
What makes these tradeoffs smart? I believe its that we achieved a relatively simple solution with minimal negative externalities for our customers.
A good solution acknowledges and accept constraints
Constraints are a function of people, process, and technology.
People: What skills does the team have? What experiences do they have? What else are they working on? What support load burdens them? Are they specialists or generalists? Do they learn new things quickly?
Processes: Are there legal hurdles? Ethical questions (AI)? Existing workflows to consider? What are the time constraints? When does the business need this solution?
Technology: What are the environmental constraints? Do I have the right technical tools? Is there a sub-problem that is historically intractable? Can the problem be solved in a cost-reasonable way?
We faced numerous technical constraints in our Edge Computing work, specifically with our restaurant network. First, let me say that we have a VERY solid foundation for our restaurant network and it the best it has ever been. That said, there were some capabilities we needed that it did not present. For example, we had no control over local DNS, DHCP reservations, or static IP assignment from in-restaurant APIs.
This forced us to make tradeoffs with our time (opportunity cost) to engineer solutions that bootstrapped our restaurant clusters from the cloud by inventing services that played similar logical functions.
Questions to ask yourself
Is it intractable? Rolling out a new restaurant network or new capabilities was not an option. A refresh had recently been completed. To continue our edge work, we had find some other way forward. Providing the services we needed was not feasible.
Can it be engineered around? Yes. We felt icky inside because of the way we had to build so many services from scratch that had much more solid and battle-proven solutions (what are the chances we do a better job than DHCP? 0%) and we sometimes felt like we were making dumb tradeoffs, but there was a path forward.
If so, at what cost? The cost was not too high. It was several weeks of engineering time but we were able to find paths forward.
What alternatives exist? Nothing at the timeā¦ it was build or acknowledge our restaurants could not support this type of architecture.
What do we give up if we solve this problem? For us it was velocity on all the other features in the platformā¦ but if we were not able to bootstrap a cluster without humans, the entire edge computing thing was dead on arrival.
In our example, we traded overall feature velocity for something we didnāt want to do, but had to accept in order for the solution to work at all. Ask these questions to yourself as you consider the challenges in the solution you are working on.
Tradeoffs vs Debt
Are tradeoff outcomes the same thing as technical or enterprise debt? It dependsā¦
Yes: If dumb tradeoffs are made due to risk aversion, lack of courage or leadership, lack of understanding of the problem, unwillingness to re-imagine the solution, or anything of that natureā¦ then yes, the outcome is really debt.
Maybe: If a smart tradeoff is made, the outcomes the solution is created for can still prevail and the constraints that were managed around can be mitigated long-term. It may create some debt but in manageable form.
No: In the ideal situation, a tradeoff is made (to manage a constraint) and the resulting solution is still well-implemented, achieves business outcomes, is supportable, etc. It has simply not done all that it might have been able to do, but has not accumulated a debt that must be repaid. This is the goal.
In reality, most tradeoffs produce some sort of debt. Remember that debt is a tool and that it can be paid off if it has a disciplined debtor. Consider that one way to manage that debt is to pay it back when the opportune time arrives, perhaps even trading off new feature work to establish a better foundation..
Towards making better tradeoffs
Navigating the complex terrain of technology choices is an art that necessitates making smart tradeoffs. As we've seen from my experiences at Chick-fil-A, this involves building a solid foundation of principles, knowing when to call a project "good enough", solving problems step-by-step, and keeping solutions as simple as possible.
As we navigate the world of building, buying and implementing software, we all must make tradeoffs. Careful consideration and disciplined thinking can ensure they're smart ones, leading to well-implemented solutions that deliver on business outcomes without amassing overwhelming amounts of debt.
I leave you with a challenge: How can you apply the principles of smart tradeoffs in your current project? What āgood enoughā solution might you be over-sanding? Are there constraints that you can creatively engineer around or simply need to accept? Remember, the key is to approach these tradeoffs not as stumbling blocks but as stepping stones toward achieving your goals.
The Chamber š° of Tech Secrets is closed. As you build the future, may your tradeoffs be smart ones. I wish you a productive and fulfilling week ahead. Until next timeā¦
Beyond the Goal, by Eliyahu M. Goldratt (The Goal - Theory of Constraints)
Technology can bring benefits, if and only if, it diminishes a limitation.
Before technology we developed rules to accommodate the limitations.
The question becomes, what benefits would any technology bring if we neglect to change the rules? The rules that helped us accommodate the limitation before the technology was there.
As long as we obey these rules that were there to bypass the limitations, we will continue to behave as if the limitations still exist.