The Chamber 🏰 of Tech Secrets is open. This week I was in San Francisco for Datadog Dash to talk about Observability at the Edge. I’ll share more about that next week. For this week, let’s talk about local optima.
Climbing the Highest Mountains
Imagine you are a member of an expeditionary team that is seeking to climb the highest mountain in the region. These mountains are in a dense, tree-covered jungle. It is hot and humid and the horizon is often obscured with light, fluffy clouds. Intel about the area is hard to come by, but the team believes they know where the highest mountain is. After weeks of climbing and descending smaller hills, the team approaches what they believe is the highest mountain in the area. They will be the first to conquer the mountains of this jungle. They slip and slide as they climb muddy pitches and slippery rocks. Finally, they reach the top, which provides a narrow, directional view over the surrounding landscape. It appears there is no higher mountain in sight. The team celebrates the accomplishment with a bottle of warm champagne that they have been carrying for this moment. They have done it. They have reached the highest point… of their journey.
Unknown to the team, a scientific expedition (Team 2) has been mapping this jungle, including its mountains. They have access to more sophisticated technologies and tools that have given them the ability to identify all of the peaks across the jungle, though they are less certain about the perils and challenges of journeying through the terrain. As it turns out, Team 1 is a long way, across numerous valleys and mountains, away from their true objective.
The Local Optima Problem
Team 1 has reached a local maxima: the highest and most optimal point within their local area (that they were aware of). In order to reach the true maxima, they would have needed to descend and ascend several more times into even more new territory to reach the highest peak in the area.
This is the problem of localization, which is a barrier to optimal (or global) solutions.
A similar problem occurs for teams building products and solutions within the multi-dimensional space of a multi-team environment, such as an enterprise. The solution local maxima — the best decision for a group working on the problem — might not be anything like the best possible solution when you take a broader view. It is even possible to miss the mark completely. To use our metaphor, the highest mountain in the local area may be small compared to those looming behind the clouds.
This problem is a function of missing context. If our heroic Team 1 expedition knew that there was a higher mountain, they would have likely set their sights on it instead. If Team 2 knew someone was looking for the highest mountain, they would have happily shared that the team was headed in the wrong direction. Team 1 was intimately familiar with traveling over the terrain and reaching objectives. Team 2 had a broader context and closer-to-perfect information that enabled them to see local optima for what they were.
Some Techniques to Avoid Local Optima 📈
In machine learning algorithms like regressions, techniques like stochastic gradient descent are used to help the algorithm avoid getting stuck in local minimums. What techniques can we apply to avoid getting stuck in local optima with software solutions?
Include outside influencers: To avoid local optima (max or min), teams need decision makers or influencers that can take an outsider view and consider the goals of multiple teams at once while finding the best path forward for the organization. This is often the role of Staff Engineers or Enterprise Architects who, by nature of their role, have the “Team 2” type of context and can help teams avoid locals.
Ask for beta: I am a big fan of going on adventures, visiting national parks and climbing high mountains (Grand Teton climb coming in a few weeks). Several years ago, I started visiting the highest point in each US state, which you can read about here if interested. “Beta” about routes is important. Knowing where to start, which routes pose which challenges or what mountain to set your sights on are critical to a successful trip. Teams can ask for beta from those who have been on the same journey in the past. Organizations can make finding beta easy with tools like Backstage.
Invite secondary stakeholders in: One issue that occurs in enterprises is that a team can get overly focused on a a particular stakeholder’s needs. In my experience, this is especially true when the organizational structure says that stakeholder is who they are here to support. This can lead to a myopic approach to solutions that ignores the needs of other stakeholders in the company. A few examples of this are not making APIs for functions others need, not sharing data in a reliable timely manner, or ignoring the parts of a business process that are outside of the team’s purview. Inviting other stakeholders to give requirements and feedback can help with this.
Reverse Conway: Conway’s Law says that organizations will produce architectures that reflect the organizational structure of those producing them. Design an organizational structure that reflects the type of systems outcomes you wish to see.
Recognize the power of incentives: People respond to incentives. Do the incentives encourage local optima, or global? What do you celebrate? What do you reward?
Proactively Defined the Organizational API Surface: Kubernetes thrived when the API stabilized and people could begin to depend on it. Organizations would do well to proactively define their API surface and ensure it is complete while not becoming overly bloated. A well-defined API surface, accompanied by a clear division of ownership across teams, builds a logical framework that tips teams towards working on the right things for the organization in close enough to the right ways.
This is certainly not an exhaustive list, but covers several of the techniques that I have seen work first hand. When these approaches are absent, it is easy for myopia to set in and lead to team “successes” that are not the best possible outcome, but simply a local optima.
Refuse to settle for less than your best 💪
Every year, the best CrossFit athletes in the world compete in the CrossFit Games. I have been in-person in Madison the last two years, but took a year off this year and watched online. The man in black, Roman Khrennikov, led the field for most of this year’s competition. Unfortunately, he broke his foot and tore a ligament in the event prior to what you will see below. Nobody would fault him if he withdrew or “participated” but just stood at the start line.
Instead, we see struggling through pain and a demonstration of maximum effort until the clock stops. Why? “I want to be a hero for my son and have him see me fight to the end“. 🤯 Jeffrey Adler, who goes on to win the CrossFit Games, stays side by side encouraging Roman along until the end. Whatever it is you are working on… do the job and do your best. Others are watching.
By the way, Roman ended the games on the podium and in third place, even after finishing dead last in the final event of the day. 👏
The Chamber of Tech Secrets is closed. Thanks for reading! 🙏 Have a great week.
I did not appreciate the power of incentives until I read “Freakonomics”.
“This book, then, has been written from a very specific worldview, based on a few fundamental ideas:
Incentives are the cornerstone of modern life. And understanding them—or, often, ferreting them out—is the key to solving just about any riddle, from violent crime to sports cheating to online dating.”
There are three basic flavors of incentive: economic, social, and moral. Very often a single incentive scheme will include all three varieties.
— Freakonomics Rev Ed: A Rogue Economist Explores the Hidden Side of Everything by Steven D. Levitt, Stephen J. Dubner
https://a.co/2A7yAlm
Love the metaphor and richly relevant picture here, Brian! I think it was implied, but maybe more explicitly stated: that second expedition team, having spent their time building great expertise in surveying broadly and in analysis, likely is more poorly equipped to do the mountain climbing than the first team, but is far better equipped to tell which mountain to climb. There is a certain humility and confidence needed for both roles: for team 1 a humility that is required to admit that you might be on a local optima but incredible confidence to know you can summit the highest peaks once identified, and for team 2 a certain confidence to share your surveying insights with others and a level of humility to admit that your breadth of awareness is best complemented with hands-on summiting expertise. Love the thinking!
The org incentives are super real. Excited about the idea of “Reverse Conways law”!