The Chamber 🏰 of Tech Secrets is open. This is part one of a multipart series on why Information Technology and software delivery is so difficult.
Why is operating an application so difficult?
“Modern” software development is nearly 60 years old. The “C” programming language was completed and operational in 1972. I wrote my first lines of code (BASIC) in high school in the late 90’s. Cloud Computing began when Amazon released S3 in March 2006. DevOps has been around since ~2009.
Yet building and operating software is seemingly as difficult today as it has ever been. It might be more difficult. In this series we’ll explore why building, securing and operating applications is so difficult after so many years.
The Enemy, Entropy
A few weeks ago I closed on a new home and have been getting moved, unpacked, and settled. This weekend, I deployed 160 bags of mulch to my yard and hauled off an entire trailer of branches and bush clippings from around the house. 🤯 It looks a lot better than it did, but there is still more to be done over the coming weeks. I also have a swimming pool that is amazing, but needs consistent cleaning and water management to keep the pH levels in balance. As you might guess, the inside of the house needs to be cleaned regularly as well.
Nobody wants to do maintenance, but we all have to. The word actually makes me shiver a little as I see my time for hobbies systematically destroyed. We would all prefer a world without entropy, but that is not the world we have.
It is no different in the world of software. Maintenance is difficult and mostly unappreciated work, but its critical to having and… maintaining… a successful system.
What do I mean when I say “maintenance”? Anything that pays back technical debt. Patching, refactoring, documenting, updating and perfecting tests, rearchitecting, or investing into reliability and scale.
A few observations
Maintenance has a cost: Everything has an opportunity cost. For yard maintenance, it meant trading off other hobbies or relaxing. The same is true with software. We have to trade off feature delivery for maintenance activities.
Make maintenance routine: Don’t let it get away from you. When it comes to a yard, its much easier to continually do small amounts of work instead of spending an entire weekend (or months of weekends) catching up on things that need to be done. Big maintenance tasks are more grueling and harder to get motivated to undertake. With software this is true as well. Big maintenance tasks (like upgrading from version 1.6 to 4.2) can also be much more risky. Small, continuous change and maintenance is key to reliability and therefore long term success.
There’s nothing wrong with outsourcing maintenance: You don’t have to do all the maintenance work yourself. I pay someone to cut my yard, trim some of the bushes, pressure wash my outdoor fireplace, and so on. Outsourcing maintenance work does not mean you have not done “DevOps”, whatever that means these days (future post, I have lots of opinions). If you can outsource standard maintenance tasks from a budgetary perspective, its probably a smart move. One trick is to not be too cute in your design such that others can’t easily augment you in operations of your system.
Everything grows even (especially) when you do nothing: My grass grows every week. Tree limbs that were pruned back have grown from the same shoot but in a new direction. Mulch fades and weeds grow. With software, we weeds grow in our application, too. Software packages need to be upgraded over time to avoid the major version jump challenges in the future. New vulnerabilities are discovered post-deployment and need to be update (meaning running through our tests and such again).
Leverage technology to help you with maintenance: I am leveraging technology to help me with maintenance wherever I can. A Roomba cleans my hardwood floors on the main level of the house. A robot named Naughty cleans my pool floor, walls, and waterline a few times a week. Technology saves me a lot of work. In software, we can leverage automation to reduce our maintenance. For organizations doing DevOps, this will likely revolve around smart developments in pipelines to address potential maintenance tasks proactively. Good infrastructure as code practices can help here as well.
Maintenance is demanding: Maintenance work is tedious and tiring. Make sure to appreciate and reward the people who live in the world of frequent or regular maintenance. It is not visibly business impacting but is one of the most important things in [running software | digital transformation | etc]. People respond to incentives.
You can always buy a condo: Nobody made me but a house and undertake the maintenance that comes with it (speaking about outside in particular). I could have bought a condo (again). Nobody says you have to build your own software, so you could just “live” in a multi-tenant SaaS environment instead. In many cases, that’s a great choice. As with all things, this is ripe with tradeoff decisions.
Why is this all so difficult? It’s hard work. It takes careful planning, attention to detail, and hours of time. There is always something more fun or gratifying to do. But if we want to have a long-lasting, appreciating-in-value home it must be done. If we want a long-lasting, reliable, business-impacting application it must be done as well.
The Chamber of Tech Secrets is closed. Thanks for reading and see you next week. 🙏
This was a great quote on the value of K8s along these lines.
“The most misunderstood economic value of Kubernetes deals with complexity. The innovations from Microservices and Cloud Computing has created Cloud Entropy (Disorder); versions become out of data; services run and never shut down.
Kubernetes is an entropy killer by providing a way to get a hold of the complexity by keeping track of performance, availability and security.
The design of Kubernetes as a combination of microservices and small control loops is an example of control through choreography — achieving a desired emergent behavior by combining the effects of separate, autonomous entities that collaborate.”
Your post was excellent. It made me think of this analogy I heard of legacy technology, processes, engineers as archeological layers upon layers that build up over time. Uggg...
Maintenance and redesign is necessary, but 😢😭...