In his recent article "A Eulogy for DevOps," Mathew Duggan reflects on the origins and evolution of DevOps and its state today. According to him, the DevOps and infrastructure space has grown overly complex for most organisations. He wishes we return to designing solutions to match current business expectations rather than the potential of infinite growth.
Duggan also briefly talks about the increase in workload DevOps has brought to engineers involved in software operations, which prompted me to reflect on the expectations placed on engineers in the software industry and the implications of these demands.
What is expected from engineers today
These days, as Duggan pointed out, sysop/cloud/DevOps/SRE/platform engineers (for simplicity, I’ll call them ops engineers) are expected to know more than just system administration. They are expected to understand a bit of database administration and network engineering, which used to be more often separate roles.
An engineer working in the field is also expected to know how to operate a multitude of distributed software: container orchestration platforms (Kubernetes), build/deployment platforms and pipelines, and observability platforms. On top of that, they are also expected to know enough software development to build ops tooling and infrastructure-as-code as well as statistics to understand what to do with the observability tooling. Engineers working on developer platforms also now need to know how to build useful UIs for provisioning software.
I don't think the expectations are any lower in other engineering disciplines either.
Today, software engineers are regularly expected to know how to operate their software. Not only does this mean that they are expected to know more details about the operations topics mentioned earlier, but they are also expected to participate in operations duties such as on-call.
Similarly, many QA engineers of today are expected to be able to write software for test automation, integrate them in CI, and even deploy the software.
The economics behind expectations
I believe a big part of the increase in scope of expectations has been caused by the adoption of the cloud.
Cloud platforms such as AWS make it very easy to build an adequate software stack without deep knowledge of the details. There are templates available to spin up a stack that contains the network, databases of various kinds (object, document, relational), a container orchestration platform, monitoring, load balancing, and more in less than an hour. Of course, deep knowledge is still very useful for knowing how to optimise the setup for efficiency and maintenance, but the barrier to entry has been significantly lowered.
Cloud platforms may also abstract away many of the detailed tasks we used to do. If you’re leveraging hosted solutions from the cloud platform for running software or databases, you most likely get details such as scaling and backups preconfigured. I don’t remember when was the last time I did configurations for Linux machines.
Therefore, many companies no longer have enough work to make it economical to employ specialists in things that are now considered subfields of cloud operations. As a result, more and more of the responsibilities are piled on the ops engineer. For many companies, it's now more economical to hire generalists who can learn on the go and are willing to take whatever challenge you hand them instead of specialists focused on one specific thing.
Is this sustainable?
Personally, I have found it useful to learn about all aspects that go into delivering software, all the way from writing it to running it. I believe that being a generalist has improved my software engineering skills. The idea is that since I can understand more in breadth (e.g. more components in the round-trip of customer interaction) and depth (i.e. details on how each component in the interaction works), I’m better able to pick the solutions that fit well together. In other words, the added expectations in operations has provided me with some benefits I may have otherwise overlooked.
On the other hand, I keep wondering how sustainable it is to pile so many hats on engineers.
An increase in responsibilities also increases cognitive load and context switching, both of which have negative effects on quality of work, job satisfaction, and mental health.
As an engineer today, it’s common to not be able to focus only on one topic that you know well and are prepared for. You are required to learn something new on the spot fast and almost constantly. For example, as a software engineer, your main goal and knowledge are built around translating business requirements to working code, but now you’re also expected to somehow understand the nuances of well-architected infrastructure code as well as deliver it. In this case, an engineer will constantly hop between software and infrastructure code, which often use different tech stacks.
The reality is that not everyone has the time to work on all of the pieces they’re expected to work on these days, which leads to inconsistent quality. For example, as a software engineer, you are likely to take risks by either copy-pasting infrastructure solutions to “get the thing done” or spending less time on testing the code, which often leads to faults and performance issues surfacing later. Similarly, on the ops engineering side, because you are dealing with a multitude of infrastructure components, you do not have enough time to build a deep understanding of the components to optimise them beyond the most basic settings, which results in inefficient or costly infrastructure.
Not everyone wants to learn new things constantly either, especially the things that are outside their core skill set. Instead, many people prefer to focus on mastering their core skills, and wish to be given time to do it. When you are continually trying to learn on the spot and switch between the things you are doing, your productivity drops, and faults in your work increase, which negatively affects your self-esteem. The things you learn, that are outside your core skills, may not help you develop as a professional on the next job, which reduces job satisfaction.
In summary, I believe a lot of engineers right now miss consistency in their work and are now burning out because of the constant barrage of new expectations.
Where do we go from here?
It is too late to go back and bring back the hard separation of duties we had before DevOps. That ship has sailed.
However, perhaps there's a chance that we can still lower the load on engineers by reducing the complexity of the systems we build. Perhaps we don’t need everything to run in a Kubernetes cluster or a microservice architecture that scales infinitely. Perhaps it is fine to run a single database instance for all our data storage needs.
Instead of building more interconnected systems composed of more abstractions, perhaps we can dial things back to a handful of systems running on high-performance hardware, and rethink how we develop, deploy, and run software. A lot of the techniques we use today for distributed computing patch over the lack of compute capacity and inefficient runtimes from years ago, but we've forgotten that those have developed alongside the techniques. There is a lot to uncover on this topic, so I’ll reserve it for later posts.
What are your thoughts? Do engineers across disciplines have too much on their plates these days? Are we on the road to coalescing the different engineering disciplines into just a single generalist role? Let me know what you think.
Thanks for reading!
