“Show me a hero and I'll write you a tragedy.” - F. Scott Fitzgerald
Many organizations stumble on their DevOps journeys because, even as they focus on breaking down silos, the teams formed to champion DevOps end up becoming silos themselves. Here's a look at how these DevOps heroes are created, why it’s problematic, and a few techniques to help you avoid this trap.
Imagine a company with a fairly large, established IT department. The company has set ways of managing its software releases but would like to go faster, so it sets out to embrace DevOps. Because DevOps practices often require skills and experience in specific automation tools and techniques, the company focuses on acquiring those skills. The organization either trains existing employees from the development or operational teams, hires DevOps experts, or pays external consultants.
The company then sets up a small DevOps team that includes a handful of experts. Although this is a natural first step, it is a common antipattern that will eventually cause issues.
Next, the DevOps team sets about understanding the existing release process and identifies areas that can be automated. It introduces virtualization technologies that make it easier to provision new server space. The team values having infrastructure-as-code, uses configuration management tools to create scripts that set up the company’s servers, and spends time creating a continuous delivery pipeline that deploys new code into production automatically.
Before long, the company seems to have taken great strides: Code is moving into production faster and more reliably, and the lead times for new infrastructure and disaster recovery have been significantly reduced. This is a great win for the company in the short term, but it often leads to long-term problems.
The three downfalls of DevOps heroes
1. The go-betweens
The most common cause of pain and a lack of innovation in an organization's release process is a lack of communication between IT departments. In many medium-to-large companies, the developers don’t really know the people who look after the applications in production. In large corporations in particular, there may even be separate teams looking after tasks as specialized as networking, operating systems, and certificate management.
The company’s DevOps team must navigate these different silos to perform the tasks described above. Team members need to meet the networking expert, buy coffee for the person who manages certificates, and spend time talking to the operating system team about the new "Star Wars" movie. They’ll build these relationships and learn what they need to know about the company’s existing infrastructure and processes. They’ll need access to repositories and software, and they'll need help understanding where to look for the configuration they want to automate.
Because the DevOps team is small, it's able to build these relationships quickly. The problem is that the rest of the organization is still siloed. The developers still don’t know who configured the operating system on which their code runs, and the networking experts still don’t know what the developers are working on next.
So the DevOps team becomes the go-between. The developers ask it to organize a new SSL certificate or firewall rule. The operating system team asks it to get the developers to upgrade the version of SSH they’re using. Now the DevOps team has become another silo. What’s worse, the team's time is now taken up aiding communication between the other silos. The company ends up just as fragmented as it was before, and DevOps team members become increasingly annoyed at the new role they’ve found themselves playing.
2. Key-person dependencies
Another problem is that, very quickly, the DevOps team becomes too invaluable. Team members are the "heroes" who manage to make big changes quickly—but now they’re the only people who understand how it all works. The developers still don’t know exactly how their code ends up in production. When a deployment fails, they call the DevOps team to come debug the continuous delivery pipeline. The operational team doesn’t know how the configuration management scripts work, so they call the DevOps team when they have additional settings to deploy. Most annoying, it’s now DevOps team members who get phone calls at 1:00 a.m. when a deployment fails or a server runs out of memory.
3. Implicit hierarchies
These DevOps folks seem to have some neat skills. They automate everything and use all kinds of exciting tools. They’re even using Docker. People see the DevOps team as a desirable place to work, and many IT employees would like to be part of the action. Suddenly, joining the DevOps team is seen as a form of promotion. Some companies may formalize this hierarchy, but even when it's implicit, it's harmful: Instead of spreading DevOps practices throughout the organization and helping everyone gain new skills they can use in their existing teams, the DevOps team unintentionally becomes an elite pocket of innovation, leaving the rest of the company to stagnate.
This effect is compounded by an apparent shortage of DevOps skills in the industry as a whole. Your team's DevOps experts become highly sought after, which leads to high staff turnover, as employees who learn new DevOps-related skills are snapped up by companies with bigger budgets or more innovative work environments. Meanwhile, the rest of the IT department, which has not been privy to these new skills, is left high and dry.
A better way: DevOps without heroes
The key to avoiding these pitfalls is to look past those shiny new DevOps tools and focus on the culture of your organization. At its core, DevOps culture is all about getting people to talk to one another and work together. When people work together, they are able to avoid unnecessary work, find solutions that work for everyone, and build a common understanding of what it takes to build and maintain software in their organization.
Avoid the temptation to impress quickly through automation. Changing how an organization functions is slow work. A gung-ho approach to automating everything right away may impress whoever is paying for the DevOps transformation, but it won’t help much in the long run.
Instead of forming a DevOps team, encourage your teams to build their own DevOps culture and capabilities. Encourage the networking and operating system specialists to spend time with development teams to learn about what they need, and encourage the developers in each team to start automating small tasks as they learn more about parts of the release process in which they’re not normally involved. This may seem counterintuitive initially, but you need to go slow at first so you can go fast later.
You may still have a need to hire, train, or bring in some DevOps experts to bolster the organization’s automation skills. But instead of allowing these experts to hide in an innovative corner of the organization, consider embedding them in the development and operations teams. In this way, everyone in the organization can become more skilled in DevOps practices and build an appreciation for what it takes to establish a DevOps culture. The process will proceed much more slowly, but you'll end up with a much more positive, sustainable transformation where everyone is part of the solution.
DevOps is largely about building relationships between people. Coffee and cupcakes may be much more effective DevOps transformation tools than all the shiny new virtualization and infrastructure-as-code tools.
Image credit: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
