DevOps Ranching: Horse v. Mule

3 min read

I think it's safe to say nearly everyone in the IT world has heard of DevOps, and fully 3% of those people actually understand it. I want to start with a little personal anecdote. Almost everyone thinks that DevOps is about speed: reducing time to delivery through automation and process improvements. It is, but it's also not guaranteed.

A month or so ago I ended up in a discussion with a CIO on a flight between Chicago and Toronto and we got to chatting, and he said that his team was 'converting to DevOps' and they were going to hire some 'DevOps Engineers' to help out with that. Of course, having been to this party previously, I asked him what he meant by DevOps.

"You know, some systems guys... that can make things faster."

So of course, I asked him if he had systems guys now. "Sure, but they're slow," came the reply.

"So to your thinking, your current systems folks are slow, and you want to bring in DevOps engineers to make them faster?" This is a wonderful technique called mirroring that many people do all the time, but is specifically called out by name in Chris Voss' great book titled Never Split the Difference.

The CIO looked at me grinning and said "Yes! You get it! I can't get anyone else to understand."

So I decided to blow up his happy win with the question of "But have you identified why your systems guys are slow?" whereupon the the grin faded.

"Well, they say that they're shackled by paperwork and approvals, and it takes 3-4 weeks to get new purchases approved, plus they say there aren't enough of them so they're constantly behind. They also say that our developers don't want to use new tools, so it takes a long time to get things to deployment, and the deployments take 10-12 hours."

Mirroring, again: "So the organization is resistant to change, there's lots of red tape, and the team is understaffed?"

"Yeah, that's right." And wow, Chris Voss, "that's right" is so magical the first time you actually hear it.

"So what will DevOps engineers help you solve that your team couldn't solve if the red tape was removed, someone championed the changes, and the team were staffed properly?"

I got a blank stare, but then, "Well... they're faster!"

"But how can they be fast if the organization refuses to do the things necessary to allow them to be fast?"

The conversation died off about there. I'm not going to lie. I think the guy had just really never recognized that the problem wasn't necessarily the team, but the organization itself throwing up roadblocks. I gave the guy my name and email, and asked him to reach out to me and let me know how things went, but I hope that he recognized that while names are important, names are not necessarily the solution.

So that brings me to my real topic of "DevOps Ranching."

Systems Reliability Engineering or DevOps tends to lean towards speed of delivery, and so I think it could could be said easily that if we were comparing our team members, we'd prefer they'd be horses - right? They're fast, get where they're going with some guidance, generally without problems, but may need special care and feeding. A good horse is an amazing companion that can really get the work done. Some down sides are that they can't carry much. Some are thin skinned. They can be easily distracted and wander off if no one is paying attention.

What about Mules? Mules carry more weight, are fairly intelligent, they tend to be thicker skinned, and interestingly are good at repetitive tasks. They can be stubborn and set in their ways, but they can accomplish quite a bit more - albeit sometimes at a slower pace.

In the end, does a good team of SREs or DevOps-oriented folks consist of horses or mules? In my opinion, both. There are lightweight, quick tasks that need to be done by your 'horses' and there are heavy lifting tasks that need to be done by 'mules.' You can try to do the heavy tasks with your speedier people, but you may find that they're constrained by how much of a load they're carrying, and actually end up being slower to complete than your heavy lifters. Alternatively, you can still pile light tasks on the the heavy lifters, but that won't necessarily make them faster.

So what does all of this happen to do with the CIO from earlier? Simple: nothing gets done if all the workers trapped in a barn of organizational, financial, and staffing limitations. If the team can't even get out, how does it become fast? Open the gates, knock down the walls, have a right-sized team ready to go and you'd be surprised how unnecessary bringing in people from the outside that have "DevOps" on their most recent title actually are, and how much one's own team might be capable of just given the freedom to do their jobs.

Image of Stephen Sadowski

Stephen Sadowski

Leader focusing on quality, delivery, technical debt management, and leadership education about DevOps and SRE practices