I have been wanting to write this for some time now, and was inspired by Vicki Boykis' great blog post "IT runs on Java 8" - which is true and funny for those of us who know exactly what she is getting after in the article.

I want to stop you early, though - if you are in startup land writing amazing new machine learning bits with Python and compiling your new front end to Web Assembly using Rust, because your lead time to market is probably very short for that startup you're working for: DevOps is a cultural shift in systems management that many companies will never be able to successfully implement in the next 5 years.

I hate the term DevOps; to fractionally quote the movie 'Love Actually,' - "it covers all manner of sins, doesn't it?"

It's true though. Even DevOps teams often can not really tell you what it means to any other team in the world, but I will share a list of things that it might mean, based on people I actually know:

  • Systems Administration
  • Release Management
  • Test Running (but not deployment)
  • Development of Unit and Integration tests
  • Developer (bugfix only)
  • Developer (SCM flow)
  • Developer (just developer)
  • Infrastructure Architect (cloud only)
  • Infrastructure Architect (virtual only, no cloud)
  • Infrastructure Architect (hybrid)
  • Front-end tool UX designer
  • Configuration Manager (enforces configurations, but does not write)
  • Configuration Manager (writes configurations, but does not enforce)

In fact, where the term DevOps is applied, as you can see from the short list above, the role might actively exclude a current job role from its scope!

And while that is part of the problem - whatever the massive employer-based schizophrenia related to whatever they lump under the "DevOps" umbrella is, the bigger problem is culture. Legacy culture ties itself to legacy employers, and as I walked away from a conversation with some quite sharp gentlemen last week whose role it has become to introduce proper DevOps practices to organizations across the board, I concluded the issue goes much deeper: people, especially STEM types fall generally in two categories: those who embrace change and those who oppose it. I have rarely found those who walk the line.

I propose we walk through a scenario that could be entirely real, but for our purposes is made up and based on hundreds of other similar scenarios that have played out since the term DevOps became somewhat more recognized around 2011-12. DevOps was primarily pitched at a leadership level at that time as a way to increase a software release cadence.

Based on several "legacy" companies with storied histories of greater than 70 years that I have worked for, I shall break down some the functional groups I am aware of still existing during a software release some years later:

  • Change Advisory Board
  • Development (multiple groups with multiple responsibilities, in reality)
  • Database Management (the database admins)
  • Systems (architecture, engineering, and support)
  • Release Management (an offshoot of the larger Project Management group)
  • Quality Assurance

With these groups in mind, we can break it down further: Development and QA are constantly at each other's throats. QA constantly fails things that get pushed to the QA environment because the feature changes break the environment, or they do not meet the business requirements, or the feature to be tested can not be tested before another feature is completed on which it depends, and all sorts of fun problems.

Release Managers are constantly irritated at the Developers because the bugs to be fixed were eschewed for new features, but the new feature is also buggy. They are also constantly annoyed at the QA team for failing features and bugfixes, even though the QA team is probably right, because it impacts the release schedule.

DBAs are annoyed because features require a schema change, but a schema change requires separate change board approval from the feature approval. By the same token, the DBAs are also annoyed because the schema changes, if they do get approved, can only be completed with the assistance of the systems team who has the boxes so locked down internally that accessing them requires some sort of share session.

Everyone is annoyed at the CAB which for some reason only meets the day before a full moon, and the meeting is skipped if the full moon happens on a Sunday.

Most of this is a bit extreme, but it is truly rooted in realities I have been a part of in the past. A huge problem is that there is a cultural animosity to overcome that has developed as part of the long-lived practices of the past. That does not change overnight.

Hiring a "DevOp," as one VP I encountered put it, does not magically fix the relationships (and all manner of their sins) and make things better. It is not like when a five year old gets a superficial "owwie" and you put the Dora the Explorer band-aid on it and suddenly everything is better for everyone.

So problem one: cultural animosity.

How about problem two? In this scenario, it is the CAB. That may not be the case for everyone, but the CAB which has ultimate control over feature sets, bug fixes, and release cadence is a blocker. If there is a highly requested feature that comes up the day after the CAB meets and it is shelved until the next CAB, where it is debated and either accepted, rejected, or tabled for a future CAB meeting, you have people that are not promoting change but hindering it.

Problem three could be the release process itself. I worked for a company whose release process could top 14 hours because of the number of steps involved and who was allowed to execute them. Switching the load balancer pools to the backup pool? Networking! Shutting down the application for patching/upgrades? The Application Support team! Schema changes - the DBAs! I will not list the entire process, but it was exhausting.

Problem four: funding. In legacy companies, despite each of the above groups contributing to the revenue of the company through work completed, they are seen as an expense driver, often underfunded for equipment and manpower. Some of the worst I-just-don't-care-but-they'll-have-to-fire-me-before-I-leave burnout has been in those situations where the employees who are generally smart and capable people have accepted that no matter what they do, they can not win, but they can collect a paycheck. Generally due to the volume of work (when the CAB approves it) being so massive that they could not complete it on a year-long schedule, let alone a two or three week sprint.

Problem five: expectations of magic. When management starts pitching magic to the employees in the variety of "this will solve all of our problems" you can either drink the koolaid or collect a pink slip. The sins here could be any number of things: virtualization - circa 2006, hyperconverged infrastructure - circa 2010, "the cloud" - circa 2013, or "DevOps" - circa now.

None of these problems are unsurmountable, but the issues have to be tackled over time, though not necessarily in the order I proposed. The "DevOps" reality for legacy IT is that you are starting to bind job roles together more tightly and cohesive teams with a variety of positions are more likely to succeed.

Let me use a sports analogy: If I have four groups of people who are active on a football pitch, but only at one time, can they be effective? I have a group of goalies, a group of defenders, a group of middies, and a group of forwards, how successful can they be? Perhaps very successful of both teams only have goalies on the field - it is unlikely any points will be scored. The opposite is true if both teams only have forwards available - it will probably be a high scoring game. But what if the game itself requires you to coordinate? If, once the ball moves past a certain point, you have to take all of your forwards off the field, and put your middies in. Say the ball gets past the midfielders - then in go your defenders, and finally your goalies. The process itself becomes arduous and the chance of success falls very low.

Now imagine that you have a team with a few forwards, a few midfielders, a few defenders, and a goalkeeper. If you have a couple of alternates to sub in when your players get tired, that is even better. Now you have a real football team!

So what if legacy companies stop grouping everyone solely by function? Then it starts to look like a successful DevOps landscape. You do not need someone called (ahem) a "DevOp," instead you have a team oriented around driving your technical success forward. For example I have a development team designed to drive a variety of features (or microservices, these days): A two backend developers, a front-end developer, a systems admin, a database admin, and a project manager, along with a manager who serves as "coach" - the team itself can wholistically produce a fully realized single feature or microservice, and the lines between delivery and function are reduced.

Back to reality, however: this is a fight that needs to be taken up at an executive level in Legacy companies. In order to compete with startups that think like this from day one, that shake up needs a champion from the C-levels and must be supported (or enforced) all the way down the line with the knowledge that people get set in their ways and resist change at every opportunity because even though they know it is the right thing to do, change is hard.

I have skipped over nearly everything: proper equipment, knowledge of new practices, the imperative requirement of testing, testing, testing - and documentation! What if someone gets hit by a bus? Who takes up the cause?

The reality of DevOps in "Legacy" IT is this: it will be a long, difficult, thankless task to change the cultures, find the funding, and develop the knowledge that is required in order to succeed. That takes time.

If you start taking piano lessons today, in five years there is a good chance you will be able to play the piano decently so long as you stick to it. The same goes for learning Chinese or building a boat or implementing DevOps practices. I have picked things that sound hard to me to pair with implementing DevOps practices because, in my mind, that is also hard. It is also, to me, reality - and worth doing.