I recently sat in a zoom call with about 10 other people, talking about a project that will eventually steal a lot of my thunder by co-opting a big part of something I have been doing, quite successfully for 5 years, using a new database using a different technology. Specifically, we’re talking about recreating my SQL database holding a bunch of engineering data with a Graph database. And, sure, it looks like there might be some advantages, but right now it’s about a wash, though I admit it may make it easier to implement things we aren’t even talking about yet. Maybe. Even if it does, will we ever get to them before the heat death of the universe? I digress. Anyway, after all these years, my successful little project has finally caught the fancy of someone who wants to take over and own the concept, and who has enough connections and political capital to try to make it happen.
Sigh.
We’ve already been talking about this for about a year, and my boss and I only got involved with this team after they had already been working on the idea for a year before that. Using contractors, it had taken them roughly a year to do a proof of concept. I created my own PoC by bootstrapping my own graph DB with a bunch of scripts to convert my existing data, and doing an initial rewrite to get my import scripts and existing web app to interface with it. For comparison, it took me about four man-weeks to validate that this could work, see some of the advantages, and discover, to my surprise, that I couldn’t really argue against the switch. There seemed to be no downsides.
Now that there are several other groups involved, and we’ve finally got a gatekeeper from the ivory tower of corporate IT to walk our project through all the different reviews and approvals and signoffs. In essence, we’re looking good to start bringing up development environments in, oh, I don’t know, maybe another year or something. Honestly, we’re probably 3 years away from a working product. And if and when we finally get all this put together, and successfully run through the IT gauntlet, this new database — which is being designed for any number of arbitrary applications to hook into it — will have become part of the machine, and every other application that may try to use it will become mired in the tar pit when they try to get access to it.
As I sat in the call, I struggled to express how to characterize why this achingly over-complicated structure and process exists. But then I stumbled on this tweet:
This results in companies that are process-oriented, rather than result-oriented.
The purpose of a process-oriented culture is to dilute responsibility so no one can be held accountable for failure.
Now, I had understood the second part of the quote for a long time, but the first part was the expression I was looking for. Process oriented, as opposed to results oriented. The rigamarole I’m going through is process oriented. The “work” we are doing is for the process, not the end result. The end result is an after-thought. A by-product. It’s not the goal.
Out of 10 people on the call, there were just two of us who had the slightest concern about having the working product at the end of the process to serve the needs of the business. Everyone else was there to be a cog in the machine which we will be required to crank to get this done. Diagrams to make. Reviews to complete. Documentation to verify. Web forms to fill out. Spreadsheets to create. Approvals to be granted. Resources to be assigned. Meetings, meetings, and more meetings.
None of them understand what my application does, how it solves a business need, why it’s a critical part of the engineering process now (which is a whole post on its own), or why it’s architected the way it is. None of them care about swapping out the database layer with something else, why that is attractive, or what it may help us do in the future. No one is curious about any of this, because that’s not what they’re paid for. They’re paid to check boxes that some high-dollar consultants told the company had to be checked when doing this kind of work.
At every step in the journey, there are entire departments that exist only to add to the complexity of the final project. Middleware. My god, middleware. Middleware between the app servers and the database. Middleware between the application and other applications. Middleware between the database and other databases. And then, after everything is setup according to all the white papers, every change will need to be approved by committees, and moved through each environment in turn.
Everyone involved from IT can point to “security” as the reason for all the hoops we have to jump through, but I hardly think that’s a fair assessment. The data we’re storing is not sensitive. It is not financial. There is no need for separation of responsibilities. It is not personal. There’s no call for privacy. This isn’t real-time data. There is no need to make it highly available. It’s just historical engineering data. There are no particular “secrets” involved. It’s just data. Mountains of data. It reveals nothing about our IP. It is useless without the code which is tuned with it. And even if it weren’t, it’s also prescribed by our particular hardware and implementation, which someone else wouldn’t be using, and very likely can’t. All in all, it would take a competitor far less time to generate their own data for their own code and hardware from first principles than to figure out what ours means. But we’re going to treat it as though it were literally all of these things, and apply every system of control and management at our disposal to it, because that’s just how things are done.
This whole thing is the literal antithesis of Agile. Which is only appropriate, since this company wrote a whole book on waterfall 25 years ago — which I actually reviewed as part of a consultancy, in a life-is-strange kind of coincidence — and despite using some new lingo here and there, every reporting structure and process in IT is still organized around this methodology.
Everything hinges on having a project number so that all of this internal IT work can be assigned a cost, and the money making departments can be charged. As if this were a thing that needed to be done in order to help the business meet the needs of, you know, their customers. This pernicious, stupid practice of internal “cross charges” effectively ended US leadership in manufacturing decades ago, yet here we are, still not imagining doing it any other way. Corporate America has still never gotten the memo, and now this managerial malfeasance strangles IT right along with engineering.
If you want to adapt to the times — er, adapt to the times 15 years ago — you can’t just throw the word “agile” around occasionally, and call it good. And you can’t even just change process. You have to change organization. The thing that’s implied by the “process oriented culture” is that these systems become entrenched by organizational structures, which give people with no skills or ideas a “kingdom” to rule over.
They do not exist to help in any way. And, boy howdy, they don’t! They don’t look at your project as subject-matter experts and point out things that could be done to help. They only exist to go through their checklists, and force every part of your stack to adhere to every known buzzword, regardless of applicability. They don’t even know why! They can only say no. They can only slow down the process. They can only hinder the implementation of a good idea. Unless you break up this ossification to align with modern principles, every new idea will continue to get put through the same soul-crushing process, waste literal man-decades of time, and take years longer to implement than it should.