The Pain Zone… is an enterprise data warehouse platform. At the small scale we operate at, with little loss of detail, a data warehouse platform simply means that we copy a bunch of text files from different systems into a single place every morning.
The word enterprise means that we do this in a way that makes people say “Dear God, why would anyone ever design it that way?”, “But that doesn’t even help with security” and “Everyone involved should be fired for the sake of all that is holy and pure.”
For example, the architecture diagram which describes how we copy text files to our storage location has one hundred and four separate operations on it. When I went to count this, I was expecting to write forty and that was meant to illustrate my point. Instead, I ended up counting them up three times because there was no way it could be over a hundred. This whole thing should have ten operations in it.
…
Almost every large business in Melbourne is rushing to purchase our tooling, tools like Snowflake and Databricks, because the industry is pretending that any of this is more important than hiring competent people and treating them well. I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets. It would take me a month tops.
I’ve known for a long time that I can’t change things here. But in this moment, I realize that the organization values things that I don’t value, and it’s as simple as that. I could pretend to be neutral and say that my values aren’t better, but you know what, my values are better.
PS:
… I gave a webinar to US board members at the invitation of the Financial Times. Suffice it to say that while people are sincerely trying their best, our leaders are not even remotely equipped to handle the volume of people just outright lying to them about IT.
Source: Get Me Out Of Data Hell — Ludicity
(Emphasis mine.)
That last part is really the kicker. Every middle manager in all the various IT organizational structures inside of a Fortune-sized public company are lying about things, whether by omission or by fact. They’re lying about what it is they do. They’re lying about their problems. They’re lying about their capabilities. They’re lying about their timelines.
They’re lying to people who are either don’t care, or aren’t equipped to understand how the things they’re being told are lies, even if they do care. They’re lying to build “kingdoms” in the company by justifying more people, more machines, and more software than is required to solve a problem. And not just by a little; by orders of magnitude.
Unless senior management — and I mean the guys right under the officers, because the officers are never going to care, and the upper-middle guys don’t have the political clout to do it — unless they are curious, concerned, and knowledgeable enough to ask illuminating questions to piece the veil, the lies will go unchallenged, and the technical debt will continue to grow with every new project, and every project that is introduced to fix one that just failed.