The Pain Zone… is an enterprise data warehouse platform. At the small scale we operate at, with little loss of detail, a data warehouse platform simply means that we copy a bunch of text files from different systems into a single place every morning.
The word enterprise means that we do this in a way that makes people say “Dear God, why would anyone ever design it that way?”, “But that doesn’t even help with security” and “Everyone involved should be fired for the sake of all that is holy and pure.”
For example, the architecture diagram which describes how we copy text files to our storage location has one hundred and four separate operations on it. When I went to count this, I was expecting to write forty and that was meant to illustrate my point. Instead, I ended up counting them up three times because there was no way it could be over a hundred. This whole thing should have ten operations in it.
…
Almost every large business in Melbourne is rushing to purchase our tooling, tools like Snowflake and Databricks, because the industry is pretending that any of this is more important than hiring competent people and treating them well. I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets. It would take me a month tops.
I’ve known for a long time that I can’t change things here. But in this moment, I realize that the organization values things that I don’t value, and it’s as simple as that. I could pretend to be neutral and say that my values aren’t better, but you know what, my values are better.
PS:
… I gave a webinar to US board members at the invitation of the Financial Times. Suffice it to say that while people are sincerely trying their best, our leaders are not even remotely equipped to handle the volume of people just outright lying to them about IT.
Source: Get Me Out Of Data Hell — Ludicity
(Emphasis mine.)
That last part is really the kicker. Every middle manager in all the various IT organizational structures inside of a Fortune-sized public company are lying about things, whether by omission or by fact. They’re lying about what it is they do. They’re lying about their problems. They’re lying about their capabilities. They’re lying about their timelines.
They’re lying to people who are either don’t care, or aren’t equipped to understand how the things they’re being told are lies, even if they do care. They’re lying to build “kingdoms” in the company by justifying more people, more machines, and more software than is required to solve a problem. And not just by a little; by orders of magnitude.
Recently, it took me seven weeks of emails eventually involving fifty-odd people to get something done that took literally 30 seconds to do. Part of it was because I didn’t understand what I was asking for. I was asking for the wrong thing. Part of that is because the system is stupid, and no right-thinking person would have implemented it that way. Someone, somewhere, a long time ago (who probably left the company now) decided that this is how it should work, because someone at a consultancy told them that this is what “everyone” does.
I was asking for the logical, straightforward thing that would have fixed my issue, now and in the future. After it became clear that this would never happen, the 50+ “subject matter experts” involved had dozens of chances to respond and explain how what I was asking for actually worked, and clarified that I was asking for the wrong thing. But that didn’t happen.
Why? Because explaining why it works the way it does in front of God and everyone would reveal how idiotic it is. This can’t even be admitted over voice, but after several Zoom calls, you eventually see the pattern. It’s like the old magic eye pics in the 90’s. Eventually, you get your focus depth correct, and see the real picture. The image that no one else sees. They’re not paid to, so they don’t care.
Not only is the process stupid, the “self-help” web site that’s supposed to allow people to address this problem themselves is opaque, and doesn’t explain what’s going on. It masks the issue that I was having, when it would be very easy to show. This is a recurring pattern. Various IT functions have implemented “self-help” web sites that simply do not work, for reasons they are completely blind to because they never use them. They could make two small changes to this page, and it really would (mostly) address this stupid, broken policy. After all this wasted effort, someone involved seemed to finally understand my confusion and understand how this could be fixed, but I’ll bet they never do it.
Unless senior management — and I mean the guys right under the officers, because the officers are never going to care, and the upper-middle guys don’t have the political clout to do it — unless they are curious, concerned, and knowledgeable enough to ask illuminating questions to pierce the veil, the lies will go unchallenged, and the technical debt will continue to grow with every new project, and every project that is introduced to fix one that just failed.
At some point, when your personal sensibilities and the demonstrated collective priorities of the organization repeatedly come into conflict, you have to make a decision if you’re in the right place. For instance, I currently have personal issues which make the “switching costs” prohibitive, but this is an extremely individualistic equation to balance.
