Tag: Modern Fuedalism

Get Me Out Of Data Hell — Ludicity

The Pain Zone… is an enterprise data warehouse platform. At the small scale we operate at, with little loss of detail, a data warehouse platform simply means that we copy a bunch of text files from different systems into a single place every morning.

The word enterprise means that we do this in a way that makes people say “Dear God, why would anyone ever design it that way?”, “But that doesn’t even help with security” and “Everyone involved should be fired for the sake of all that is holy and pure.”

For example, the architecture diagram which describes how we copy text files to our storage location has one hundred and four separate operations on it. When I went to count this, I was expecting to write forty and that was meant to illustrate my point. Instead, I ended up counting them up three times because there was no way it could be over a hundred. This whole thing should have ten operations in it.

…

Almost every large business in Melbourne is rushing to purchase our tooling, tools like Snowflake and Databricks, because the industry is pretending that any of this is more important than hiring competent people and treating them well. I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets. It would take me a month tops.

I’ve known for a long time that I can’t change things here. But in this moment, I realize that the organization values things that I don’t value, and it’s as simple as that. I could pretend to be neutral and say that my values aren’t better, but you know what, my values are better.

PS:

… I gave a webinar to US board members at the invitation of the Financial Times. Suffice it to say that while people are sincerely trying their best, our leaders are not even remotely equipped to handle the volume of people just outright lying to them about IT.

Source: Get Me Out Of Data Hell — Ludicity

(Emphasis mine.)

That last part is really the kicker. Every middle manager in all the various IT organizational structures inside of a Fortune-sized public company are lying about things, whether by omission or by fact. They’re lying about what it is they do. They’re lying about their problems. They’re lying about their capabilities. They’re lying about their timelines.

They’re lying to people who are either don’t care, or aren’t equipped to understand how the things they’re being told are lies, even if they do care. They’re lying to build “kingdoms” in the company by justifying more people, more machines, and more software than is required to solve a problem. And not just by a little; by orders of magnitude.

Recently, it took me seven weeks of emails eventually involving fifty-odd people to get something done that took literally 30 seconds to do. Part of it was because I didn’t understand what I was asking for. I was asking for the wrong thing. Part of that is because the system is stupid, and no right-thinking person would have implemented it that way. Someone, somewhere, a long time ago (who probably left the company now) decided that this is how it should work, because someone at a consultancy told them that this is what “everyone” does.

I was asking for the logical, straightforward thing that would have fixed my issue, now and in the future. After it became clear that this would never happen, the 50+ “subject matter experts” involved had dozens of chances to respond and explain how what I was asking for actually worked, and clarified that I was asking for the wrong thing. But that didn’t happen.

Why? Because explaining why it works the way it does in front of God and everyone would reveal how idiotic it is. This can’t even be admitted over voice, but after several Zoom calls, you eventually see the pattern. It’s like the old magic eye pics in the 90’s. Eventually, you get your focus depth correct, and see the real picture. The image that no one else sees. They’re not paid to, so they don’t care.

Not only is the process stupid, the “self-help” web site that’s supposed to allow people to address this problem themselves is opaque, and doesn’t explain what’s going on. It masks the issue that I was having, when it would be very easy to show. This is a recurring pattern. Various IT functions have implemented “self-help” web sites that simply do not work, for reasons they are completely blind to because they never use them. They could make two small changes to this page, and it really would (mostly) address this stupid, broken policy. After all this wasted effort, someone involved seemed to finally understand my confusion and understand how this could be fixed, but I’ll bet they never do it.

Unless senior management — and I mean the guys right under the officers, because the officers are never going to care, and the upper-middle guys don’t have the political clout to do it — unless they are curious, concerned, and knowledgeable enough to ask illuminating questions to pierce the veil, the lies will go unchallenged, and the technical debt will continue to grow with every new project, and every project that is introduced to fix one that just failed.

At some point, when your personal sensibilities and the demonstrated collective priorities of the organization repeatedly come into conflict, you have to make a decision if you’re in the right place. For instance, I currently have personal issues which make the “switching costs” prohibitive, but this is an extremely individualistic equation to balance.

OKAY | Why the status quo is so hard to change in engineering teams

Before we talk about how to prevent it, let’s see how learned helplessness can happen in a development team. There are 2 main patterns:

Pattern #1: Process-related Learned Helplessness

In this case, the team needs to follow processes that have either been externally imposed, or internally imposed but no-one remembers exactly why.

Pattern #2: Complexity-related Learned Helplessness

In this other case, the source of powerlessness is sheer scale and/or complexity. There is truly no-one who understands the emergent behavior of the system.

Source: OKAY | Why the status quo is so hard to change in engineering teams

So I’m currently reading the book Learned Optimism, by Martin Seligman, which just so happens to be the seminal psychological work which teaches people how to combat learned helplessness. I’ll skip trying to give a primer on the process, but the one key that I need to talk about here is that you have to be objective in finding explanations for negative things in your life. Pessimists generally blame themselves for everything, and we all know that’s not fair. Even pessimists wouldn’t blame other people for their own problems when it’s obvious that they aren’t, but they will happily continue to blame themselves for problems that they know they are not responsible for.

People often think I’m a pessimist, but I’m not. I’m objective. I’m so objective that it’s almost a superpower. In that objectivity, I often realize that I am the source of the problem. That may appear to others that I’m a pessimist. However, I’m not afraid of calling a spade a spade, and pointing out that a problem is someone else’s fault, either. And I’m about to explain how this whole topic of discussion is, in fact, someone else’s fault, it’s systemic, and it’s not going to change.

The quoted article found its way to “Hacker” “News,” but you have to understand that this is a blog post from a company which sells software that purports to solve these problems for teams of programmers. I say, “hogwash.”

Every one of the problems listed in the article is the result of bad management.

Period. There’s no getting around it, and there’s no sugar coating it. In the vein of the article, “there are 2 main patterns” why.

First, in our modern feudalism of corporations, the low-level managers are like barons, the middle managers are like earls, upper managers are like dukes, and the C-levels are like princes. Everything flows from the top down. Everyone is playing political intrigue for more power, in the only terms that the corporate structure can understand and deal with: budget and headcount. No one will deal with fixing a problem if it doesn’t directly contribute to their standing with their peers or superiors, and problems with software build systems, infrastructure, or technical debt are simply invisible to anyone not dealing with it directly.

Second, the days of building a company to last 100+ years is gone. J. Irwin Miller built Cummins into a world-spanning empire, and then retired. Then the company started implementing all the trendy business-school ideas, and was getting run into the ground. Miller came out of retirement and righted the ship, and then retired again. Those days of people caring about a company like this are gone. Everyone is out to “get theirs,” and then hit the brcks. Everyone knows this, but we still like to pretend that anyone in the bridge cares if the company actually survives in its efforts to make the next $100M.

From HR systems to compensation to benefits, companies are deeply, deeply afflicted with myopic vision. It’s driven from the very top, where every decision of significance is made in view of the almighty stock price. Corporate boards encourage this behavior by tying executive pay to stock options. For decades now, we’ve watched big companies sacrifice long-term performance for short-term gains, because it will net the corporate officers tens of millions of dollars in stock grants. If there’s any hope in this situation, it’s that most of the excess capacity in corporate America has already been wrung out (q.v., the recent supply chain issues), and there’s nothing left to pillage, either internally, or by corporate raiding. (The play now is to merge to form effective monopolies and duopolies, and extract all the profit from your market, but that’s another topic.)

When you combine a lack of visibility with a lack of vision, you get an appalling lack of concern for issues that the rank-and-file serfs deal with.

The fact that issues that programming teams deal with are highly-technical doesn’t matter at all. In fact, it works against them, because fixes are more expensive than in other teams. Say, like buying yet another web service, on top of all the others you already subscribe to, in order to apply metrics to problems that management will never fix, because they neither understand nor care.

There are a lot of examples I could write about here, but one stands out. I know of a person who was briefly involved with an effort to quantify the development process for his Fortune 150-sized company, much like the linked article. When he talked with a knowledgable insider about the system, it was one shocking revelation after another. The system was down more often than it was up. There was no way to estimate how long a build would take. There was no way to check on the current load of the system. There was no way to check on jobs, other than to log into a remote machine and tail a log file. The system wouldn’t even send out an email at the end of the build to indicate success or failure, because it would show how badly the whole thing performed.

So a “baron” in another group got the bright idea that they would oversee the writing of a dashboard to overlay metrics on the process, so that the “earl” who was responsible for it could “see what parts to focus on.” Let that sink in… Do you see the problem here? This manager thought that he would be allowed access to the various running processes of the system, to quantify, graph, and display them — like, on the shared screens in common spaces — when the owner of the system wouldn’t even allow emails about the success or failures of builds to be sent out, which is the most basic of all metrics, and which would have required only a single checkbox to be ticked. The mere idea of this project was so hopeless as to border on insanity.

The people responsible for the system already knew about its problems. They didn’t need metrics. They didn’t care! Whether that was from a lack of budget or a lack of headcount or a lack of technical ability or a lack of communication with other teams… none of that mattered. The people responsible for the system weren’t willing to spend their political capital to fix the problems of the system. And whether that lack of will came from their own political ambitions or a lack of understanding or a lack of vision… none of that mattered either.

The article talks about why developers get frustrated, and proposes that metrics will fix it. Their system won’t help, because, in most companies, it’s not about a lack of metrics. It’s a lack of will, motivated by personal ambition, gamed by incentivizing short-term productivity. As a developer in a group like this, you can blame yourself, or you can look for opportunities internally, or you can take your show to a different stage. All of those responses are in the HN thread about the article. We all have to find our own balance points between our predilections and our companies’.

As a developer, you can alter your internal “explanatory style” to move the blame for the problems you have to deal with to systems or software or people, but all of that misses the real point.

In post-modern, Fortune-sized, American corporations, the problem is management, and all management starts from the top. The corporate officers set the leadership paradigm the rest of the company will adopt, and they are being strongly incentivized against long-term investment in tooling and process. This mindset trickles all the way down to the bottom. If there are problems with “getting stuff out the door,” they will almost always hire more people, because that’s almost always the easier and cheaper solution than retooling an entire department with new software and process. And then, every 5-10 years or so, they will bury their failure to address actual productivity problems with yet another “reorg” or sweeping IT platform change.

The Two Middle Classes – Quillette

The struggle between the two middle classes is not just a matter of wealth and power, but also of retaining the social basis for democracy itself. Without a strong, independent middle class operating outside the control of large institutions, be they tech giants or governments, we may be heading towards a technocratic future, that as one Silicon Valley wag put it, resembles  “feudalism with better marketing.”

Source: The Two Middle Classes – Quillette

I’ve been calling our corporatocracy a modern form of fuedalism for awhile now, which is where this article ends up. However, along the way, it explains the ascendancy of the “clerisy” — a liberal middle class made up of people like college professors and government bureaucrats — which does a good job at explaining the historically-different battle lines of the cultural war we witnessed in the last election. Expanding the thesis: It’s no longer about race, because race is no longer the determinant factor in which sector you work. I think this nails the current political climate, and current social evolutionary stage, much better than my small pull quote and comment would suggest.