Corporate IT – The Mind of David Krider

More AI for all the Corporate IT Things

Last month, I was talking about how I didn’t understand what my bluechip Fortune 250 is doing with AI. From AI for all the Corporate IT Things:

Well, it’s a good thing I don’t understand, because he’s not talking about using AI to fix IT. He wants to use “technology” to improve our “safety-ness.” Say wha..? Like, he wants to use AI to improve safety on the factory floor. Huh?! Are we going to buy Tesla robots to pull people’s fingers out of the way of presses?! I’m confused.

I sat in a Zoom call where someone discussed the first pilot program for our official corporate AI efforts. On the one hand, they’ve done exactly what they said they were going to do. They’re trying to use AI to try to reduce OSHA incidents. Surely that’s a noble effort, right? But on the other hand, I have trouble imagining a real-world scenario that would be less applicable to AI. I mean, first of all, safety incidents are already scrutinized with a microscope. Second of all, there are so relatively few, I don’t believe you can use AI to analyze them. There’s not enough data to establish patterns. On top of that, every incident is an outlier, and gets dealt with immediately, and not in a performative way, but, like, for real. New rules are put in place, guard rails are installed, etc. So these outliers are very, very unlikely to happen again. Ergo, the data is not statistically significant, and whatever else you know about AI, it’s ALL based on statistics. So I don’t get it.

The other thing that strikes me is that we’re using — er, “renting,” I’m quite certain, and at an exorbitant rate — an off-the-shelf AI product called GenAI by Palantir. You know, the love child of the so-called Five Eyes multinational intelligence conglomerate, and the company that spies on everyone, everywhere, all of the time. So we’re not using our company’s vast resources to invest in creating our own AI models. We’re just paying our contractors to learn how to operate someone else’s machine. In this golden age where instructions on how to create models are readily accessible, and the coding libraries to implement them proliferate, we’re eschewing the opportunity to create custom models that could help our specific business problems.

Over a year ago, I talked with people about what I think we could do with AI, but I didn’t get anywhere. In the past months, several other engineers have spoken to me about similar ideas. In the part of the company I inhabit, there is a glaringly obvious use for AI staring us in the face. The problem is that we don’t have all the data we need to make it work, and getting the owners of the systems we would need to tie together with our data to open up their databases to us is simply impossible from where we sit. That sort of thing is simply never going to happen without a strong, direct proclamation from the CEO, and, even then, getting those people to give up some of their “power” in the company so that someone else can have more is going to be fought up and down the org chart. So we seem stuck. The only things we can use AI for won’t matter, and the things that would make a difference will never be done.

AI for all the Corporate IT Things

I got an email with a link to a “town hall” about IT. I said to myself, alright, I dare you to tell me something interesting or actionable, and started watching the replay.

The CIO leads off, of course. His first slide is about DEIC, and celebrating/observing Black History Month and the Lunar New Year.

Sigh.

I mean, that’s great and all, but that’s 10 minutes we’re not talking about IT, which is what this meeting is supposed to be about, and which is all I care to hear about. I seriously doubt that people in, say, Europe or China care much about the US Black History Month, or that people in the US care about the Chinese Lunar New Year, for that matter. But, sure, let’s waste time pandering in the name of the current thing.

And then he says he’s able to relax, now that we know Taylor Swift was going to the at the Super Bowl. He doesn’t know what teams were going to play, but he spent a few minutes talking non-ironically about Swift being there.

Again, I mean, that’s great and all, but a half hour in, we’ve now spent thousands of man-hours not talking about IT.

When we finally get around to talking about, you know, information technology, and I find out that we’re apparently using AI to modernize our “corporate operating system.” I know a little about AI. I know a lot about how our internal procedures and organizational systems works. I do not understand how we can get AI to fix any part of this.

Well, it’s a good thing I don’t understand, because he’s not talking about using AI to fix IT. He wants to use “technology” to improve our “safety-ness.” Say wha..? Like, he wants to use AI to improve safety on the factory floor. Huh?! Are we going to buy Tesla robots to pull people’s fingers out of the way of presses?! I’m confused.

Next, we’re apparently going to minimize all “risks” to IT uniformly, without specifying or identifying what any of those “risks” are. So, at least we’ve got that going for us, which is nice. We’re going to do this by 1) reducing “new” findings, 2) eliminating repeat “findings,” and 3) closing “findings” faster. Well, that certainly seems simple. A little light on details, but I’m sure we’ll figure it out.

Then we’re going to “partner” with AI, and it’s going to help us be more “exponential.” Except that we’ve also been sent a company-wide email that says we’re not allowed to use AI for, well, anything!

After an hour and a half, I gave up watching. I just want to note that the leader of “transformation” just bought a new-fangled “Mac” and says he’s “challenged” to set it up.

The Six Dumbest Ideas in Computer Security

The opposite of “Default Permit” is “Default Deny” and it is a really good idea. It takes dedication, thought, and understanding to implement a “Default Deny” policy, which is why it is so seldom done. It’s not that much harder to do than “Default Permit” but you’ll sleep much better at night.

Source: The Six Dumbest Ideas in Computer Security

This was from 2005, and if your organization’s approach was always Default Deny, then it’s a great idea. The problem with Default Deny isn’t the idea, it’s switching away from Default Permit after decades of acclimation. One comment in the HackerNews discussion about this article says it adds “three more steps” to the conversation with IT to fix your app after the change. I would argue that it’s a whole lot more than that.

A friend of mine has trouble running a very important vendor application used in his department. It stopped working some time ago, so he opened a ticket with IT. It was so confusing to them that it got to a point that they allowed him to run Microsoft’s packet capture application on his machine. He followed their instructions, and captured what was going on. But, despite the capture, they were unable to get it working through the firewall.

Out of frustration, he sent the capture to me and asked me to take a look. Even though our laptops are really locked down, as a dev, I get admin on my machine, and I have MSDN, so I downloaded Microsoft’s tool, and looked over the capture. It was a complicated architecture, using a client/server implementation on the local machine. The front end was working over networking ports to talk to its back end, and the back end was using other ports to talk to the vendor’s servers.

I knew what to look for because I had just undergone a lot of pain with my own development workflow, because the company had started doing Default Deny, and it was messing with my development workflow in several ways. Not too long before, they dropped the bomb that EVERY network communication through the firewall required the Cisco Umbrella certificate (so that they can intercept and inspect all traffic, despite SSL). I had to figure this all out on my own, based on the behavior I was seeing, but I was able to figure out how to download that cert and include it in the chain to work with both bundle and HTTP calls in Rails, and psql for Postgres.

Then they locked out Postgres’ default port entirely. They reopened it for me at my request at first. Then six months later, they closed it again, and told me that I needed to formally request permissions for that, which was going to be a many-years-long process. I “just” rewrote the functions I needed in my Rails app to use HTTPS instead of using a direct database connection.

Anyway, I told my friend what ports IT needed to open, and how that he needed to explain to the vendor that they had to include the Umbrella cert in their chain on the back end, but he’s still having problems. Why am I being vague about the details here? It’s not because of confidentiality, though that would apply. No, it’s because my friend had been “working with IT” for over a year to get to this point, and this was two years ago, and I’ve simply forgotten a lot of the details.

“I said all of this to say” that saying it will take “3 extra rounds” is a bit of an understatement when IT starts doing “default deny,” at least in legacy manufacturing companies.

Windows Being Windows, Shills Being Shills

Windows stays prominent because Microsoft caters to corporations which abuse the poor, defenseless OS into doing things like locking users out of changing the desktop background and the sleep timeout. Until Apple offers power-hungry corporate IT middle managers the same level of user-hostile malfeasance in the name of “security,” Microsoft will hold the high ground in corporate deployments.

This becomes a self-perpetuating cycle of not-so-micro-aggressions, as the corporate use of Windows continues to skew all the Microsoft-bought-and-paid-for industry polls that show how much more prevalent Windows is over OSX, and self-justify corporate America that they’re doing the right thing by continuing to stick with it. This, in turn, leads to an entire sub-industry of corporate “security” software which must be installed on Windows, because, well, the bought-and-paid-for auditors told them they had to.

Thus, I wind up with a corporate laptop with 3 different “endpoint” security products installed on it, and something like 30-40 different scripts and checks that run almost by the hour to make sure that the inherent weaknesses of Windows hasn’t compromised our precious meeting PPT’s, which #CorporateIT apparently considers as sensitive as the US nuclear arsenal codes.

Apple offers an alternative to this madness, and I’m very glad they do. In my experience, almost no one runs Windows personally, except for gaming purposes. Numbers like StatCounter vastly over-report Windows usage, because everyone working for a large corporation and in the government is forced to use Windows.

I wish someone would produce a market share report that 1) separates corporate purchases and 2) includes phones as primary computing devices. I think we would see that the “computing world” is vastly different than Gartner would have us believe.

Corporate IT “Support”

I couldn’t mount network drives on my work laptop, so I opened a ticket. That was FIVE weeks ago. The ticket was closed by the first person after 3 days without even contacting me. I immediately re-opened it, and it proceeded to worm its way through an additional EIGHTEEN people, and finally came back to the original person who closed it, who — for at least the 4th time in the email chain — asked me questions I had “pre” answered in the original complaint.

Disgusted, I finally took a couple of hours, and figured it out myself. When I was finishing up, and documenting the fix on the ticket, the first guy contacted me on Teams, and asked me to bring the laptop into the office and try to do what I’ve shown over and over in screenshots couldn’t be done.

I guess I’m just lucky that I have admin rights on my laptop.

FWIW, the problem started when I installed WSL to test using for developing Rails apps. When I recognized the coincidence, and since I hadn’t actually used it yet, I uninstalled the Ubuntu image. This wasn’t enough to fix the problem.

The error message was clearly about not being able to resolve machine names, but I didn’t want to monkey around with the resolution settings because #CorporateIT does nasty things to our laptops, and I was afraid of screwing up their configuration.

I finally removed WSL entirely, and got a different error message when trying to mount a drive, which led me to this Stack Exchange post. I copied the registry setting from a computer which works (which was the same as what was listed in the top answer), and all was sorted again.

To me, the worst part of this charade is that the “original” guy will get the credit for “resolving” the issue, and that really burns me.

Introducing Surface Pro 10 for Business and Surface Laptop 6 for Business | Microsoft Devices Blog

AI-powered PCs built for a new era of work We are excited to announce the first Surface AI PCs built exclusively for business: Surface Pro 10 for Business and Surface Laptop 6 for Business. These new PCs re

Source: Introducing Surface Pro 10 for Business and Surface Laptop 6 for Business | Microsoft Devices Blog

I just love how Microsoft sticks “FOR BIZNESSSSSSS!” on the end of things. “Skype FOR BIZNESSSSSSSS!” “Teams FOR BIDNESS!” “Outlook FOR BIZNAAAAAS!” Sigh. How can you take this stuff seriously? Anyway.

Windows stays because Microsoft caters to corporations which abuse the poor, defenseless OS into doing things like locking users out of changing the desktop background and the sleep timeout. Until Apple offers power-hungry corporate IT middle managers the same level of user-hostile malfeasance in the name of “security,” Microsoft will hold the high ground in corporate deployments.

This becomes a self-perpetuating cycle of not-so-micro-aggressions, as it continues to skew all the Microsoft-bought-and-paid-for Gartner polls that show how much more prevalent Windows is over OSX, and self-justify corporate America that they’re doing the right thing. This, in turn, leads to an entire sub-industry of corporate “security” software which must be installed on Windows, because, bought-and-paid-for auditors told them they had to.

Thus, I wind up with a corporate laptop with something like 3 or 4 different “endpoint” security products installed on it, and something like 30-40 different scripts and checks that run almost by the hour to make sure that the inherent weaknesses of Windows hasn’t compromised our precious meeting PPT’s, which corporate IT considers as sensitive as the US nuclear arsenal codes.

Apple offers an alternative to this madness, and I’m very glad they do. I would rather they stay as they are rather than corrupt their ecosystem in this way to make some corporate sales, and I will happily continue to use my personal MBP to do as much of my work as I can.

We keep making the same mistakes with spreadsheets, despite bad consequences | Ars Technica

Spreadsheets represent unknown risks in the form of errors, privacy violations, trade secrets, and compliance violations. Yet they are also critical for the way many organizations make their decisions. For this reason, they have been described by experts as the “dark matter” of corporate IT.

Source: We keep making the same mistakes with spreadsheets, despite bad consequences | Ars Technica

As I often say, making real applications out of these Frankenstein monsters of data has been my bread and butter throughout my career. The function that the central IT departments in blue chip manufacturing companies could never quite wrap its arms around keeps getting bigger and bigger, and making larger and larger gaps to fill by people in the trenches. So… too right, mate, and keep it up.

Why do people use VBA?

Why do people use VBA? In order to answer this question, we must first look at another question – who actually uses VBA in the first place? In 2021 I ran a poll on /r/vba where I asked redditors why they code in VBA. From these data, we can clearly see that the majority of people who use VBA do so mainly because they have no other choice. Many organisations run their entire business processes with Excel, and when a little bit of automation is required VBA is usually #1 on the list.

Source: Why do people use VBA?

I was just ranting about this to my kids a couple days ago.

Even in large companies, with massive IT departments, and lots and lots of internal databases and information systems, US businesses are still run on Excel. That’s not subjective. I’ve worked for decades inside three Fortune 250’s (and a couple smaller shops), and bad Excel “applications” are in use at all of them. And after one person learns enough VBA to get a spreadsheet dealing with a particular issue to save a little time for themselves, they start sharing them with their colleagues, and the problem gets worse. Half of my career has been built on making “real” applications out of Excel spreadsheets that were wobbling under their own weight.

But why?

Back in the old days, IT grew out of the accounting department. They had the only computer in the building, and it was an IBM mainframe. Great stuff, right? Saved a lot of time and paperwork, right? Except that it didn’t. It quickly ossified the company’s work flow, and permanently hobbled its ability to adapt to change. It would take years to get any changes made in the mainframe group, and people were frustrated. Along came spreadsheets, and everything changed.

I saw it myself in my first engineering job in 1993. We got new computers with Windows 3.1 and Quattro Pro. (And AutoCAD. And, of course, on mine: DOOM!) After weeks of bugging the lady who ran the mainframe — who apparently had to write a whole program — I got her to dump the BOM for a couple of our products to compare for similarities. I downloaded the 2 files to my PC with a token ring mainframe interface card. I think they were only about 1MB each. With 8MB of RAM, I had twice as much memory as our System 36, and I could open both BOM’s in a spreadsheet, and analyze them to my heart’s content. Understanding that I had more processing power on my desk than the freezer-sized unit in the other room was eye-opening.

American manufacturing companies (at least) never got the message. The invention of the spreadsheet spared them from facing the fact that the mainframe had become the black hole of their IT world. As changes were becoming impossible to get from the mainframe group, PC’s with Windows and Excel allowed people at all levels and in all job functions to start working around the mainframe and its limitations.

Now, these kinds of companies are decades behind the curve. They thought “outsourcing” was going to fix all of their problems. When it didn’t, they thought “consultants” would be the trick. Surely “agile” will do it this time, right? No. It’s not the process; it’s the mainframe. Forcing every corporate workflow and piece of data to be kept canonically inside a 40-50-year-old legacy system’s limitations is quite literally killing the company. It’s certainly killing their competitive advantage.

My current company still breaks our primary software component into 8 pieces because that’s what would fit on floppies to send to the plant to program the hardware. Every IT system — and every spreadsheet — in the company has to deal with this 40-year-old legacy issue because that’s what we programmed the mainframe to expect, and now that’s the only way to bill a customer for it. So the logistics of dealing with multiple trees and branches of software (and multiple trees and branches of documentation about the software) is multiplied by a factor of 8 to this day. There is no escape from this black hole. You can’t re-engineer this situation. It’s too ingrained.

I worked for one group which, on every engineering release, had to get a giant table of software versions — each with their 8 part numbers — into the mainframe. The process was so onerous that they would spend days clicking through mainframe terminal emulator screens to get the information they needed, to make a spreadsheet in a particular format, which they would send to another group to actually enter back into the mainframe. Part of the problem was the spreadsheet had to be in 3 columns, but the mainframe screens were in 4 columns (or vice versa), so a lot of it was purely formatting. I wrote a little program to automate all of this, but I’ve left the group, and I’m sure no one uses it any more. The particularly stupid part of this story is that people fought me to write a little software that saved these people 10’s of hours a week in the name of their own job security.

And no one in the corporate hierarchy cares. In this day and age, the executives are all just playing the waiting game, letting things atrophy — saying all the right things publicly — while they wait until the financials are inverted enough to make the company a juicy prospect for a buyout in an industry-wide rollup by private equity.

Meanwhile, actual people have to get stuff done to stay employed and feed their families. Inside the company, the managers have to look at the three year lead times to get a simple application written by “corporate IT,” and can do nothing but just continue to throw bodies and VBA macros at it. Or, in my case, have me write something to do it. That is, until it gets successful enough that people notice, and it gets taken away from me, but that’s another story…

All Your Base Are Belong to Us

If you have a corporate- or school-issued computer, you have no control over it. Unless you wipe it and reinstall the OS, and even then, of course, they could leave things in the BIOS, and probably do. Then again, you barely “own” devices you buy, but that’s another rant. Here’s the task list for my corporate laptop.

Sigh

So let’s see…

Seven different reports about what I’m uploading to OneDrive.
Five jobs to keep Chrome and Edge up to date. Firefox and IE are also installed.
A job to make sure you keep Zoom around.
A [REDACTED] hourly job to make sure you haven’t elevated your privileges.
A job to make sure you haven’t (apparently) installed the npcap library. I mean, God forbid you should try to use this at a corporate site, which has probably used switching since… 1996 or so.
Three other [REDACTED] jobs to make sure you don’t do other things “they” don’t want you to do.
At least 5 jobs to make sure you don’t change… anything about the way they’ve installed Office, apparently.

Thirty one jobs. Only one of these is mine, to do the one thing I need this (secondary) computer to do.

This machine bypasses my carefully-curated and ad-blocked local DNS. I don’t know what it uses for DNS, but I see that it doesn’t operate over port 53, and I don’t care to know any more.

It also won’t print to a printer in your house. I think I tried to print to a printer at the office once, and give up after one try, because I knew it was going to be futile. Basically, no one prints anything. They must save a TON on printer costs as a company. Most printouts are a waste of resources anyway, so this might actually be genious.

Microincentives and Enshittification – Pluralistic: Daily links from Cory Doctorow

That increased profitability can only come from enshittification. Every product manager on Google Search spends their workdays figuring out how to remove a Jenga block.

Source: Microincentives and Enshittification – Pluralistic: Daily links from Cory Doctorow

Internally, every powerful person at Google is committed to ensuring that their rival-peers don’t stake out fresh territory as their own. The one thing every top exec can agree on is that the one guy who’s trying to expand the company into an adjacent line of business must not succeed.

What’s worse, these princelings compete with one another. Their individual progression through the upper echelons of Google’s aristocracy depends as much on others failing as it does on their success. The org chart only has so many VP, SVP and EVP boxes on it, and each layer is much smaller than the previous one. If you’re a VP, every one of your colleagues who makes it to SVP takes a spot that you can no longer get.

Those spots are wildly lucrative. Each tier of the hierarchy is worth an order of magnitude more than the tier beneath it. The stakes are so high that they are barely comprehensible.

That means that every one of these Jenga-block-pulling execs is playing blind: they don’t — and can’t — coordinate on the ways they’re planning to lower quality in order to improve profits.

I don’t know if I’ve read a clearer description of the things I’ve seen in 30 years of (mostly) Fortune 250 corporations. Over the course of my career, about a half dozen of my successful software projects — with many happy users — have been sabotaged because they made someone else look bad, or just had the unacceptable side-effect of making me look like I knew what I was doing. Seriously. I could write a book.

Hey, I got paid, and have had a comfy ride along the way. What could I expect as a developer toiling away in the bowels of some faceless blue chip corporation? The only thing the kind of companies I have worked for could offer would be a role with more responsibility but no more pay. Uh… pass.

What really frustrates me in all of this is the tireless effort and work to make sure that software never actually improves workflows or processes for the company, so that eternal middle managers can preserve their tiny little fiefdoms. Sure, we’ll make some software to do something, but by the time the managers “manage,” the people who don’t do the job write the specs, the outsourced programmers who don’t understand anything about the process write the application, the years go by, and the poor schmucks who have to use the thing sign off on the acceptance testing, just to move on, everyone is left with a piece of crap they can’t stand to use, and they wonder why anyone bothered. They’d have been better served just continuing to use horrific, shared Excel spreadsheets.

Google spends a whole-ass Twitter, every single year, just to make sure you never accidentally try another search engine.

I never want to hear another word about what else Elon Musk could have done to supposedly improve the world with the money he spent buying Twitter.

Likewise Google/Apple’s mobile duopoly is more cozy than competitive. Google pays Apple $15–20 billion, every single year, to be the default search in Safari and iOS. If Google and Apple were competing over mobile, you’d expect that one of them would drop the sky-high 30 percent rake they charge on in-app payments, but that would mess up their mutual good thing. Instead, these “competitors” charge exactly the same price for a service with minimal operating costs.

Since the 80’s, American corporations have learned to toe the precise line that will allow them to point fingers at their “competitors” in court to wriggle out of the en vogue legal definition of monopoly, but it’s all such a naked joke. The app stores are the same way. It is a certainty that very-high level execs at Apple and Google have concluded to keep their fees the same, so that the market for app development doesn’t actually work, and is anything but “free.”