Tag: Corporate IT

Even More on the Capabilities of Current-Gen “AI”

Another Twitter post led me to this Reddit post:

Further down in the comments, OP had this to say:

I copy-paste all of this for full context, but I want to emphasize this paragraph:

It tried a bunch of things before it had the breakthrough, but as you probably know it always said: “I found it! This is finally the root case of the problem!” but every AI does that on almost every prompt, so it wasn’t anything special. It was just another thing it did that I tested and noticed it worked without also regressing other stuff, and then I looked at it and compared it, and then realized what it did. Then I had to go and delete a bunch of other unnecessary changes that Opus also did that it insisted was good to leave in and it didn’t want to remove, but wasn’t actually pertinent to the issue.

Now, I make this sentiment on various social media sites so often that I have it as a shortcut on Mac and iOS: when I type, “reddits”, it autocompletes to say, “Reddit is a CCP-funded deep state psyop against a very specific demographic, and nothing on it should be taken at face value.” But in this case, it rings authentic. I deal with this most every day at this point.

A few years ago, a few Google searches would dig up StackOverflow answers and personal blog posts that specifically dealt with whatever you were asking about. But within the span of a few years, Google has become almost worthless for searching on programming problems. It’s lucky that LLM’s have come along now, or this work would be that much harder. I’d suggest that Google let their search product slide into terribleness in order to push people to their AI product, but they don’t have one yet, so their awfulness can just be ascribed to basic late-stage capitalism and utterly-predictable monopolistic malfeasance.

Anyway, this last quote is so appropriate. AI can’t figure out if what it did actually worked, but it always says it does, and when you see a move in the right direction, you have to figure out what part made it work, and then delete all the other stuff that the AI broke in the process. In this regard, it is exactly like a guy I used to work with who would break half a dozen things on my servers before he got something working. He never cleaned up his failed attempts. Every time he told me he had done something, I’d ask a few questions about his process, and then just quietly go put things back in order.

I just went through this process over the last weekend. I’m trying to move a codebase from Rails 6 to Rails 8. There’s a lot of changes in the way Rails handles Javascript between these two versions, from the bundling to the running. I’ve gotten left behind on a lot of this. Even when I spun up this Rails 6 app six years ago, I was using the old asset bundling technique from Rails 3/4. I was happy to make the jump to “no-build” in 8 all at once, but my application servers needed upgrading from a version of Ubuntu which is no longer getting security updates. This upgrade forced me into upgrading NodeJS. This upgrade broke the asset building process in Rails because the dreaded SSL upgrade has moved to this part of the stack now. So I moved to Webpacker, which took way too long to work out. I tried to use AI throughout, but it was of almost no help at all.

After finally getting moved to Webpacker, just in time to move to ImportMap, I have had to tackle how Stimulus, Turbo, and Hotwire work. Rails 7 focused on Turbolinks, which utterly broke the Javascript datatable widget, AgGrid, which I use all over the site, so I removed Turbolinks from my Rails 6 app, and never upgraded to 7. Now I’m learning how to do things in the new “Rails way,” and AI has been helpful… in precisely the same way that this Reddit poster describes. I’ve had to run many prompts and tweak and cajole it’s “thinking,” but I finally got a really neat and tiny solution to a question… All while verifying the approach with videos from GoRails. (Which I subscribed to just to make this turn of learning.)

After I had a working function to a particular feature I wanted, I had an “aha!” moment. I could see what all this new tooling and plumbing was about. I felt a little foolish, because it winds up just being a way to move the Javascript you need out of the view code. That’s a Good Thing (TM), but I couldn’t see the forest for the trees until that moment.

And even after this success, I’m plagued with a more philosophical question. The way that Claude factored the feature I wanted was all Javascript. Meaning, once the view was loaded, it dealt with the interactivity without going back through a server-side render step. It relied on the browser doing the DOM manipulation. Which is the “old” way of doing things, right? I asked it to specifically use Turbo streams to render the HTML that the browser could use to simply replace the div, and it said, “Oh, yes, right, that’s the more idiomatic way to do this in Rails,” and gave me a partial that the Stimulus controller use to do the same thing. But now I have a clean, one-file, entirely-Stimulus approach, versus having extra calls to format_for in a controller, a turbo-stream ERB file, and a partial. Seems to me like too much extra to make things idiomatic.

Also, when I asked Claude for this refactor, it broke the feature. So now I have to figure out if I want to fix the turbo-stream approach to keep things “the Rails way,” or just let Stimulus handle the whole thing. I think I will be using turbo-streams to refresh data in my AgGrid tables, but I think I’ll let Stimulus do all the work when it can. It keeps things simpler, and it’s basically what I was doing before anyway.

I want to go back to what I was saying before about how you have to “clean up” after the AI. This is critically important, and it’s a problem in the making that I’m hoping corporate managers figure out before it becomes permanent. If you hire juniors and expect them to produce code like seniors with AI, you’re going to wind up with a bunch of instability because of extraneous code that AI leaves behind. I expect that this problem is too “viral” to survive. I don’t think an actual, useful, non-trivial application could last very long being vibe-coded. It would start to teeter, and people would hit it with more AI, and it would fall over and shatter, and then they’d get to keep all the pieces. I worry that current applications will be patched here and there by juniors who don’t clean up the mess left behind, and these errors will accumulate until the codebase is so broken that…

Oh, for Pete’s sake. What am I even saying!? The same thing will happen here that has been happening for 40 years of corporate IT development: systems get so wonky and unmaintainable and misaligned that new middle managers come in, pitch senior management into doing a massive system replacement, spend twice as much time and three times as much money as they said it would take, launch the system with dismal performance and terrible UI, piss everyone off, polish their resume, get a new job, and leave the company and everyone in it holding the bag with the accretion their terrible decisions made by committee over years.

AI will change absolutely nothing about this. The problem isn’t technology, or code, or languages, or databases, or API’s, or anything else. The problem is people. It’s always BEEN people. I’m not clear that it ever NOT be about people.

More AI for all the Corporate IT Things

Last month, I was talking about how I didn’t understand what my bluechip Fortune 250 is doing with AI. From AI for all the Corporate IT Things:

Well, it’s a good thing I don’t understand, because he’s not talking about using AI to fix IT. He wants to use “technology” to improve our “safety-ness.” Say wha..? Like, he wants to use AI to improve safety on the factory floor. Huh?! Are we going to buy Tesla robots to pull people’s fingers out of the way of presses?! I’m confused.

I sat in a Zoom call where someone discussed the first pilot program for our official corporate AI efforts. On the one hand, they’ve done exactly what they said they were going to do. They’re trying to use AI to try to reduce OSHA incidents. Surely that’s a noble effort, right? But on the other hand, I have trouble imagining a real-world scenario that would be less applicable to AI. I mean, first of all, safety incidents are already scrutinized with a microscope. Second of all, there are so relatively few, I don’t believe you can use AI to analyze them. There’s not enough data to establish patterns. On top of that, every incident is an outlier, and gets dealt with immediately, and not in a performative way, but, like, for real. New rules are put in place, guard rails are installed, etc. So these outliers are very, very unlikely to happen again. Ergo, the data is not statistically significant, and whatever else you know about AI, it’s ALL based on statistics. So I don’t get it.

The other thing that strikes me is that we’re using — er, “renting,” I’m quite certain, and at an exorbitant rate — an off-the-shelf AI product called GenAI by Palantir. You know, the love child of the so-called Five Eyes multinational intelligence conglomerate, and the company that spies on everyone, everywhere, all of the time. So we’re not using our company’s vast resources to invest in creating our own AI models. We’re just paying our contractors to learn how to operate someone else’s machine. In this golden age where instructions on how to create models are readily accessible, and the coding libraries to implement them proliferate, we’re eschewing the opportunity to create custom models that could help our specific business problems.

Over a year ago, I talked with people about what I think we could do with AI, but I didn’t get anywhere. In the past months, several other engineers have spoken to me about similar ideas. In the part of the company I inhabit, there is a glaringly obvious use for AI staring us in the face. The problem is that we don’t have all the data we need to make it work, and getting the owners of the systems we would need to tie together with our data to open up their databases to us is simply impossible from where we sit. That sort of thing is simply never going to happen without a strong, direct proclamation from the CEO, and, even then, getting those people to give up some of their “power” in the company so that someone else can have more is going to be fought up and down the org chart. So we seem stuck. The only things we can use AI for won’t matter, and the things that would make a difference will never be done.

AI for all the Corporate IT Things

I got an email with a link to a “town hall” about IT. I said to myself, alright, I dare you to tell me something interesting or actionable, and started watching the replay.

The CIO leads off, of course. His first slide is about DEIC, and celebrating/observing Black History Month and the Lunar New Year.

Sigh.

I mean, that’s great and all, but that’s 10 minutes we’re not talking about IT, which is what this meeting is supposed to be about, and which is all I care to hear about. I seriously doubt that people in, say, Europe or China care much about the US Black History Month, or that people in the US care about the Chinese Lunar New Year, for that matter. But, sure, let’s waste time pandering in the name of the current thing.

And then he says he’s able to relax, now that we know Taylor Swift was going to the at the Super Bowl. He doesn’t know what teams were going to play, but he spent a few minutes talking non-ironically about Swift being there.

Again, I mean, that’s great and all, but a half hour in, we’ve now spent thousands of man-hours not talking about IT.

When we finally get around to talking about, you know, information technology, and I find out that we’re apparently using AI to modernize our “corporate operating system.” I know a little about AI. I know a lot about how our internal procedures and organizational systems works. I do not understand how we can get AI to fix any part of this.

Well, it’s a good thing I don’t understand, because he’s not talking about using AI to fix IT. He wants to use “technology” to improve our “safety-ness.” Say wha..? Like, he wants to use AI to improve safety on the factory floor. Huh?! Are we going to buy Tesla robots to pull people’s fingers out of the way of presses?! I’m confused.

Next, we’re apparently going to minimize all “risks” to IT uniformly, without specifying or identifying what any of those “risks” are. So, at least we’ve got that going for us, which is nice. We’re going to do this by 1) reducing “new” findings, 2) eliminating repeat “findings,” and 3) closing “findings” faster. Well, that certainly seems simple. A little light on details, but I’m sure we’ll figure it out.

Then we’re going to “partner” with AI, and it’s going to help us be more “exponential.” Except that we’ve also been sent a company-wide email that says we’re not allowed to use AI for, well, anything!

After an hour and a half, I gave up watching. I just want to note that the leader of “transformation” just bought a new-fangled “Mac” and says he’s “challenged” to set it up.

The Six Dumbest Ideas in Computer Security

The opposite of “Default Permit” is “Default Deny” and it is a really good idea. It takes dedication, thought, and understanding to implement a “Default Deny” policy, which is why it is so seldom done. It’s not that much harder to do than “Default Permit” but you’ll sleep much better at night.

Source: The Six Dumbest Ideas in Computer Security

This was from 2005, and if your organization’s approach was always Default Deny, then it’s a great idea. The problem with Default Deny isn’t the idea, it’s switching away from Default Permit after decades of acclimation. One comment in the HackerNews discussion about this article says it adds “three more steps” to the conversation with IT to fix your app after the change. I would argue that it’s a whole lot more than that.

A friend of mine has trouble running a very important vendor application used in his department. It stopped working some time ago, so he opened a ticket with IT. It was so confusing to them that it got to a point that they allowed him to run Microsoft’s packet capture application on his machine. He followed their instructions, and captured what was going on. But, despite the capture, they were unable to get it working through the firewall.

Out of frustration, he sent the capture to me and asked me to take a look. Even though our laptops are really locked down, as a dev, I get admin on my machine, and I have MSDN, so I downloaded Microsoft’s tool, and looked over the capture. It was a complicated architecture, using a client/server implementation on the local machine. The front end was working over networking ports to talk to its back end, and the back end was using other ports to talk to the vendor’s servers.

I knew what to look for because I had just undergone a lot of pain with my own development workflow, because the company had started doing Default Deny, and it was messing with my development workflow in several ways. Not too long before, they dropped the bomb that EVERY network communication through the firewall required the Cisco Umbrella certificate (so that they can intercept and inspect all traffic, despite SSL). I had to figure this all out on my own, based on the behavior I was seeing, but I was able to figure out how to download that cert and include it in the chain to work with both bundle and HTTP calls in Rails, and psql for Postgres.

Then they locked out Postgres’ default port entirely. They reopened it for me at my request at first. Then six months later, they closed it again, and told me that I needed to formally request permissions for that, which was going to be a many-years-long process. I “just” rewrote the functions I needed in my Rails app to use HTTPS instead of using a direct database connection.

Anyway, I told my friend what ports IT needed to open, and how that he needed to explain to the vendor that they had to include the Umbrella cert in their chain on the back end, but he’s still having problems. Why am I being vague about the details here? It’s not because of confidentiality, though that would apply. No, it’s because my friend had been “working with IT” for over a year to get to this point, and this was two years ago, and I’ve simply forgotten a lot of the details.

“I said all of this to say” that saying it will take “3 extra rounds” is a bit of an understatement when IT starts doing “default deny,” at least in legacy manufacturing companies.

Windows Being Windows, Shills Being Shills

Windows stays prominent because Microsoft caters to corporations which abuse the poor, defenseless OS into doing things like locking users out of changing the desktop background and the sleep timeout. Until Apple offers power-hungry corporate IT middle managers the same level of user-hostile malfeasance in the name of “security,” Microsoft will hold the high ground in corporate deployments.

This becomes a self-perpetuating cycle of not-so-micro-aggressions, as the corporate use of Windows continues to skew all the Microsoft-bought-and-paid-for industry polls that show how much more prevalent Windows is over OSX, and self-justify corporate America that they’re doing the right thing by continuing to stick with it. This, in turn, leads to an entire sub-industry of corporate “security” software which must be installed on Windows, because, well, the bought-and-paid-for auditors told them they had to.

Thus, I wind up with a corporate laptop with 3 different “endpoint” security products installed on it, and something like 30-40 different scripts and checks that run almost by the hour to make sure that the inherent weaknesses of Windows hasn’t compromised our precious meeting PPT’s, which #CorporateIT apparently considers as sensitive as the US nuclear arsenal codes.

Apple offers an alternative to this madness, and I’m very glad they do. In my experience, almost no one runs Windows personally, except for gaming purposes. Numbers like StatCounter vastly over-report Windows usage, because everyone working for a large corporation and in the government is forced to use Windows.

I wish someone would produce a market share report that 1) separates corporate purchases and 2) includes phones as primary computing devices. I think we would see that the “computing world” is vastly different than Gartner would have us believe.

Corporate IT “Support”

I couldn’t mount network drives on my work laptop, so I opened a ticket. That was FIVE weeks ago. The ticket was closed by the first person after 3 days without even contacting me. I immediately re-opened it, and it proceeded to worm its way through an additional EIGHTEEN people, and finally came back to the original person who closed it, who — for at least the 4th time in the email chain — asked me questions I had “pre” answered in the original complaint.

Disgusted, I finally took a couple of hours, and figured it out myself. When I was finishing up, and documenting the fix on the ticket, the first guy contacted me on Teams, and asked me to bring the laptop into the office and try to do what I’ve shown over and over in screenshots couldn’t be done.

I guess I’m just lucky that I have admin rights on my laptop.

FWIW, the problem started when I installed WSL to test using for developing Rails apps. When I recognized the coincidence, and since I hadn’t actually used it yet, I uninstalled the Ubuntu image. This wasn’t enough to fix the problem.

The error message was clearly about not being able to resolve machine names, but I didn’t want to monkey around with the resolution settings because #CorporateIT does nasty things to our laptops, and I was afraid of screwing up their configuration.

I finally removed WSL entirely, and got a different error message when trying to mount a drive, which led me to this Stack Exchange post. I copied the registry setting from a computer which works (which was the same as what was listed in the top answer), and all was sorted again.

To me, the worst part of this charade is that the “original” guy will get the credit for “resolving” the issue, and that really burns me.

Introducing Surface Pro 10 for Business and Surface Laptop 6 for Business | Microsoft Devices Blog

AI-powered PCs built for a new era of work We are excited to announce the first Surface AI PCs built exclusively for business: Surface Pro 10 for Business and Surface Laptop 6 for Business. These new PCs re

Source: Introducing Surface Pro 10 for Business and Surface Laptop 6 for Business | Microsoft Devices Blog

I just love how Microsoft sticks “FOR BIZNESSSSSSS!” on the end of things. “Skype FOR BIZNESSSSSSSS!” “Teams FOR BIDNESS!” “Outlook FOR BIZNAAAAAS!” Sigh. How can you take this stuff seriously? Anyway.

Windows stays because Microsoft caters to corporations which abuse the poor, defenseless OS into doing things like locking users out of changing the desktop background and the sleep timeout. Until Apple offers power-hungry corporate IT middle managers the same level of user-hostile malfeasance in the name of “security,” Microsoft will hold the high ground in corporate deployments.

This becomes a self-perpetuating cycle of not-so-micro-aggressions, as it continues to skew all the Microsoft-bought-and-paid-for Gartner polls that show how much more prevalent Windows is over OSX, and self-justify corporate America that they’re doing the right thing. This, in turn, leads to an entire sub-industry of corporate “security” software which must be installed on Windows, because, bought-and-paid-for auditors told them they had to.

Thus, I wind up with a corporate laptop with something like 3 or 4 different “endpoint” security products installed on it, and something like 30-40 different scripts and checks that run almost by the hour to make sure that the inherent weaknesses of Windows hasn’t compromised our precious meeting PPT’s, which corporate IT considers as sensitive as the US nuclear arsenal codes.

Apple offers an alternative to this madness, and I’m very glad they do. I would rather they stay as they are rather than corrupt their ecosystem in this way to make some corporate sales, and I will happily continue to use my personal MBP to do as much of my work as I can.

We keep making the same mistakes with spreadsheets, despite bad consequences | Ars Technica

Spreadsheets represent unknown risks in the form of errors, privacy violations, trade secrets, and compliance violations. Yet they are also critical for the way many organizations make their decisions. For this reason, they have been described by experts as the “dark matter” of corporate IT.

Source: We keep making the same mistakes with spreadsheets, despite bad consequences | Ars Technica

As I often say, making real applications out of these Frankenstein monsters of data has been my bread and butter throughout my career. The function that the central IT departments in blue chip manufacturing companies could never quite wrap its arms around keeps getting bigger and bigger, and making larger and larger gaps to fill by people in the trenches. So… too right, mate, and keep it up.

Why do people use VBA?

Why do people use VBA? In order to answer this question, we must first look at another question – who actually uses VBA in the first place? In 2021 I ran a poll on /r/vba where I asked redditors why they code in VBA. From these data, we can clearly see that the majority of people who use VBA do so mainly because they have no other choice. Many organisations run their entire business processes with Excel, and when a little bit of automation is required VBA is usually #1 on the list.

Source: Why do people use VBA?

I was just ranting about this to my kids a couple days ago.

Even in large companies, with massive IT departments, and lots and lots of internal databases and information systems, US businesses are still run on Excel. That’s not subjective. I’ve worked for decades inside three Fortune 250’s (and a couple smaller shops), and bad Excel “applications” are in use at all of them. And after one person learns enough VBA to get a spreadsheet dealing with a particular issue to save a little time for themselves, they start sharing them with their colleagues, and the problem gets worse. Half of my career has been built on making “real” applications out of Excel spreadsheets that were wobbling under their own weight.

But why?

Back in the old days, IT grew out of the accounting department. They had the only computer in the building, and it was an IBM mainframe. Great stuff, right? Saved a lot of time and paperwork, right? Except that it didn’t. It quickly ossified the company’s work flow, and permanently hobbled its ability to adapt to change. It would take years to get any changes made in the mainframe group, and people were frustrated. Along came spreadsheets, and everything changed.

I saw it myself in my first engineering job in 1993. We got new computers with Windows 3.1 and Quattro Pro. (And AutoCAD. And, of course, on mine: DOOM!) After weeks of bugging the lady who ran the mainframe — who apparently had to write a whole program — I got her to dump the BOM for a couple of our products to compare for similarities. I downloaded the 2 files to my PC with a token ring mainframe interface card. I think they were only about 1MB each. With 8MB of RAM, I had twice as much memory as our System 36, and I could open both BOM’s in a spreadsheet, and analyze them to my heart’s content. Understanding that I had more processing power on my desk than the freezer-sized unit in the other room was eye-opening.

American manufacturing companies (at least) never got the message. The invention of the spreadsheet spared them from facing the fact that the mainframe had become the black hole of their IT world. As changes were becoming impossible to get from the mainframe group, PC’s with Windows and Excel allowed people at all levels and in all job functions to start working around the mainframe and its limitations.

Now, these kinds of companies are decades behind the curve. They thought “outsourcing” was going to fix all of their problems. When it didn’t, they thought “consultants” would be the trick. Surely “agile” will do it this time, right? No. It’s not the process; it’s the mainframe. Forcing every corporate workflow and piece of data to be kept canonically inside a 40-50-year-old legacy system’s limitations is quite literally killing the company. It’s certainly killing their competitive advantage.

My current company still breaks our primary software component into 8 pieces because that’s what would fit on floppies to send to the plant to program the hardware. Every IT system — and every spreadsheet — in the company has to deal with this 40-year-old legacy issue because that’s what we programmed the mainframe to expect, and now that’s the only way to bill a customer for it. So the logistics of dealing with multiple trees and branches of software (and multiple trees and branches of documentation about the software) is multiplied by a factor of 8 to this day. There is no escape from this black hole. You can’t re-engineer this situation. It’s too ingrained.

I worked for one group which, on every engineering release, had to get a giant table of software versions — each with their 8 part numbers — into the mainframe. The process was so onerous that they would spend days clicking through mainframe terminal emulator screens to get the information they needed, to make a spreadsheet in a particular format, which they would send to another group to actually enter back into the mainframe. Part of the problem was the spreadsheet had to be in 3 columns, but the mainframe screens were in 4 columns (or vice versa), so a lot of it was purely formatting. I wrote a little program to automate all of this, but I’ve left the group, and I’m sure no one uses it any more. The particularly stupid part of this story is that people fought me to write a little software that saved these people 10’s of hours a week in the name of their own job security.

And no one in the corporate hierarchy cares. In this day and age, the executives are all just playing the waiting game, letting things atrophy — saying all the right things publicly — while they wait until the financials are inverted enough to make the company a juicy prospect for a buyout in an industry-wide rollup by private equity.

Meanwhile, actual people have to get stuff done to stay employed and feed their families. Inside the company, the managers have to look at the three year lead times to get a simple application written by “corporate IT,” and can do nothing but just continue to throw bodies and VBA macros at it. Or, in my case, have me write something to do it. That is, until it gets successful enough that people notice, and it gets taken away from me, but that’s another story…

All Your Base Are Belong to Us

If you have a corporate- or school-issued computer, you have no control over it. Unless you wipe it and reinstall the OS, and even then, of course, they could leave things in the BIOS, and probably do. Then again, you barely “own” devices you buy, but that’s another rant. Here’s the task list for my corporate laptop.

Sigh

So let’s see…

  • Seven different reports about what I’m uploading to OneDrive.
  • Five jobs to keep Chrome and Edge up to date. Firefox and IE are also installed.
  • A job to make sure you keep Zoom around.
  • A [REDACTED] hourly job to make sure you haven’t elevated your privileges.
  • A job to make sure you haven’t (apparently) installed the npcap library. I mean, God forbid you should try to use this at a corporate site, which has probably used switching since… 1996 or so.
  • Three other [REDACTED] jobs to make sure you don’t do other things “they” don’t want you to do.
  • At least 5 jobs to make sure you don’t change… anything about the way they’ve installed Office, apparently.

Thirty one jobs. Only one of these is mine, to do the one thing I need this (secondary) computer to do.

This machine bypasses my carefully-curated and ad-blocked local DNS. I don’t know what it uses for DNS, but I see that it doesn’t operate over port 53, and I don’t care to know any more.

It also won’t print to a printer in your house. I think I tried to print to a printer at the office once, and give up after one try, because I knew it was going to be futile. Basically, no one prints anything. They must save a TON on printer costs as a company. Most printouts are a waste of resources anyway, so this might actually be genious.