Corporate IT, NodeJS, “Tech” Companies, and Freaking Microsoft Windows

The Scene

A few years back, as part of a long, slogging series of unfortunate events, I had been tasked with developing a new web application, which circumstances dictated should be written in Java. Books could be written about this one-year period of my career. (And not, like, inspirational ones.) Anyway, part of the process included trying to get people to realize that no one, these days, wrote web apps in Java without using one of the many, popular Javascript libraries for the front end (like React or Angular), and get my management and corporate IT to understand that I needed to install NodeJS on my machine to facilitate this. Up until this point — and despite the fact that it was obviously used by other development teams in the company — it was not on the “approved” list of software to be installed on local machines. Through several strained meetings and rounds of email, someone, somewhere, deep in the bowels of IT, corrected the obvious oversight, and put it on the list.

The production version of NodeJS was 8, at the time of approval.

This kerfuffle was but one small facet in the gem that was this job posting. In the middle development process, I jumped at another job opportunity, and left my Fortune-250 for a different Fortune 250. The IT environment was eerily similar, and led to this post about making Windows tolerable. It was this experience that got me to see the real root of what I’m complaining about here.

And then, through a short series of more unfortunate events — and one amazing event — I came back to the original Fortune 250, in a different department.

Some months later, just after getting settled back in, I got an email asking me if I would approve a new version of NodeJS to be officially blessed and uploaded to the internal repository.

A Symptom, not the Disease

Strangely, I was being asked to approve NodeJS version 9. If you’re not familiar, NodeJS uses a version numbering system like the Linux kernel used to, where even-numbered releases are for production use, and odd-numbered releases are development versions, intended only for development of the software itself. In no way should 9.x be considered for use in projects inside a blue-chip Fortune 250.

I explained this situation to a laundry-list of TO: and CC: recipients in a long email thread that had already been making rounds inside the company before someone finally saw my name attached to the original request, and added me to the chain. Of course, my explanation was ignored, but I only discovered this 6 months later, when I was being asked, again, to approve version 9. Apparently, I was preventing some developer in India from doing his work on a “high priority project” by not having approved it already, and I needed to get on the stick.

I become more blunt, at that point. First, I didn’t do whatever was done to get it certified the first time, so I didn’t know why I was being called on to do it again. Second, I tried to make a case for exempting development libraries, like NodeJS, from the slow process of getting them approved for internal use, and uploaded to our internal software delivery site. This led to another important person added to the chain, who, surprisingly, supported my argument, but, again, nothing changed.

A month later — seven months into this “discussion,” and presumably still holding up a “high priority” project with a “requirement” for 9.x — I got another email, which included a screenshot of an error from Angular, saying that it no longer supported NodeJS 8.x, and that it needed at least version 10.x or 12.x. Again, I pled with the list of people involved in the email chain that we needed to treat development libraries and applications differently than we treated, say, Office applications. I pointed out that, in the time that we had been fussing over version 9, version 14 was now shipping.

Six months after this exchange, I got an email from a desktop support technician. He was asking for clarification about details when installing… wait for it… version 8 on a developer’s computer. That’s right: After over a year of this exercise, we were still fighting to get a version that’s now a year and a half out of support installed on a developer’s machine.

And then, the situation actually got even worse. The developer’s “computer” was really a shared environment (like Citrix, et. al.), and the shared NodeJS install was being constantly re-configured between multiple developers using the same computer between projects. The support person was actually savvy enough to have suspected this, and was asking me about how it worked. I confirmed that this would, indeed, be a problem, and we figured out the flags to install it into each person’s personal directory, and keep the node_modules directory separate, per user. So, at least we figured out how to successfully install a version of Node that was dangerously out of date to a shared computer.

Actually trying to use NodeJS for the job it was created for, and downloading a stack of Javascript libraries to support Angular or React, led to another discussion of how to get it to play nicely with our corporate, Active Directory-authenticated firewall, which — naturally — blocks all access to the internet from anything that doesn’t run through the Windows TCP/IP stack. Say, like npm or yarn trying to access the NPM repository. I had figured out a workaround for that in the first few months of working at the company, and just pointed them at Corkscrew, which transparently handles the NTLM authentication for command-line utilities like npm (or Ruby’s Bundler).

The Root of the Problem: Microsoft, and Windows

If the shared computer had been Linux or Mac, none of these problems would have existed. Each account on Linux and Mac has a proper personal directory, and things like Node and Ruby assume this, and take advantage of it. Each user could install whatever he wanted to in his home directory, and not need administrative permissions on their machine, or have to rely on some internal application-distribution site. Also, if developers could use anything other than Windows, corporate IT would probably not assume that everything which gets forced through the corporate firewall can do NTLM authentication, and force people running tools like NodeJS to rely on a squirrely tool like Corkscrew. Windows has gotten a lot better over the past several years about installing things into a user’s AppData directory, and Microsoft has spent a lot effort in recent years to develop and astroturf WSL(2), Visual Studio Code, and the new Terminal, but Windows is still a second-class citizen for modern web programming.

I try to temper my frustration with this situation with the knowledge that IT departments of large companies have been forced into many, cascadingly-obtuse compromises by their use of Windows. So many frustrations in a company’s user community can be traced back to the relatively quirky, and single-user-oriented way Windows has always worked, and the monoculture that using Windows requires, thanks to Microsoft’s legacy of embrace-and-extend, especially in directory services. The size of the company exacerbates the problem. At my current company, I know of at least 5 different IT org trees. After 6 years of working with various people in these groups, I still have very little understanding who actually owns what. To be fair, most of this is felt by only a small portion of the “power user” community at a company, but that’s most of the people I deal with.

The Distortion of Scale

The biggest problem here is the scale of the operation. When you have 50,ooo nails, you make sure they’re all the same size and finish, and you use the exact same kind of hammer and technique on all of them. You’d think it would be possible to use a bit of manpower in these various IT departments to treat some of these nails differently, but the vast ecosystem required to take care of Windows just eats up all available resources. Anti-virus. VPN. Standard desktops. Scripts to prevent people from doing things they shouldn’t. Scripts to report all activity on the things they should. Office 365. One Drive. Teams. Zoom. Forced password rotations. Worldwide hardware and software upgrades. Locking out how long the screensaver takes to kick in. Preventing changing of custom login screen backgrounds. It’s a lot. I get it. Using Windows as a corporate desktop environment automatically assumes so much work, it leaves little room for treating a computer like a tool that needs to be customized for the job it needs to do, and the work it needs to support, even when those goals are, ostensibly, incidentally, also primary goals of the larger IT organization. It’s a counter-intuitive situation.

I started this post by pointing out that this stack of regrettably-predictable compromises, which result in suboptimal policies and outcomes, is primarily a problem with traditionally non-“tech” companies, but the real, underlying problem is much deeper.

The truth is that all companies are now “tech” companies, whether they realize it or not. And those that can’t change their approach to IT to adapt to this new reality — or change it fast enough to matter — will wither on the vine, and their remaining assets, eventually, will be picked up in a corporate yard sale to companies that have “tech” embedded in their DNA from birth.

I worry that a company which, 30 years later, still breaks up it’s most-important digital asset into 8 pieces because that’s what would fit on a floppy disk will not make the turn in time.

The reason I started writing all of this down was because — after all of this time and discussion — I was asked to approve NodeJS version 10 for the internal software repository. At the time I was asked, version 10 didn’t even show up on the NodeJS release page any more. They were shipping version 16. I guess 10 is better than 8, but let’s be honest: The only reason they gave up on version 8 or 9 is because the version of Angular that they’re using is refusing to work with anything pre-v10. That happened back in Angular version 8, which is now also out of support.

As part of the great email chain, I pleaded with the various people involved with the internal software approval process that keeping up with the shifting versions of your tools and supporting libraries is just part of the job of being a web app developer, yet no one even batted an eye. You would have thought that this concept would have fallen directly under the multi-headed hydra of “security,” and the company’s philosophy seemed to be you can never have too many software layers or policies about it. You would have thought they would have pounced on the concept in order to at least seem serious. I even invoked the specter of the recent, infamous log4j bug, as an example of the risks of letting things get out of date. This issue caused an audit of every Java-based application in the company, so it should have been a touchstone issue which everyone in the chain could relate to. But if anyone could understand what I was trying to say, they apparently didn’t care.

IT Best Practice vs IT Policy

I didn’t much care for The Big Bang Theory, but one scene has stuck with me for a long time. In S1E16, Sheldon is shopping in a store like Best Buy, and some woman comes up to him and asks, “Do you know anything about ‘this stuff?'” He replies, “I know… everything about ‘this stuff.'” And that’s the heck of this situation. It’s almost like every single person concerned with this process has absolutely no idea how any of “this stuff” actually works, and won’t listen to someone who does. And I realize how conceited that may sound, but, in this case, I don’t know how else to put it.

The only other explanation is simply apathy in the face of bureaucracy, and I wish senior IT management would take it on themselves to root out this sort of intransigence, and fix it. It would seem to be their job, and would go a long way to justifying a C-level salary. Unfortunately, this isn’t the first time I’ve found myself trying to explain a direct contradiction of IT best practice versus IT corporate policy to the very people who are supposed to be in charge of both, and I’d like to think I’ve learned how to convey my thoughts in a less confrontational way, but I obviously still haven’t figured out how to motivate people to rise above the internal politics and align the two, and that makes me sad.

I’m finally posting this because I just got another request to approve version 8, now three and a half years on, and I needed to vent.

¯\_(ツ)_/¯

UPDATE: A couple weeks after posting this, I got CC’d on a long desktop support email chain from a developer in India who can’t get angular-cli version 7.x working with npm. <sigh> And there are 4 references to how urgent and how high a priority this is. A simple search shows a pretty detailed SO post about the particular error message, and the general answer seems to be to either play games with the particular versions of the dependencies, or just upgrade to a 8 or 9… three years ago. In any case, this isn’t a desktop support question. IMNSHO, this is squarely a developer’s issue. Sorry, but that’s the job, brother. Do I try, feebly, to make another point, or just let this go?

The Crushing Weight of Knowing What You’re Doing

“Who are you and why are you here?” –Dave Cutler (DaveC)

Source: 012. I Shipped, Therefore I Am

Steven Sinofsky, once a huge wheel at Microsoft, for a very long time, is writing a series of articles chronicling the halcyon days of the early PC business at Substack. I can’t quite bring myself to subscribe, because most of it is free already. Plus, there aren’t many surprises for me, since I was living it during that time.

When Windows NT was introduced, I was quick to jump on board. I was already experimenting with Linux towards the end of ’94. But then I saw a disc of NT 3.5 (not even 3.51 yet) on someone’s bookshelf. He said he wasn’t using it, so I snapped it up and installed it. For the next 20 years, I would dual boot my PC’s between Windows and Linux. I only used Windows for gaming, but for that use, it was obstinate. I tried every incarnation of wine and Crossover and PlayOnLinux and everything else. Nothing has ever let me run Windows games on Linux well enough to warrant getting rid of a native partition.

The content of the slide above is of no consequence, as is pretty much the case with all presentation slides. What’s interesting to me is the little toolbar on the top, left side. It’s from the early Office XP days, back when Microsoft was new and cool. “Before the dark times. Before the empire.” Seeing it evoked a visceral response. As a computer nerd, those really were interesting and exciting times to live through. From the article, that screencap is from 1992. Competing against giants like IBM, HP, and Sun, Microsoft’s eventual dominance was anything but sure at that time. And that’s what’s prompted me to write this anecdote.

In 1995, my Fortune 250 company didn’t even have an internet connection yet. I was using a phone line, and a modem that I conned my boss into letting me get. It was over this modem that I downloaded all 54 floppy drive images of Slackware Linux, on a computer running Windows 3.11 with Trumpet Winsock, connecting to a free SLIP dialup bank in California.

At first, I was much more into NT than Linux. I skipped Windows 95 entirely. I don’t think I ever had a computer that ran it.

I remember how easy it was to setup a dialup connection in NT. By 1996, I was running a dual Pentium Pro with 384 MB of RAM, SCSI hard drives, and a $2,500 video card to do FEA work. The total cost was about $10,000. A coworker got a SGI Indy to do the same sort of work, to the tune of $80,000. The company still didn’t have an internet connection, so he also got an external modem, and hired a local ISP to come set it up. The guy came and screwed around with the connection for 4 hours. I kind of razzed him, by pointing out that it took me all of 15 minutes to configure the same thing on NT. That’s how smug I was about NT versus Unix at the time.

The best part was still to come.

For the next week, the ISP guy still couldn’t get that Indy on the internet. Every time it would connect, the kernel would segfault, and the machine would crash.

But that’s not the best part.

The ISP guy worked with SGI to patch IRIX to fix the modem driver, and finally got it working. My coworker left it connected to the internet all the time to get his email. Things worked fine for a few weeks.

Then the company got a T1 internet connection, and then connected our facility to the main office via a sonet ring. I was really looking forward to not needing my dialup connection any more. But, the first morning, no one could access the internet. Complaints were made. Investigations were performed. Our internal IT would fix the problem, and then it would come back.

Here comes the best part.

Finally, someone realized that computers inside our facility were getting the wrong gateway address to get to the internet. They realized that they were picking up the IP address of my workmate’s Indy, which was advertising itself as a route to the internet, and since the number of hops from computers in the office to the Indy were less than skipping over to the central office, they were preferring its modem, and the Indy’s phone line would choke from the load.

I recall very clearly that there was a simple checkbox in the dialog for setting up a dialup connection in Windows NT for advertising the connection to the LAN as a route to wherever you were connecting. It was on by default, but when I was running through the process, I quickly realized that this was NOT what I wanted, and un-ticked it.

And I felt pretty smug about being serious about NT at the time.

I stuck with NT as my primary interest until some time around 1998 or so. Then Nat Friedman and Miguel de Icaza released Ximian Desktop for Linux, which made Linux on the desktop really pleasant to use. I wasn’t doing analysis work any more. I had transferred to become the system admin of all the Unix machines in the advanced engineering group, so running Linux was a perfect fit. After that, it was pretty much all Linux, all the time, until switching to Macs just a few years ago.

IT Project “Thermocline”

Source: https://brucefwebster.com/2008/04/15/the-wetware-crisis-the-themocline-of-truth

A thermocline is a distinct temperature barrier between a surface layer of warmer water and the colder, deeper water underneath. It can exist in both lakes and oceans. A thermocline can prevent dissolved oxygen from getting to the lower layer and vital nutrients from getting to the upper layer.

In many large or even medium-sized IT projects, there exists a thermocline of truth, a line drawn across the organizational chart that represents a barrier to accurate information regarding the project’s progress. Those below this level tend to know how well the project is actually going; those above it tend to have a more optimistic (if unrealistic) view.

This is all true, but the article assumes that everyone is acting rationally, in service to the stated goal(s) of the project, and that problems with the timeline are just honest mistakes. Unfortunately, in my 25 years, I’ve witnessed a nauseating amount of political infighting that sought to undermine projects in attempts to build and/or preserve personal power. This behavior employs the two things readily at-hand for ruining estimating: bad-faith technical decisions, and good, old-fashioned feet dragging. So the problem isn’t just people being wrong, there’s also a large component due to people actively sabotaging a project for their own purposes. 

I’ve spent most of my career in Fortune 250’s, but I’ve seen this happen in a couple of very small companies too. As someone with a personality that is honest to a fault, this has caused me a significant amount of distress in my career. More than once, I’ve been the lone voice in the wilderness crying about the forthcoming train wrecks, only to be ignored, and then ultimately blamed for the crash, because I was the only one that people could point to for having said anything about it at all.

The Gervais Principle II: Posturetalk, Powertalk, Babytalk and Gametalk

Finally, Sociopaths and Losers speak rarely to each other at all. One of the functions of the Clueless, recall, is to provide a buffer in what would otherwise be a painfully raw master-slave dynamic in a pure Sociopath-Loser organization. But when they do talk, they actually speak an unadorned language you could call Straight Talk if it were worth naming. It is the ordinary (if rare) utilitarian language of the sane, with no ulterior motives flying around. The mean-what-you-say-and-say-what-you-mean stuff between two people in a fixed, asymmetric power relationship, who don’t want or need to play real or fake power games. This is the unmarked black triangle edge in the diagram.

Source: The Gervais Principle II: Posturetalk, Powertalk, Babytalk and Gametalk

I am re-reading the whole series, because it came up as a response to something I said on HN. Basically, I had reinvented this 3-layer dynamic from first principles, based on my observations of the past few years of my career. Now that someone pointed me back to it, I remember reading it originally, but this was written twelve years ago now.

Anyway, this passage really resonates with me. Every time I’ve gotten face time with a serious power broker in a company, this has been true. No games. No BS. Just straight down to business. I have something to say that will help the organization, and they’re ready to hear it and incorporate it. It never accomplishes the full intention, but I understand that they have a lot more pressures that I can see from my vantage point.

… for Sociopaths, conditions of conflict of interest and moral hazard are not exceptional. They are normal, everyday situations.  To function effectively they must constantly maintain and improve their position in the ecosystem of other Sociopaths, protecting themselves, competing, forming alliances, trading favors and building trust. … They never lower their masks. In fact they are their masks. There is nothing beneath.

Though distant from our worlds, criminal worlds have the one advantage that they do not need to maintain the fiction that the organization is not pathological, so they are revealing to study.

For me, as a non-sociopath, this is a source of continual failing: to recognize that the the people pulling the levers of power in the organization are, in fact, sociopathic, and out for their interests, without regard for anyone else’s feelings or fortunes, not mine, or even necessarily the organization’s. Forgetting this base and simple fact has bitten me in the rear end more times than I can count.