Blogging to work out my thinking about issues in technology, programming, religion, and politics.
Taxpayers spent $15 million on research to build a breakthrough battery. Then the U.S. government gave it to China.
Source: The U.S. made a breakthrough battery discovery — then gave the technology to China
There’s a lot of give and take about the technology here, but it doesn’t seem right that we handed the Chinese government the sole rights to it, and can’t adjust course now that an American company is wanting to get involved again.
A few years back, as part of a long, slogging series of unfortunate events, I had been tasked with developing a new web application, which circumstances dictated should be written in Java. Books could be written about this one-year period of my career. (And not, like, inspirational ones.) Anyway, part of the process included trying to get people to realize that no one, these days, wrote web apps in Java without using one of the many, popular Javascript libraries for the front end (like React or Angular), and get my management and corporate IT to understand that I needed to install NodeJS on my machine to facilitate this. Up until this point — and despite the fact that it was obviously used by other development teams in the company — it was not on the “approved” list of software to be installed on local machines. Through several strained meetings and rounds of email, someone, somewhere, deep in the bowels of IT, corrected the obvious oversight, and put it on the list.
The production version of NodeJS was 8, at the time of approval.
This kerfuffle was but one small facet in the gem that was this job posting. In the middle development process, I jumped at another job opportunity, and left my Fortune-250 for a different Fortune 250. The IT environment was eerily similar, and led to this post about making Windows tolerable. It was this experience that got me to see the real root of what I’m complaining about here.
And then, through a short series of more unfortunate events — and one amazing event — I came back to the original Fortune 250, in a different department.
Some months later, just after getting settled back in, I got an email asking me if I would approve a new version of NodeJS to be officially blessed and uploaded to the internal repository.
Strangely, I was being asked to approve NodeJS version 9. If you’re not familiar, NodeJS uses a version numbering system like the Linux kernel used to, where even-numbered releases are for production use, and odd-numbered releases are development versions, intended only for development of the software itself. In no way should 9.x be considered for use in projects inside a blue-chip Fortune 250.
I explained this situation to a laundry-list of TO: and CC: recipients in a long email thread that had already been making rounds inside the company before someone finally saw my name attached to the original request, and added me to the chain. Of course, my explanation was ignored, but I only discovered this 6 months later, when I was being asked, again, to approve version 9. Apparently, I was preventing some developer in India from doing his work on a “high priority project” by not having approved it already, and I needed to get on the stick.
I become more blunt, at that point. First, I didn’t do whatever was done to get it certified the first time, so I didn’t know why I was being called on to do it again. Second, I tried to make a case for exempting development libraries, like NodeJS, from the slow process of getting them approved for internal use, and uploaded to our internal software delivery site. This led to another important person added to the chain, who, surprisingly, supported my argument, but, again, nothing changed.
A month later — seven months into this “discussion,” and presumably still holding up a “high priority” project with a “requirement” for 9.x — I got another email, which included a screenshot of an error from Angular, saying that it no longer supported NodeJS 8.x, and that it needed at least version 10.x or 12.x. Again, I pled with the list of people involved in the email chain that we needed to treat development libraries and applications differently than we treated, say, Office applications. I pointed out that, in the time that we had been fussing over version 9, version 14 was now shipping.
Six months after this exchange, I got an email from a desktop support technician. He was asking for clarification about details when installing… wait for it… version 8 on a developer’s computer. That’s right: After over a year of this exercise, we were still fighting to get a version that’s now a year and a half out of support installed on a developer’s machine.
And then, the situation actually got even worse. The developer’s “computer” was really a shared environment (like Citrix, et. al.), and the shared NodeJS install was being constantly re-configured between multiple developers using the same computer between projects. The support person was actually savvy enough to have suspected this, and was asking me about how it worked. I confirmed that this would, indeed, be a problem, and we figured out the flags to install it into each person’s personal directory, and keep the node_modules directory separate, per user. So, at least we figured out how to successfully install a version of Node that was dangerously out of date to a shared computer.
Actually trying to use NodeJS for the job it was created for, and downloading a stack of Javascript libraries to support Angular or React, led to another discussion of how to get it to play nicely with our corporate, Active Directory-authenticated firewall, which — naturally — blocks all access to the internet from anything that doesn’t run through the Windows TCP/IP stack. Say, like npm or yarn trying to access the NPM repository. I had figured out a workaround for that in the first few months of working at the company, and just pointed them at Corkscrew, which transparently handles the NTLM authentication for command-line utilities like npm (or Ruby’s Bundler).
If the shared computer had been Linux or Mac, none of these problems would have existed. Each account on Linux and Mac has a proper personal directory, and things like Node and Ruby assume this, and take advantage of it. Each user could install whatever he wanted to in his home directory, and not need administrative permissions on their machine, or have to rely on some internal application-distribution site. Also, if developers could use anything other than Windows, corporate IT would probably not assume that everything which gets forced through the corporate firewall can do NTLM authentication, and force people running tools like NodeJS to rely on a squirrely tool like Corkscrew. Windows has gotten a lot better over the past several years about installing things into a user’s AppData directory, and Microsoft has spent a lot effort in recent years to develop and astroturf WSL(2), Visual Studio Code, and the new Terminal, but Windows is still a second-class citizen for modern web programming.
I try to temper my frustration with this situation with the knowledge that IT departments of large companies have been forced into many, cascadingly-obtuse compromises by their use of Windows. So many frustrations in a company’s user community can be traced back to the relatively quirky, and single-user-oriented way Windows has always worked, and the monoculture that using Windows requires, thanks to Microsoft’s legacy of embrace-and-extend, especially in directory services. The size of the company exacerbates the problem. At my current company, I know of at least 5 different IT org trees. After 6 years of working with various people in these groups, I still have very little understanding who actually owns what. To be fair, most of this is felt by only a small portion of the “power user” community at a company, but that’s most of the people I deal with.
The biggest problem here is the scale of the operation. When you have 50,ooo nails, you make sure they’re all the same size and finish, and you use the exact same kind of hammer and technique on all of them. You’d think it would be possible to use a bit of manpower in these various IT departments to treat some of these nails differently, but the vast ecosystem required to take care of Windows just eats up all available resources. Anti-virus. VPN. Standard desktops. Scripts to prevent people from doing things they shouldn’t. Scripts to report all activity on the things they should. Office 365. One Drive. Teams. Zoom. Forced password rotations. Worldwide hardware and software upgrades. Locking out how long the screensaver takes to kick in. Preventing changing of custom login screen backgrounds. It’s a lot. I get it. Using Windows as a corporate desktop environment automatically assumes so much work, it leaves little room for treating a computer like a tool that needs to be customized for the job it needs to do, and the work it needs to support, even when those goals are, ostensibly, incidentally, also primary goals of the larger IT organization. It’s a counter-intuitive situation.
I started this post by pointing out that this stack of regrettably-predictable compromises, which result in suboptimal policies and outcomes, is primarily a problem with traditionally non-“tech” companies, but the real, underlying problem is much deeper.
The truth is that all companies are now “tech” companies, whether they realize it or not. And those that can’t change their approach to IT to adapt to this new reality — or change it fast enough to matter — will wither on the vine, and their remaining assets, eventually, will be picked up in a corporate yard sale to companies that have “tech” embedded in their DNA from birth.
I worry that a company which, 30 years later, still breaks up it’s most-important digital asset into 8 pieces because that’s what would fit on a floppy disk will not make the turn in time.
The reason I started writing all of this down was because — after all of this time and discussion — I was asked to approve NodeJS version 10 for the internal software repository. At the time I was asked, version 10 didn’t even show up on the NodeJS release page any more. They were shipping version 16. I guess 10 is better than 8, but let’s be honest: The only reason they gave up on version 8 or 9 is because the version of Angular that they’re using is refusing to work with anything pre-v10. That happened back in Angular version 8, which is now also out of support.
As part of the great email chain, I pleaded with the various people involved with the internal software approval process that keeping up with the shifting versions of your tools and supporting libraries is just part of the job of being a web app developer, yet no one even batted an eye. You would have thought that this concept would have fallen directly under the multi-headed hydra of “security,” and the company’s philosophy seemed to be you can never have too many software layers or policies about it. You would have thought they would have pounced on the concept in order to at least seem serious. I even invoked the specter of the recent, infamous log4j bug, as an example of the risks of letting things get out of date. This issue caused an audit of every Java-based application in the company, so it should have been a touchstone issue which everyone in the chain could relate to. But if anyone could understand what I was trying to say, they apparently didn’t care.
I didn’t much care for The Big Bang Theory, but one scene has stuck with me for a long time. In S1E16, Sheldon is shopping in a store like Best Buy, and some woman comes up to him and asks, “Do you know anything about ‘this stuff?'” He replies, “I know… everything about ‘this stuff.'” And that’s the heck of this situation. It’s almost like every single person concerned with this process has absolutely no idea how any of “this stuff” actually works, and won’t listen to someone who does. And I realize how conceited that may sound, but, in this case, I don’t know how else to put it.
The only other explanation is simply apathy in the face of bureaucracy, and I wish senior IT management would take it on themselves to root out this sort of intransigence, and fix it. It would seem to be their job, and would go a long way to justifying a C-level salary. Unfortunately, this isn’t the first time I’ve found myself trying to explain a direct contradiction of IT best practice versus IT corporate policy to the very people who are supposed to be in charge of both, and I’d like to think I’ve learned how to convey my thoughts in a less confrontational way, but I obviously still haven’t figured out how to motivate people to rise above the internal politics and align the two, and that makes me sad.
I’m finally posting this because I just got another request to approve version 8, now three and a half years on, and I needed to vent.
¯\_(ツ)_/¯
A couple weeks after posting this, I got CC’d on a long desktop support email chain from a developer in India who can’t get angular-cli version 7.x working with npm. <sigh> And there are four references to how urgent and how high a priority this is. A simple search shows a pretty detailed SO post about the particular error message, and the general answer seems to be to either play games with the particular versions of the dependencies, or just upgrade to a 8 or 9… three years ago. In any case, this isn’t a desktop support question. IMNSHO, this is squarely a developer’s issue. Sorry, but that’s the job, brother. Do I try, feebly, to make another point, or just let this go?
Because everyone got new laptops, I was looking around the internal company web page for software installation. And what do you think I happened to see? That’s right! Got it in one try! To be fair, there’s a newer version, but this version should simply not exist, anywhere, for any reason, at this point.
FedEx is to close its data centers and retire all of its remaining mainframes within the next two years.
Source: FedEx to close data centers, retire all mainframes by 2024, saving $400m – DCD
Has any company ever actually retired all of their mainframes?
I’ve worked for a company which spent millions to “retire” the mainframe in favor of a OneWorld ERP system, and, of course, we wound up with an ERP system designed by consultants, and, years later, still couldn’t get rid of the mainframe.
That company was “bought” by another company who was implementing Oracle financials to bypass their mainframe for one particular product line. I was the sysadmin for the $20M of Sun/EMC hardware that ran it. At last count, they had spent $110M in the effort, and then my company was raided and sold, and I lost track of what happened.
In my current place, we have numerous little groups all over the company because the mainframe 1) exists, and 2) is a mainframe. One group of people I work with have to spend hours working through mainframe screens to fill out a specifically-formatted Excel spreadsheet to send to another group to actually do the data entry into the mainframe. This type of “solution” to technical debt exists all over the company.
Our technical debt is such that for forty years, we still break our most-important internal software in 8 pieces BECAUSE THAT’S HOW MANY PIECES IT TOOK TO FIT IT ON FLOPPIES to send to the plants to upload it into the product’s computer. EVERY piece of software in the company has to deal with the fact that there are 8 individually-tracked, part-numbered sub-pieces, which are often shared between builds, in a feeble attempt to be slightly more efficient with 1.5 MB files. I worked with another group of people who spend literally DAYS on every release, going through a process of making sure all the pieces disassemble and reassemble correctly.
Does anyone in the world think that we could modernize our internal systems to stop doing this? Could anyone possibly imagine getting rid of the mainframe, which is the central source of authority of this complexity, and the single reason that things can never change? I hate it, but I don’t think there’s any way the company could muster the resources to change this… at least for EXISTING products…
The co-living concept from Brownstone Shared Housing currently has two locations: one in Palo Alto, and another in Bakersfield.
Here’s this article again. It get rewritten every year or so. The dystopian take is that we’re headed for the “pod living” in some Asian countries, but the US has so much land, people will move to other places before living like this in aggregate. We’re just not bound by lack of available land like Japan or Hong Kong. If nothing else, the financials will basically make it happen.
Literally and figuratively
“Contrary to popular belief, Netflix did not kill Blockbuster,” Keyes said. “Blockbuster actually had a better opportunity to be Netflix today than Netflix did, and that’s what I was hoping … to accomplish.”
Source: Former CEO Jim Keyes: Why Blockbuster Really Died and What We Can Learn from It – D Magazine
No, Keyes killed Blockbuster.
I’m watching the Netflix documentary on Prime. It paints a sympathetic picture of Netflix (at least, so far), so I don’t understand why it’s not on Netflix. I’d like to understand why this is the case. Anyway.
In 2007, Blockbuster’s foray into DVD’s-by-mail was going pretty well. They had successfully navigated bringing up a complicated service, and getting a couple million customers. Even though they were still hemorrhaging money at the time, they had something. Around that time, major Blockbuster stakeholder, Carl Icahn, refused to pay the current CEO, John Antioco, his bonus, so he left. Icahn installed Jim Keyes, formerly of 7-11. Keyes wanted to “double down” on the physical stores, and scuttled their postal offering. The documentary has Antioco and the guy running their by-mail service on camera explaining all of this, so this isn’t second-hand hearsay. Yet, here’s Keyes, 10 years later, in 2018, saying that he was trying to lean into the subscription offering, and blaming all of their troubles on banking. I mean, say that you nixed the offering because you had insurmountable debt problems, and hoped that cutting it loose would help you refinance in the current market, but don’t claim that you were hoping to be a “better Netflix than Netflix” when you killed the service.
And, of course, Keyes continued to collect his $750,000/yr salary and $500,000 bonus, in the same year as the company was filing for bankruptcy. This is the disconnect in the American oligarchy. We Americans pride ourselves on our supposed meritocracy, but if we really had a meritocracy, Keyes would only have been able to collect his bonus if he had successfully navigated the banking climate back then, and procured a better exit strategy for Blockbuster than selling it wholesale to Dish. He’s rewriting history here, and I’m betting it’s because he’s looking for another gig. Wikipedia doesn’t list his age, but the date of his MBA puts him still in his early 60’s.
We’ve reached a point with the web now that you can go back pretty far, and still get to actual, reported sources. There’s no running from history when Google makes it so easy to find, and major web sites’ content management systems have gotten so good at keeping their links working…
With the advent of electric and autonomous vehicles, the amount of software required has grown exponentially. That means newly formed teams must grow overnight, which has two extremes of competing difficulties.
I have watched firsthand as a Napolean-esque CEO fired great talent because the CEO’s hardware background didn’t permit comprehension of building a creative, software team. I’ve seen a frustrated director literally (and dangerously) slam a glass-top, conference table while summarily dismissing managers for being honest about the overwhelming deluge of software defects. I’ve watched dozens of corporations believe a revolving door of low-cost staff augmentation from offshore corners of the world may outperform a more-expensive-per-salary team.
Source: The Complex Creation Of Newly Needed Software Teams In The Auto Industry
Legacy transportation companies making the transition from the ECM-centered software development world to the vehicle-as-a-network-of-computers world should probably rethink their software development methodology, tools, and processes as part of the move. If you adopt a completely-new development platform, but continue to use the legacy process — and the middle management that has been running it — you will transplant 25 years of technical debt into the new paradigm, right at the start. It would seem to be the very definition of the job of “information” and “technology” officers of the company to recognize that their company’s software development systems are 25 years behind the current “meta,” reject the entrenched power structures that have restricted progress for decades, and bring in new people who understand modern software development to setup new teams, tools, and processes.
Well, I guess, that would be the approach if you wanted the company to stay relevant for the foreseeable future. However, if your goals are otherwise, it might not make sense. For instance, if you just wanted to spend the next 10 years collecting stock options to exercise at the opportune time — say, for instance, when you know the company has slid into technical irrelevance to the point of becoming an irresistible target for acquisition by a company which has its software development act together — then your approach to the technical debt problem might be different than mine. It might look a lot like doing nothing at all. Which would hardly be surprising. I mean, look at how much companies in the US are willing to sacrifice long-term success for short-term profits, and how well rewarded such behavior is. You wouldn’t be able to fault people inside the company if they look like they are following the exact same strategy, personally.
The biggest issue humanity faces right now isn’t “rogue AI” — it is narrowly selfish people using computer technology to exploit other people in various ways (or at least create a technosphere which does that, with extensive intrusive monitoring and attention direction).
Source: Ask HN: Does Anyone else working in a crypto company feel this is all a scam? | Hacker News
HN finally had a thread about the crypto elephant in the room. This comment contained a truth about the present situation, but the underlying problem is much broader. The mortgage crisis revealed the same structural issues. If you replace the weak phrase “narrowly selfish people” with the proper term — i.e., psychopaths — you can begin to see the general problem.
Whether it’s all the malfeasance around crypto — NFT’s, exchanges, DAO’s, etc. — or colluding to sell bad mortgages to fuel the mortgage-backed securities market, or liquidating excess capacity in your supply chain for short-term cash flow, raising prices, declaring record profits, and then giving yourself a massive bonus for your great work, while at the same time denying workers a raise to at least keep up with the inflation you’re causing, it’s all the same thing: psychopathy. That is, being fine with profiting handsomely while someone else suffers directly and visibly from the decisions you’ve made to do so.
Sideloading would enable bad actors to evade Apple’s privacy and security protections by distributing apps without critical privacy and security checks. These provisions would allow malware, scams and data-exploitation to proliferate.
As if malware, scams, and data-exploitation apps don’t already exist in the App Store. I would argue the opposite of what Apple is claiming, in fact. If some scammer was tricking people into installing a sideloaded app that stole all your data, word would get around, and the traffic pointing to that app would eventually die off. Instead, what we have are lots of crummy apps in the App Store, doing specious things, with Apple’s implicit blessing, with an overwhelmingly-spammed review score. And these things are stubborn.
This guy has made waves pointing out how widespread the problem is.
Apple’s recalcitrance around their walled garden smells funny to me. I get it. I mean, when there is literally no other option for people, you get to act as a middleman on every transaction. But how much money is enough for a company which vies to be the world largest market cap from month to month? Whatever that figure might be, they surely flew past it a long time ago.
“Who are you and why are you here?” –Dave Cutler (DaveC)
Source: 012. I Shipped, Therefore I Am
Steven Sinofsky, once a huge wheel at Microsoft, for a very long time, is writing a series of articles chronicling the halcyon days of the early PC business at Substack. I can’t quite bring myself to subscribe, because most of it is free already. Plus, there aren’t many surprises for me, since I was living it during that time.
When Windows NT was introduced, I was quick to jump on board. I was already experimenting with Linux towards the end of ’94. But then I saw a disc of NT 3.5 (not even 3.51 yet) on someone’s bookshelf. He said he wasn’t using it, so I snapped it up and installed it. For the next 20 years, I would dual boot my PC’s between Windows NT and Linux. I only used Windows for gaming, but for that use, it was obstinate. I tried every incarnation of Wine and Crossover and PlayOnLinux and everything else. Nothing ever let me run Windows games on Linux well enough to warrant getting rid of a native partition.
The content of the slide above is of no consequence, as is pretty much the case with all presentation slides. What’s interesting to me is the little toolbar on the top, left side. It’s from the early Office XP days, back when Microsoft was new and cool. “Before the dark times. Before the empire.” Seeing it evoked a visceral response. As a computer nerd, those really were interesting and exciting times to live through. From the article, that screencap is from 1992. Competing against giants like IBM, HP, and Sun, Microsoft’s eventual dominance was anything but sure at that time. And that’s what’s prompted me to write this anecdote.
In 1995, my Fortune 250 company didn’t even have an internet connection yet. I was using a phone line, and a modem that I conned my boss into letting me get. It was over this modem that I downloaded all 54 floppy drive images of Slackware Linux, on a computer running Windows 3.11 with Trumpet Winsock, connecting to a free SLIP dialup bank in California.
At first, I was much more into NT than Linux. I skipped Windows 95 entirely. I don’t think I ever had a computer that ran it.
I remember how easy it was to setup a dialup connection in NT. By 1996, I was running a dual Pentium Pro with 384 MB of RAM, SCSI hard drives, and a $2,500 video card to do FEA work. The total cost was about $10,000. A coworker got a SGI Indy to do the same sort of work, to the tune of $80,000. The company still didn’t have an internet connection, so he also got an external modem, and hired a local ISP to come set it up. The guy came and screwed around with the connection for 4 hours. I kind of razzed him, by pointing out that it took me all of 15 minutes to configure the same thing on NT. That’s how smug I was about NT versus Unix at the time.
The best part was still to come.
For the next week, the ISP guy still couldn’t get that Indy on the internet. Every time it would connect, the kernel would segfault, and the machine would crash.
But that’s not the best part.
The ISP guy worked with SGI to patch IRIX to fix the modem driver, and finally got it working. My coworker left it connected to the internet all the time to get his email. Things worked fine for a few weeks.
Then the company got a T1 internet connection, and then connected our facility to the main office via a SONET ring. I was really looking forward to not needing my dialup connection any more. But, the first morning, no one could access the internet. Complaints were made. Investigations were performed. Our internal IT would fix the problem, and the next day, it would come back.
Here comes the best part.
Finally, someone realized that computers inside our facility were getting the wrong gateway address to get to the internet. They realized that they were picking up the IP address of my workmate’s Indy, which was advertising itself as a route to the internet, and since the number of hops from computers in the office to the Indy were less than skipping over to the central office, they were preferring its modem, and the Indy’s phone line would choke from the load.
I recall very clearly that there was a simple checkbox in the dialog for setting up a dialup connection in Windows NT for advertising the connection to the LAN as a route to wherever you were connecting. It was on by default, but when I was running through the process, I quickly realized that this was NOT what I wanted, and un-ticked it.
And I felt pretty smug about being serious about NT at the time.
I stuck with NT as my primary interest until some time around 1998 or so. Then Nat Friedman and Miguel de Icaza released Ximian Desktop for Linux, which made Linux on the desktop really pleasant to use. I wasn’t doing analysis work any more. I had transferred to become the system admin of all the Unix machines in the advanced engineering group, so running Linux was a perfect fit. After that, it was pretty much all Linux, all the time, until switching to Macs just a few years ago.