This, I think, is my point: The system is complicit in the abuse.
And right now, right here, YouTube and Google are complicit in that system. The architecture they have built to extract the maximum revenue from online video is being hacked by persons unknown to abuse children, perhaps not even deliberately, but at a massive scale. I believe they have an absolute responsibility to deal with this, just as they have a responsibility to deal with the radicalisation of (mostly) young (mostly) men via extremist videos — of any political persuasion. They have so far showed absolutely no inclination to do this, which is in itself despicable. However, a huge part of my troubled response to this issue is that I have no idea how they can respond without shutting down the service itself, and most systems which resemble it. We have built a world which operates at scale, where human oversight is simply impossible, and no manner of inhuman oversight will counter most of the examples I’ve used in this essay. The asides I’ve kept in parentheses throughout, if expanded upon, would allow one with minimal effort to rewrite everything I’ve said, with very little effort, to be not about child abuse, but about white nationalism, about violent religious ideologies, about fake news, about climate denialism, about 9/11 conspiracies.
This is simply not true. It’s not true at all. Google made 85 BILLION dollars last year. They absolutely, positively, unquestionably can invest in some more machines to flag more types of content, and hire people to review the flags.
And don’t try to tell me they couldn’t programmatically de-list the kinds of accounts that are pumping out the kind of generative garbage described in the article. I could write a 100-line Perl script to catch this. It’s like the argument about how the App Store is so big that Apple couldn’t possibly catch all the fraudulent apps, but one guy looking at it in his spare time has identified scores of easily-caught problems that scam hundreds of millions of dollars out of the ecosystem.
At the end of the day, it’s a problem with misaligned incentives. Just like with Apple and the App Store, Google doesn’t want to fix the problem, because they benefit from the algorithmic/generative advertisement click-bait fraud scheme made possible by their platform being “game-able.” Corporations being the beasts they are, the only way to solve this problem is through legislation. Unfortunately, campaign finance laws being the beasts they are, that’s not going to happen.
And, as if on cue:
Zhukov’s trial established how the trade in fake clicks works. Between 2014 and 2016, the so-called King of Fraud—a name he gave himself in a text message, revealed in court—ran an advertising network called Media Methane, which received payments from other advertising networks in return for placing brand’s adverts on websites. But the company did not place those adverts on real websites. Instead it created fake ones, spoofing more than 6,000 domains. It then rented 2,000 computer servers in Texas and Amsterdam and programmed them to simulate the way a human would act on a website—using a fake mouse to scroll the fake website and falsely appearing to be signed in to Facebook.
Click fraud has been around since the rise of Google, but I guess everyone collectively agreed to ignore it as a cost of doing business, like “shrinkage” in retail. It stands to reason that these efforts have gone full-blown industrial now, and surely must be making a dent in someone’s pocketbook, but I guess everyone in the advertising economy is too entrenched now to do anything different. Advertising may be the single biggest sector in the American economy at this point. So they go after one dude, and make an example of him, meanwhile, the algorithmically-generated advertisement-bait is considered legitimate.
“Algorithms” are ruining everything that made pop culture interesting.
Office 365 went from being an online version of Microsoft Office to being a bossware delivery-system. The Office 365 sales-pitch focuses on fine-grained employee tracking and comparison, so bosses can rank their workers’ performance against each other. But beyond this automated gladitorial keystroke combat, Offce 365’s analytics will tell you how your company performs against other companies.
That’s right – Microsoft will spy on your competitors and sell you access to their metrics. It’s wild, but purchasing managers who hear this pitch seem completely oblivious to the implication of this: that Microsoft will also spy on you and deliver your metrics to your competitors.
I feel like a fool. I watch Microsoft like a hawk, and I didn’t even know about this. Every time I think I’m too cynical about a FAANG company — and Microsoft in particular — I find that I haven’t been nearly cynical enough.
With this new LinkedIn connection, in Outlook, it’s now possible for Microsoft to connect a particular person to a particular user in your current company’s “metrics.” I suppose they could use this to juice search results for recruiters in LinkedIn, or provide reports to potential employers. I wouldn’t put any of this past them.
There’s a lot of give and take about the technology here, but it doesn’t seem right that we handed the Chinese government the sole rights to it, and can’t adjust course now that an American company is wanting to get involved again.
A few years back, as part of a long, slogging series of unfortunate events, I had been tasked with developing a new web application, which circumstances dictated should be written in Java. Books could be written about this one-year period of my career. (And not, like, inspirational ones.) Anyway, part of the process included trying to get people to realize that no one, these days, wrote web apps in Java without using one of the many, popular Javascript libraries for the front end (like React or Angular), and get my management and corporate IT to understand that I needed to install NodeJS on my machine to facilitate this. Up until this point — and despite the fact that it was obviously used by other development teams in the company — it was not on the “approved” list of software to be installed on local machines. Through several strained meetings and rounds of email, someone, somewhere, deep in the bowels of IT, corrected the obvious oversight, and put it on the list.
The production version of NodeJS was 8, at the time of approval.
This kerfuffle was but one small facet in the gem that was this job posting. In the middle development process, I jumped at another job opportunity, and left my Fortune-250 for a different Fortune 250. The IT environment was eerily similar, and led to this post about making Windows tolerable. It was this experience that got me to see the real root of what I’m complaining about here.
And then, through a short series of more unfortunate events — and one amazing event — I came back to the original Fortune 250, in a different department.
Some months later, just after getting settled back in, I got an email asking me if I would approve a new version of NodeJS to be officially blessed and uploaded to the internal repository.
A Symptom, not the Disease
Strangely, I was being asked to approve NodeJS version 9. If you’re not familiar, NodeJS uses a version numbering system like the Linux kernel used to, where even-numbered releases are for production use, and odd-numbered releases are development versions, intended only for development of the software itself. In no way should 9.x be considered for use in projects inside a blue-chip Fortune 250.
I explained this situation to a laundry-list of TO: and CC: recipients in a long email thread that had already been making rounds inside the company before someone finally saw my name attached to the original request, and added me to the chain. Of course, my explanation was ignored, but I only discovered this 6 months later, when I was being asked, again, to approve version 9. Apparently, I was preventing some developer in India from doing his work on a “high priority project” by not having approved it already, and I needed to get on the stick.
I become more blunt, at that point. First, I didn’t do whatever was done to get it certified the first time, so I didn’t know why I was being called on to do it again. Second, I tried to make a case for exempting development libraries, like NodeJS, from the slow process of getting them approved for internal use, and uploaded to our internal software delivery site. This led to another important person added to the chain, who, surprisingly, supported my argument, but, again, nothing changed.
A month later — seven months into this “discussion,” and presumably still holding up a “high priority” project with a “requirement” for 9.x — I got another email, which included a screenshot of an error from Angular, saying that it no longer supported NodeJS 8.x, and that it needed at least version 10.x or 12.x. Again, I pled with the list of people involved in the email chain that we needed to treat development libraries and applications differently than we treated, say, Office applications. I pointed out that, in the time that we had been fussing over version 9, version 14 was now shipping.
Six months after this exchange, I got an email from a desktop support technician. He was asking for clarification about details when installing… wait for it… version 8 on a developer’s computer. That’s right: After over a year of this exercise, we were still fighting to get a version that’s now a year and a half out of support installed on a developer’s machine.
And then, the situation actually got even worse. The developer’s “computer” was really a shared environment (like Citrix, et. al.), and the shared NodeJS install was being constantly re-configured between multiple developers using the same computer between projects. The support person was actually savvy enough to have suspected this, and was asking me about how it worked. I confirmed that this would, indeed, be a problem, and we figured out the flags to install it into each person’s personal directory, and keep the node_modules directory separate, per user. So, at least we figured out how to successfully install a version of Node that was dangerously out of date to a shared computer.
Actually trying to use NodeJS for the job it was created for, and downloading a stack of Javascript libraries to support Angular or React, led to another discussion of how to get it to play nicely with our corporate, Active Directory-authenticated firewall, which — naturally — blocks all access to the internet from anything that doesn’t run through the Windows TCP/IP stack. Say, like npm or yarn trying to access the NPM repository. I had figured out a workaround for that in the first few months of working at the company, and just pointed them at Corkscrew, which transparently handles the NTLM authentication for command-line utilities like npm (or Ruby’s Bundler).
The Root of the Problem: Microsoft, and Windows
If the shared computer had been Linux or Mac, none of these problems would have existed. Each account on Linux and Mac has a proper personal directory, and things like Node and Ruby assume this, and take advantage of it. Each user could install whatever he wanted to in his home directory, and not need administrative permissions on their machine, or have to rely on some internal application-distribution site. Also, if developers could use anything other than Windows, corporate IT would probably not assume that everything which gets forced through the corporate firewall can do NTLM authentication, and force people running tools like NodeJS to rely on a squirrely tool like Corkscrew. Windows has gotten a lot better over the past several years about installing things into a user’s AppData directory, and Microsoft has spent a lot effort in recent years to develop and astroturf WSL(2), Visual Studio Code, and the new Terminal, but Windows is still a second-class citizen for modern web programming.
I try to temper my frustration with this situation with the knowledge that IT departments of large companies have been forced into many, cascadingly-obtuse compromises by their use of Windows. So many frustrations in a company’s user community can be traced back to the relatively quirky, and single-user-oriented way Windows has always worked, and the monoculture that using Windows requires, thanks to Microsoft’s legacy of embrace-and-extend, especially in directory services. The size of the company exacerbates the problem. At my current company, I know of at least 5 different IT org trees. After 6 years of working with various people in these groups, I still have very little understanding who actually owns what. To be fair, most of this is felt by only a small portion of the “power user” community at a company, but that’s most of the people I deal with.
The Distortion of Scale
The biggest problem here is the scale of the operation. When you have 50,ooo nails, you make sure they’re all the same size and finish, and you use the exact same kind of hammer and technique on all of them. You’d think it would be possible to use a bit of manpower in these various IT departments to treat some of these nails differently, but the vast ecosystem required to take care of Windows just eats up all available resources. Anti-virus. VPN. Standard desktops. Scripts to prevent people from doing things they shouldn’t. Scripts to report all activity on the things they should. Office 365. One Drive. Teams. Zoom. Forced password rotations. Worldwide hardware and software upgrades. Locking out how long the screensaver takes to kick in. Preventing changing of custom login screen backgrounds. It’s a lot. I get it. Using Windows as a corporate desktop environment automatically assumes so much work, it leaves little room for treating a computer like a tool that needs to be customized for the job it needs to do, and the work it needs to support, even when those goals are, ostensibly, incidentally, also primary goals of the larger IT organization. It’s a counter-intuitive situation.
I started this post by pointing out that this stack of regrettably-predictable compromises, which result in suboptimal policies and outcomes, is primarily a problem with traditionally non-“tech” companies, but the real, underlying problem is much deeper.
The truth is that all companies are now “tech” companies, whether they realize it or not. And those that can’t change their approach to IT to adapt to this new reality — or change it fast enough to matter — will wither on the vine, and their remaining assets, eventually, will be picked up in a corporate yard sale to companies that have “tech” embedded in their DNA from birth.
I worry that a company which, 30 years later, still breaks up it’s most-important digital asset into 8 pieces because that’s what would fit on a floppy disk will not make the turn in time.
The reason I started writing all of this down was because — after all of this time and discussion — I was asked to approve NodeJS version 10 for the internal software repository. At the time I was asked, version 10 didn’t even show up on the NodeJS release page any more. They were shipping version 16. I guess 10 is better than 8, but let’s be honest: The only reason they gave up on version 8 or 9 is because the version of Angular that they’re using is refusing to work with anything pre-v10. That happened back in Angular version 8, which is now also out of support.
As part of the great email chain, I pleaded with the various people involved with the internal software approval process that keeping up with the shifting versions of your tools and supporting libraries is just part of the job of being a web app developer, yet no one even batted an eye. You would have thought that this concept would have fallen directly under the multi-headed hydra of “security,” and the company’s philosophy seemed to be you can never have too many software layers or policies about it. You would have thought they would have pounced on the concept in order to at least seem serious. I even invoked the specter of the recent, infamous log4j bug, as an example of the risks of letting things get out of date. This issue caused an audit of every Java-based application in the company, so it should have been a touchstone issue which everyone in the chain could relate to. But if anyone could understand what I was trying to say, they apparently didn’t care.
IT Best Practice vs IT Policy
I didn’t much care for The Big Bang Theory, but one scene has stuck with me for a long time. In S1E16, Sheldon is shopping in a store like Best Buy, and some woman comes up to him and asks, “Do you know anything about ‘this stuff?'” He replies, “I know… everything about ‘this stuff.'” And that’s the heck of this situation. It’s almost like every single person concerned with this process has absolutely no idea how any of “this stuff” actually works, and won’t listen to someone who does. And I realize how conceited that may sound, but, in this case, I don’t know how else to put it.
The only other explanation is simply apathy in the face of bureaucracy, and I wish senior IT management would take it on themselves to root out this sort of intransigence, and fix it. It would seem to be their job, and would go a long way to justifying a C-level salary. Unfortunately, this isn’t the first time I’ve found myself trying to explain a direct contradiction of IT best practice versus IT corporate policy to the very people who are supposed to be in charge of both, and I’d like to think I’ve learned how to convey my thoughts in a less confrontational way, but I obviously still haven’t figured out how to motivate people to rise above the internal politics and align the two, and that makes me sad.
I’m finally posting this because I just got another request to approve version 8, now three and a half years on, and I needed to vent.
¯\_(ツ)_/¯
Update 1
A couple weeks after posting this, I got CC’d on a long desktop support email chain from a developer in India who can’t get angular-cli version 7.x working with npm. <sigh> And there are four references to how urgent and how high a priority this is. A simple search shows a pretty detailed SO post about the particular error message, and the general answer seems to be to either play games with the particular versions of the dependencies, or just upgrade to a 8 or 9… three years ago. In any case, this isn’t a desktop support question. IMNSHO, this is squarely a developer’s issue. Sorry, but that’s the job, brother. Do I try, feebly, to make another point, or just let this go?
Update 2, eight months later
Because everyone got new laptops, I was looking around the internal company web page for software installation. And what do you think I happened to see? That’s right! Got it in one try! To be fair, there’s a newer version, but this version should simply not exist, anywhere, for any reason, at this point.
Has any company ever actually retired all of their mainframes?
I’ve worked for a company which spent millions to “retire” the mainframe in favor of a OneWorld ERP system, and, of course, we wound up with an ERP system designed by consultants, and, years later, still couldn’t get rid of the mainframe.
That company was “bought” by another company who was implementing Oracle financials to bypass their mainframe for one particular product line. I was the sysadmin for the $20M of Sun/EMC hardware that ran it. At last count, they had spent $110M in the effort, and then my company was raided and sold, and I lost track of what happened.
In my current place, we have numerous little groups all over the company because the mainframe 1) exists, and 2) is a mainframe. One group of people I work with have to spend hours working through mainframe screens to fill out a specifically-formatted Excel spreadsheet to send to another group to actually do the data entry into the mainframe. This type of “solution” to technical debt exists all over the company.
Our technical debt is such that for forty years, we still break our most-important internal software in 8 pieces BECAUSE THAT’S HOW MANY PIECES IT TOOK TO FIT IT ON FLOPPIES to send to the plants to upload it into the product’s computer. EVERY piece of software in the company has to deal with the fact that there are 8 individually-tracked, part-numbered sub-pieces, which are often shared between builds, in a feeble attempt to be slightly more efficient with 1.5 MB files. I worked with another group of people who spend literally DAYS on every release, going through a process of making sure all the pieces disassemble and reassemble correctly.
Does anyone in the world think that we could modernize our internal systems to stop doing this? Could anyone possibly imagine getting rid of the mainframe, which is the central source of authority of this complexity, and the single reason that things can never change? I hate it, but I don’t think there’s any way the company could muster the resources to change this… at least for EXISTING products…
Here’s this article again. It get rewritten every year or so. The dystopian take is that we’re headed for the “pod living” in some Asian countries, but the US has so much land, people will move to other places before living like this in aggregate. We’re just not bound by lack of available land like Japan or Hong Kong. If nothing else, the financials will basically make it happen.
“Contrary to popular belief, Netflix did not kill Blockbuster,” Keyes said. “Blockbuster actually had a better opportunity to be Netflix today than Netflix did, and that’s what I was hoping … to accomplish.”
I’m watching the Netflix documentary on Prime. It paints a sympathetic picture of Netflix (at least, so far), so I don’t understand why it’s not on Netflix. I’d like to understand why this is the case. Anyway.
In 2007, Blockbuster’s foray into DVD’s-by-mail was going pretty well. They had successfully navigated bringing up a complicated service, and getting a couple million customers. Even though they were still hemorrhaging money at the time, they had something. Around that time, major Blockbuster stakeholder, Carl Icahn, refused to pay the current CEO, John Antioco, his bonus, so he left. Icahn installed Jim Keyes, formerly of 7-11. Keyes wanted to “double down” on the physical stores, and scuttled their postal offering. The documentary has Antioco and the guy running their by-mail service on camera explaining all of this, so this isn’t second-hand hearsay. Yet, here’s Keyes, 10 years later, in 2018, saying that he was trying to lean into the subscription offering, and blaming all of their troubles on banking. I mean, say that you nixed the offering because you had insurmountable debt problems, and hoped that cutting it loose would help you refinance in the current market, but don’t claim that you were hoping to be a “better Netflix than Netflix” when you killed the service.
And, of course, Keyes continued to collect his $750,000/yr salary and $500,000 bonus, in the same year as the company was filing for bankruptcy. This is the disconnect in the American oligarchy. We Americans pride ourselves on our supposed meritocracy, but if we really had a meritocracy, Keyes would only have been able to collect his bonus if he had successfully navigated the banking climate back then, and procured a better exit strategy for Blockbuster than selling it wholesale to Dish. He’s rewriting history here, and I’m betting it’s because he’s looking for another gig. Wikipedia doesn’t list his age, but the date of his MBA puts him still in his early 60’s.
We’ve reached a point with the web now that you can go back pretty far, and still get to actual, reported sources. There’s no running from history when Google makes it so easy to find, and major web sites’ content management systems have gotten so good at keeping their links working…
With the advent of electric and autonomous vehicles, the amount of software required has grown exponentially. That means newly formed teams must grow overnight, which has two extremes of competing difficulties.
I have watched firsthand as a Napolean-esque CEO fired great talent because the CEO’s hardware background didn’t permit comprehension of building a creative, software team. I’ve seen a frustrated director literally (and dangerously) slam a glass-top, conference table while summarily dismissing managers for being honest about the overwhelming deluge of software defects. I’ve watched dozens of corporations believe a revolving door of low-cost staff augmentation from offshore corners of the world may outperform a more-expensive-per-salary team.
Legacy transportation companies making the transition from the ECM-centered software development world to the vehicle-as-a-network-of-computers world should probably rethink their software development methodology, tools, and processes as part of the move. If you adopt a completely-new development platform, but continue to use the legacy process — and the middle management that has been running it — you will transplant 25 years of technical debt into the new paradigm, right at the start. It would seem to be the very definition of the job of “information” and “technology” officers of the company to recognize that their company’s software development systems are 25 years behind the current “meta,” reject the entrenched power structures that have restricted progress for decades, and bring in new people who understand modern software development to setup new teams, tools, and processes.
Well, I guess, that would be the approach if you wanted the company to stay relevant for the foreseeable future. However, if your goals are otherwise, it might not make sense. For instance, if you just wanted to spend the next 10 years collecting stock options to exercise at the opportune time — say, for instance, when you know the company has slid into technical irrelevance to the point of becoming an irresistible target for acquisition by a company which has its software development act together — then your approach to the technical debt problem might be different than mine. It might look a lot like doing nothing at all. Which would hardly be surprising. I mean, look at how much companies in the US are willing to sacrifice long-term success for short-term profits, and how well rewarded such behavior is. You wouldn’t be able to fault people inside the company if they look like they are following the exact same strategy, personally.
The biggest issue humanity faces right now isn’t “rogue AI” — it is narrowly selfish people using computer technology to exploit other people in various ways (or at least create a technosphere which does that, with extensive intrusive monitoring and attention direction).
HN finally had a thread about the crypto elephant in the room. This comment contained a truth about the present situation, but the underlying problem is much broader. The mortgage crisis revealed the same structural issues. If you replace the weak phrase “narrowly selfish people” with the proper term — i.e., psychopaths — you can begin to see the general problem.
Whether it’s all the malfeasance around crypto — NFT’s, exchanges, DAO’s, etc. — or colluding to sell bad mortgages to fuel the mortgage-backed securities market, or liquidating excess capacity in your supply chain for short-term cash flow, raising prices, declaring record profits, and then giving yourself a massive bonus for your great work, while at the same time denying workers a raise to at least keep up with the inflation you’re causing, it’s all the same thing: psychopathy. That is, being fine with profiting handsomely while someone else suffers directly and visibly from the decisions you’ve made to do so.
.jpeg)
