Software disenchantment @ tonsky.me

Programs can’t work for years without reboots anymore. Sometimes even days are too much to ask. Random stuff happens and nobody knows why.

What’s worse, nobody has time to stop and figure out what happened. Why bother if you can always buy your way out of it. Spin another AWS instance. Restart process. Drop and restore the whole database. Write a watchdog that will restart your broken app every 20 minutes. Include same resources multiple times, zip and ship. Move fast, don’t fix.

That is not engineering. That’s just lazy programming. Engineering is understanding performance, structure, limits of what you build, deeply. Combining poorly written stuff with more poorly written stuff goes strictly against that. To progress, we need to understand what and why are we doing.

Source: Software disenchantment @ tonsky.me

About 20 years ago, I was working as a Unix sysadmin, and sat in on a meeting about moving an internally-developed application from another data center to mine. It ran on Windows, and died, literally, every day, and required a restart of the whole machine to fix. The manager in the meeting (who, I note, I recommended not be hired, and who was fired for sexual harassment just a few months later) said, “OK, we’ll just schedule it as part of maintenance tasks to preemptively reboot the machine every night.”

I literally snorted. I asked if it were not possible to, you know, actually fix the program? Find the memory leak, or whatever was the problem? I mean, it was written by us; couldn’t we get the programmer to fix their own program? The answer was, of course, no, with the added insinuation that it ridiculous that I suggest that the programmer still had work to do!

About 4 years ago, I wrote a program that helped a lot of people get their jobs done much more easily and efficiently. Per Douglas Adams, “This has made a lot of people very angry and been widely regarded as a bad move.” I was forced to hand the program over to another team, where it has run, with only one tiny patch, for 4 years now. It is not a trivial program, or architecture. To my knowledge, neither the clients nor server ever crash, or need to be restarted. I’m very proud of this.

How 4 Chinese Hackers Allegedly Took Down Equifax | WIRED

While the operation had a certain degree of complexity, Equifax itself made their job much easier than it should have. It should have patched that initial Apache Struts vulnerability, for starters. And an FTC complaint from last summer also found that the company stored administrative credentials in an unsecured file in plaintext. It kept 145 million Social Security numbers and other consumer data in plaintext as well, rather than encrypting them. It failed to segment the databases, which would have limited the fallout. It lacked appropriate file integrity monitoring and used long-expired security certificates. The list goes on. Equifax didn’t just let the alleged Chinese hackers into the vault; it left the skeleton key for every safe deposit box in plain sight.

Source: How 4 Chinese Hackers Allegedly Took Down Equifax | WIRED

This whole incident deeply offends me. I don’t like that our capitalistic society has given these credit-reporting companies so much control over our lives. I don’t like that they seem to be completely unaccountable for being so integral to so much of our economy. I don’t like that they hold all the information you would need to ruin someone’s life by impersonating them online. I don’t like that they are not being prosecuted for being so flippant with personally-identifying data.

I don’t like the fact that a sovereign foreign power committed industrial espionage on a critical part of our economy. I don’t like that they already did basically the same thing to a government personnel database the year before. I don’t like that China’s government exists to begin with, given their treatment of their own people, Hong Kong, and the Uighers. I also don’t like that China has been committing wholesale intellectual property theft for many decades. I don’t like the fact that we all know it, and nothing seems to be getting done to stop it.

The initial vulnerability the attackers leveraged was a problem in Apache Struts, which implies that Equifax’s web application uses Java. Using Java for a web application in 2017 is like driving a Model T in 1950. Sure, it was a reliable means of transportation, and revolutionary when it was introduced, but it’s 20 years out of date. It requires an inordinate amount of maintenance, spare parts are more difficult to find. Mechanics can be lazy, because they know they have the owner of the car over a barrel, and they can charge a premium for service, and take their time. There are many better options available, which start quicker, go faster, have safety features built in, and are far more comfortable.

Not all applications require encrypted this, and sharded that, and intrusion detection systems, and real-time monitoring, and everything else, but if any application needed these sorts of treatments, it would have been this one. Also, if any application needed its owners to stay on top of CVE disclosure reports, and fix affected layers of their stack, as appropriate, it would have been this one.

In sort, there’s literally no good news here. Nothing will happen to China, its government, or the actual individuals named in the indictment. The punishment to Equifax is a slap on the wrist. Everyone jumped on the settlement, and now no one will get anything. Everything about this is wrong, and nothing good will come of it.

n-gate.com. we can’t both be right.

Python dicts are now ordered

February 07, 2020 (comments)

A webshit has something to say about Python internals, but I couldn’t focus on the article, because the first comment on the blog post involves the text “it brings Python on par with PHP,” which is such a monumentally alien thought that I think I need medical attention. Hackernews argues about who already knew this, why, and how. Another argument breaks out about whether this is the Correct and Natural approach to data structures, or if it’s Completely Wrong and Stupid because of some ridiculous edge case nobody cares about. Most of the complaints are from people who are deeply concerned that (entirely hypothetical) existing code might break in the case its author made extremely specific assumptions about one particular data structure in a programming language directly aimed at people who do not give a shit about these topics.

Source: n-gate.com. we can’t both be right.

Arguably one of the pithiest comments on this site yet.

Nibble Stew – a gathering of development thoughts: How about not stabbing ourselves in the leg with a rusty fork?

When faced with this kind of pointless and harmful routine, one might suggest not doing it any more or replacing it with some other, more useful procedure. This does not succeed, of course, but that is not the point. The reasons you get back are the interesting thing, because they will tell you what kind of manager and coworkers you are dealing with. Here are some possible options, can you think of more?

Source: Nibble Stew – a gathering of development thoughts: How about not stabbing ourselves in the leg with a rusty fork?

I can think of another…

The Sock Puppet of the Auditor
“We hired auditors at great expense of time and money, and we’re going to do what they told us that ‘all’ companies do, no matter how inappropriate it is for us and our particular workflows.”

DHH “not yet feeling the awesome” of WSL

This has been one of my all-time favorite Twitter threads. David Heinemeier Hansson, creator of Rails, is trying to “live” in Windows, and set it up to do Rails work. He’s blogging the “experience,” and it’s not going very smoothly. Everyone is telling him to use Windows Services for Linux for this, but he’s “not yet feeling the awesome.” I’ve tried using WSL several times for doing development on Rails applications. I, too, am not feeling the awesome, despite the cheerleading by Windows thought leader Scott Hanselman. Despite my personal experiences with it, given how vociferously people recommend WSL for Rails work on Windows, I kept wondering if I were missing something. I’ll take this as final confirmation that I’m not missing anything, and delete the mental bookmark to go back and try this again when it becomes WSL 3.

IMHO: The Mythical Fullstack Engineer – Stack Overflow Blog

It’s my experience that the above MVFE is pretty uncommon. The profile describes a person with skills requiring thousands of hours to master, but who doesn’t take part in the holistic decision making process. By nature, the value of a fullstack engineer stems from their ability to make competent unilateral decisions (decisions without asking anyone for permission). I’m sure there are people who mostly fit the MVFE, but I wager that they are few and far between. You could probably summarize my view about the MVFE as:

It’s very impractical to become a fullstack engineer without understanding the big picture.

In my mind, a fullstack engineer’s value is mostly derived from their ability to single-handedly design, architect, execute, and operate an entire end-to-end system. Assuming this is possible, it almost completely eliminates integration overhead.

Source: IMHO: The Mythical Fullstack Engineer – Stack Overflow Blog

There’s a lot of subjectivity in this article, but I think it covers the topic pretty well. I consider myself a full-stack engineer, and that self-identification hinges on both emphasized points above. First, I’ve spent the time to learn all the pieces. Other people don’t see all the late nights, banging away on my home lab, or my church’s setup, forcing things to work when they didn’t want to, integrating pieces all over the stack, setting up solutions to help people get things done, or just to scratch an inquisitive itch.

For instance, I’ve run my own web server on a public address. This will cause you realize how hairy the raw, unfiltered internet is, and will force you to learn about the basics of hardening a server and firewalling a network connection in a New York minute. I’ve run my own email server. That will teach you about spam, attachments, white and blacklists, abuse addresses, and lots of stuff about making your server look legit to other servers. I’ve run a Windows domain for my whole family’s computers, with roaming profiles and everything. (Side note: do NOT use roaming profiles.) I’ve run my own personal cloud. My own mobile sync server. Media servers. TV recorders. The list goes on.

In my professional career, I’ve gotten to work on some of the biggest, baddest tech ever made. I’ve setup a Sun E10000 from scratch. When I took the official Sun training on the kit, I fixed the lab’s setup when it broke. I’ve configured a 384-tape robotic backup system. I’ve commissioned $15M of EMC disk cabinetry. I’ve trained extensively on Oracle, and setup a North-American-spanning network of 20 instances. These technological implementations are fading, now, but the concepts haven’t. We just answer the same questions with different hardware and software these days.

When I encounter a new technology, at this point, it usually doesn’t take long to slot it into the larger context of computing services. For instance, I recently tried to use Elasticsearch for a project at work. While I eventually found an easier way to do what I needed to do, through several weeks of experimentation, I now know what that technology is about, what problems it solves, how it works, and what it takes to implement it. Now I have this tool in my toolbox, and it’s very possible that I will yet use it for a different project. I’m incredibly grateful that I have a job where I can occasionally do a little “R&D” like this, to learn something new, but it takes substantially less time to divert my attention like this, than other people might spend, because of the experience I already have.

Second, I understand the field I write software to support, because I studied it. I think the modern incarnation of the programmer, toiling away in the bowels of a big company — which is most developers, by simple numbers — is the total inverse of this ideal. I have a degree in mechanical engineering. I’m a good engineer, for the same reason I consider myself a “full-stack” guy. I see the big picture, and how everything underneath it contributes to making it look the way it does.

Even more than studying the math and physics, I was drawn to engineering, because that’s how my mind works. When I look at an engine, I notice the systems that are interoperating: the mechanical masses, the fluid flows, the thermals, the electrical connections, the air flow. I feel these things in my gut and see them in my mind’s eye. I understand how all of these subsystems work to produce power and torque, the difference between those two things, and when it’s appropriate to focus on one over the other. To me, it’s the same thing with an IT solution. I can picture the large subsystems working together to make up the final system in my head, and see the servers, the services, the networks, the databases, the networking, and the automation that will be needed to implement it.

In a lot of ways, the training in how to think about a problem in engineering school is perfectly suited to creating full-stack solutions. Start with restating the problem. Get to the heart of the business problem you’re trying to solve. Where’s the friction? State the givens. What do we know already? What pieces of data do we have? How do we get that into the system? Finally, specify what you’re solving for. What are we missing? How are we going to transform what we have into what we want? How will the people who will use the system need the program to work, and the data to be shown?

Most people working in software in my industry have been trained in how to write some code, and that’s about the end of it. They might understand how to write a loop in Java, but they don’t understand how to setup a Java application server, or a load balancer, or a firewall, to say nothing about the database. They also don’t understand how our products work, how they’re designed, or what the engineers working on them need to help them in that endeavor. I find myself in the rather rare position of understanding both halves of this equation. In my 25-year career, I’ve met only a handful of people who can straddle the fence between the physical, engineering problem domain, and the IT implementation like this. In the manufacturing world, we are indeed few and far between.

There’s one other thing I want to talk about, and that this article’s presumption that Javascript is the piece for the front-end, in the jigsaw puzzle that is a full-stack web application these days. I still like Rails’ templates, and, of course, Microsoft is pushing Razor. Javascript enhances both of these things. However, the article hints at how a lot of people are doing the entire front-end in Javascript now, and I find that disappointing. In an aborted effort, I tried writing an application in Java with an Angular2 front end. The amount of duplication astounded me. When you combine this duplication with the fact that Java and typed Javascript are two of the most verbose languages to work with, well, you get a mess.

Using an API back-end and a pure Javascript front-end is, perhaps, the single greatest argument against full-stack development you can make. Given the sheer amount of work involved in separating the front-end, completely, from the back-end, it almost requires two different people or teams. If you sat down, and wrote out the most terrible theoretical idea you could come up with for software development, it would probably look like “write a single application, broken right down the middle, in 2 different languages.” Unfortunately, that’s the nature of web development right now. I lament that this is where we’re at in our technological evolution, but until network bandwidth takes another leap forward, this is what we’re stuck with.

.NET Core 3.0 concludes the .NET Framework API porting project · Issue #130 · dotnet/announcements

With .NET Core 3.0, we’re at the point where we’ve ported all technologies that are required for modern workloads, be that desktop apps, mobile apps, console apps, web sites, or cloud services. That’s not to say that we don’t have any gaps or opportunities for new technologies, but we generally believe we won’t be finding them in the .NET Framework code base anymore. Moving forward, we’re focusing our resources on incorporating new technologies.

Source: .NET Core 3.0 concludes the .NET Framework API porting project · Issue #130 · dotnet/announcements

As an almost-20-year hard-core Linux zealot, who watched, in horror, what Microsoft did to the burgeoning personal computing market in the 90’s and early 00’s, I will probably be suspicious of everything they do for the rest of my career. I will never “forgive” them (as if that’s a thing) for funding the SCO v. IBM fiasco, and I still want to know who was involved in that scheme. That being said, even I have a hard time being cynical about .NET Core. This seems to be a good vision and direction, and they are executing very quickly on it. They are delivering what Mono might have been, if more “weight” had been behind it.

A reflection on the departure of RMS – Thomas Bushnell, BSG – Medium

RMS treated the problem as being “let’s make sure we don’t criticize Minsky unfairly”, when the problem was actually, “how can we come to terms with a history of MIT’s institutional neglect of its responsibilities toward women and its apparent complicity with Epstein’s crimes”. While it is true we should not treat Minsky unfairly, it was not — and is not — a pressing concern, and by making it his concern, RMS signaled clearly that it was much more important to him than the question of the institution’s patterns of problematic coddling of bad behavior.

And, I think, some of those focusing themselves on careful parsing of RMS’s words are falling into the same pitfall as he. His intentions do not matter nearly as much as his actions and their predictable effects.

Source: A reflection on the departure of RMS – Thomas Bushnell, BSG – Medium

I don’t want to rehash the story that leads to this; I just thought this was the best take I’ve seen about the situation, and worth capturing for posterity.

Epically-smart people seem highly disposed to self-destructive behavior, which results in alienation that they can only blame on other people. If someone writes an email like Stallman did, and utterly fails to account for the bigger picture, while simultaneously failing to make his comments in a way that doesn’t take enormous academic effort to interpret without revulsion, then, really, how smart is he?

A “public figure” like Stallman (as head of the FSF) must understand that you can’t make comments about something as serious as the Epstein/MIT connection in a casual manner. If you’re going to make a comment, you simply must provide total context. You can’t hide behind excuses, like it was part of a larger thread, or that it was on a private list. As a spokesperson, you have to understand that everyone is watching what you say. To his credit, he didn’t try to have a protracted fight about this.

I’ve watched with great sadness for almost 30 years while Stallman has squandered his beautiful idea with bad politics, and I’ve often wondered why. I suspect this whole situation is a large key to that puzzle. Not that my opinion matters one whit, but I agree with Bushnell that the correct outcome has been achieved.

This bunk bed is $1,200 a month, privacy not included – CNN

Housing costs have become so expensive in some cities that people are renting bunk beds in a communal home for $1,200 a month. Not a bedroom. A bed.

Source: This bunk bed is $1,200 a month, privacy not included – CNN

Given the software/tech-related bent of my news feeds, I see the ridiculous cost of housing in the Valley come up a lot, but I think it’s largely invisible here in the midwest. I’ve posted articles about single bunks in flophouses going for thousands of dollars a month, but now, finally, naturally, there’s someone who has started a bunk-as-a-service company. A “share” allows you to stay in any of their flophouses. At least the CEO seems pretty pragmatic about it, and doesn’t come across as the usual, crazed, psychopathic founder type.