Security Theater

    [nhh] yep, I can remember when everyone's Microsoft workstation was
          given a public IP. Things were done ...differently back then
    [thrig] (back then, the time for an unpatched windows system being
            connected to the network to be hacked was about 30 seconds.
            no, there was no site-wide firewall)
    [quintin] That's so wild to me.. I've never been at a place
            that didn't have some sort of firewall.

The context was PLC or Programmable Logic Controllers, and why various industrial things should maybe be kept off the internet, or hopefully be at least firewalled, as best one can. Firewalls are complicated; wouldn't it be nice to be rid of them? Sure, but, here we are.

    [pa2] I walked into one control room and they had disconnected a CCYV
          monitor and were using it to play xbox with. General rule:
          Chemical Plant operators have too much time on their hands and
          will try to "Use" anything.

In my case it was genome sequencers, which back then were big bulky expensive things connected to a Windows system (or possibly DOS, or possibly both, where you'd pull out a drive tray, and flip it 180 degrees to load the other operating system that ran different software). The problem here was the the vendor insisted that the Windows system never be patched, set against the fact that an unpatched Windows system would last about 30 seconds on the network before being hacked. One solution is to keep Windows off the network, perhaps placing a ball of masking tape in the network port and then taping that over with a "NO!" label to lower, but not eliminate, the odds of someone incorrectly networking the host up. No network would mean all data transfers would be done via sneakernet, or to walk from computer A to computer B with a portable drive. The lab techs were not a fan of such.

Sneakernet can still be a problem, if, say, someone plugs the wrong USB drive into the wrong system, and maybe if an attacker is willing to drop a bunch of zero-days on you, or close enough to you, Stuxnet being a thing. USB may be nice, but does have quite a lot of features and complexity and thus attack surface to play around with. Various operating systems also haven't been shy about executing random code found on random devices. What about better network isolation, maybe with a VLAN? VLAN were still a new thing to the central network group a decade or more after this story, and running a physically different "janky lab device" network was also not an option, nor maybe a good one: political issues with the central network group, cost, etc. And of course network wires will be run in wacky ways, so it's pretty easy for that separate network to get bridged onto the internet. One job of a sysadmin is to chase network cables, up into the ceiling, across three rooms, and who owns this system, and what is it for? They wanted to offer same-subnet NFS, but the network three rooms over was different, so they ran a wire to solve their problem. Luckily they had not yet graduated, so were able to explain the setup. A loop had been created by people in that lab plugging random cables into random switches in the room.

Not a problem if the equipment is airgapped as it should be.

Until a vendor comes in to perform service, using an infected USB drive. Then the staff getting the gel images off the device with their USB drives get theirs infected, suddenly you have a dozen machines trying to connect to a former Soviet bloc country so they can install ransomware.

Ask me how I know.

So, for the sequencer, since the lab techs did not want to sneakernet, but Windows could not be patched, the systems were networked up, but only so that data could be transferred via SCP and little if nothing else. The sequencer vendor did bristle, a lot, about the SSH client. But the servers were linux, and the admin of that group (it wasn't me, but I was involved in this mess) did not want to setup Samba, so transfer via SSH it was.

This was back when sub-groups in a department had their own sysadmin (there was money being thrown around for the Human Genome Project) rather than consolidating everything into one cost-saving IT team. Those extra humans cost money, and you wouldn't want the economy to be less productive, now would you? Insider attacks can be harder when systems are more partitioned off, but where is the training and payroll and equipment and server rooms for all those tertiary adjuncts coming from?

The next thing the lab techs asked for was web access, so they could check their email. No SmartPhone for a distraction, back then, and the sequencers did take a while to run. Web access was denied; use your linux desktop for that, it's not too far away, has a browser, games. Yes—linux desktop. The department was, for desktops, about 30 percent linux, 30 percent Mac, and 30 percent Windows, plus some SGI, and a wacky HP running some pascal-based operating system. The server room was even more diverse, Solaris, SGI, linux, HP/UX, OpenBSD, Digital UNIX, and, thank goodness, no Windows. Anyways, the web Did Not Work™ because someone had installed a transparent bridging firewall between those extraordinarily hackable Windows systems and the rest of the Internet. I forget the exact rules, but it was not much more than DHCP, maybe DNS, maybe NTP, and SSH only to the relevant local subnets (subnets were in theory one per building floor, but our floor was so populated as to eventually need two subnets). Anything else, inbound or outbound? Blocked. Are such firewalls complicated? Indeed! Are there complicated hoops involved? Indeed! But what less bad alternative has been offered? It's not like there's a plethora of sequencers to pick from, and their few competitors are doubtless just as jank. Even with formal verification (if and when that arrives…) one may still want defense in depth, just as a well designed boat may have various complications to lower the risk of catastrophic failure. Speeding near icebergs in an unsinkable ship with lifeboat problems? Can do!

This was back when NAT was a somewhat new thing, so there was no "inbound connections are by default blocked" from a wifi slinging wall-wart, and operating systems had lots of hackable services, and people looking all the time. Hence the 30 second lifetime of an exposed Windows system. The SGI would crash if you scanned them with nmap and certain flags, so it wasn't just Windows that was terrible. Again, there was no site-wide firewall. "Open access for research purposes", "be permissive in what you accept", etc. Folks therefore came up with various strange firewall designs, one where you'd put a public IP address on the server that also ran DHCP on a RFC 1918 subnet, and NAT was used so the DHCP clients could get to the Internet through the server. What's to stop a client system from assigning itself a static address on the public network for a total firewall bypass? Nothing besides the ignorance of "you can do that?". Network gear these days may have additional complexity to block traffic from rogue DHCP servers, and modern firewalls are mostly better about not allowing total bypasses, until someone wires up the new routers wrongly and forgets to test things. Yay, the network is now less bad. And is more complicated.

There remain risks for the genome sequencer even with the few protocols allowed through the firewall; a non-zero number of bugs have been (and likely will be) found in NTP, DHCP, DNS, and SSH clients whereby a malicious server can compromise the client. DNS (and ICMP!) can be a great way to exfiltrate data or even tunnel a remote shell, if an attacker can get some toe-hold. The malicious server risk was low (but can be high in other contexts); the servers were operated by admins who also had physical access to the servers and sequencers. The sequencers (and some of the servers!) were sitting in hallways that the public could in theory reach if they wandered far enough into the building. Data centers were a new thing for most departments; most of our servers were in an often overheated storage closet where someone got second degree burns getting to the "turn it off" switch back behind the racks. These days there is building keycard access and hopefully less politeness in holding the door. More complexity and barriers! And so society evolves. A much larger and more credible risk to the sequencer would have been some virus or worm directly infecting it, or one downloaded via some website or email, and then someone would need to clean and reinstall it (a bespoke and time consuming install, as contrasted with automated JumpStart installs for the Solaris systems). Therefore blocking everything except a minimum to get files off of it seemed reasonable. Complexity and hoops to jump through? Sure, but there are tradeoffs and negotiations going on between the vendor, server admins, and lab techs—no Samba, yes SSH client, no sneakernet—as weighed against the risks and annoyances involved.

Why not simply patch the Windows system? Patching Windows may have broken the sequencer software, possibly because both Windows and the vendor code was a kluge and prayer (as documented in detail by the CVE database over the subsequent years), and there may be medical regulatory compliance that any security update torpedoes. So when the tech comes out and sees Windows version $less_bad instead of the expected $most_terrible, maybe the support contract is now null and void and it'll be cost plus to fix. They did yell, a lot, upon seeing the SSH client software when they were out for some other reason; luckily we were able to convince them such an install was unlikely to break their software, or Windows. On the other hand, a SSH client install could diddle around in the Windows registry and break who knows what, so there are legitimate concerns. Anyways, one may not want to risk a breach of contract, or too much of a breach, so choices were made. The sequencers were not patched, were put behind a complicated firewall, and some risk was taken in installing a SSH client, and the lab techs did not have to sneakernet possibly infected drives hither and yon. Nobody was altogether satisfied, not the vendor (a SSH client was installed) nor the lab techs (no web browsing) nor the sysadmins (the whole system is a mess). And so on goes Lord Vishnu's theater, or game, or dance.

P.S. Modern sequencers appear to still have terrible security, and now also rely on cloud services. Jesus wept!

P.P.S. Firewalls and VLAN are very similar to pledge and unveil. Not an exact mapping, as pledge is more akin to something at maybe the application or service level, and filesystems differ from subnets, but pretty close: both set policies on what is, and is not, allowed.