If I answered some questions about Industrial Control System cybersecurity and cyberattacks during my downtime today, what would you like to know?

Some background - my expertise is in incident response and digital forensic investigation of hacking of infrastructure systems - like power, water, manufacturing, oil and gas, transportation, agriculture, etc. There aren’t a lot of people who specialize in this. My company provides consulting and products to do cybersecurity for weird stuff that powers critical infrastructure like PLCs and SCADA. These networks are quite different than enterprise IT, and doing security in them can be challenging.

My own background is pretty left field as expected, with degrees in Networks, Electronics, and Avionics. I’ve been doing this for a while now. What would you like to know?

#cybersecurity #DFIR #ics #CriticalInfrastructure #ICSCybersecurity #IndustrialControl

@hacks4pancakes from a career standpoint i am curious what comes after a) DFIR and b) the ics security space.
@spy604 do you mean what jobs you might move to after you do those as an individual contributor?
@hacks4pancakes yes, what type of jobs one may move to after doing dfir or other industries that ics skills have some synergy with.
@spy604 You can keep doing DFIR or ICS DFIR your entire career as an IC. That said, you could branch out into a management track, or specialize even further in some form of research!
@hacks4pancakes do you feel ics security skills translate well to other niches?

@spy604 @hacks4pancakes i did several years of IIoT/OT Security before i moved into #cloudsecurity

what i found remarkable is that - be it for very different reasons - there were a lot of overlaps: with the biggest problems unintentional network connections and known vulnerabilities. Microsegmentation works just as well in the cloud as it does in ICS/OT!

also, real-time networks are awesome

@hacks4pancakes @jaythvv i have seen a convergence a bit with connecting ics to cloud under the ‘digitalization’ banner.

Best of segmentation from both sides. Lol /me ducks

Curious to know if you folks have seen that.

P

@spy604 @hacks4pancakes yes, i have. i reckon it's fine as long as 1) data to the cloud follows different network paths for different device classes, and 2) any traffic to cyberphysical systems is intermediated in some way (that is, send a message to one system to translate it into another protocol before sending on to destination)

@spy604 Somewhat. I think we can learn a lot from how operators and engineers measure risk and consequences to the real world - they're a lot better at it than we generally are in enterprise. We also have seen some problems there first that are now becoming highly relevant in IoT and smart devices.

|That said, I do see a deep divergence in skills - particularly in DFIR skills. Most of the recent grads in cybersecurity I interview are missing the legacy skills that we rely upon 100%. No EDR or agent-based security in industrial networks.

@hacks4pancakes thank you. Awesome to know the value of the legacy skills. That to me applies some job security.

I found ics a frustrating space to work in personally. Mostly due to the old school way of thinking.

Operators like to see degrees, in person engagement, and lack of diversity in the workforce.

I’m curious to know that have you as a diverse person seen diversity effect ics. Positive trend?

@spy604 Just like the financial industry, we desperately need people who can do legacy forensics.
@hacks4pancakes what are the two biggest differences between forensics in these networks and ‘traditional’ enterprise networks

@SecureOwl Yeah, so we have to be a lot more gentle with industrial devices and process networks because the real life consequences of altering or modifying them can be life, safety, and damage. We don't want to cause worse consequences than the adversaries doing DFIR. So that means we have to be way more light touch and passive in our investigations - and often times we cannot even install things on machines without risking them or voiding a warranty.

The second thing that''s wildly different is the technologies. There's no XDR/EDR or agent-based forensics in most of these networks. It would cause those problems that I talked about. There's a ton of legacy devices, sometimes back to Windows 3.1 that are still doing critical functions. So a lot of gentle, legacy forensics, manually and by hand, system by system. Very different from most modern IR.

Finally, as @MisuseCase points out, we also have to be able to do forensics on niche documented/undocumented legacy network protocols and their process impact, as well as low-level devices (not usually level 0) that don't run Linux or Windows, like PLCs and RTUs.

@hacks4pancakes How do you secure systems running Windows 3.1 (no security updates for.. 30 years?) *and* without being able to install anything (to avoid voiding the warranty)? I wouldn’t even know where to begin with those restrictions.
@mkfnch you honestly get very creative with architecture, detection, and traffic control.
@hacks4pancakes I’m so glad there are smart people like you focused on these insanely-critical edge cases. 😆
@hacks4pancakes @mkfnch +1 on that.
Architecture is key. It's actually the modern systems that scare me the most. K8s/Docker deployed in various critical infrastructure projects by service teams with no experience and tight deadlines. What can go wrong, right? :)
@mkfnch @hacks4pancakes There is one advantage over most IT: The ICS stuff is single purpose so you don't have anyone trying to do general computer things on those old boxes. No browsers, office suites, email, etc.
@mkfnch @AMS I need giphy to put the “how I wish they were true” Star Wars gif here. I’ve found Fitbit scales on ICS networks, and plenty of porn.
@hacks4pancakes @mkfnch  This is why I like living down in drives land. (Though they want to bring "edge-computing" there too).
@mkfnch @hacks4pancakes In some industries installing anything would invalidate the system. You'd then need to stop production and pay Siemens et al a bucket load to come in and revalidate it before you can start production again. You can't install OS Service packs, OS security updates, anything, if the vendor hasn't tested and approved them. This is where @ErrataRob humorous threads about not patching stem from. There are things you can do as long as you document them or get it validated but that's expensive and slow. It's a PITA environment to work in for everyone.
@dan @mkfnch @ErrataRob fun for me, but I do totally understand how frustrating it can be

@hacks4pancakes @SecureOwl @MisuseCase

Sounds like archeology of wristwatches but mustn't stop them from ticking 😬 don't cut that red wire

@MisuseCase @hacks4pancakes @SecureOwl I see you’re done for now but some time if you’re doing this again I’m curious about what sorts of old network protocols you run into.
@SecureOwl @MisuseCase @rmd1023 pretty much every one, all of them back to 1970 (serial protocols encapsulated)
@hacks4pancakes @SecureOwl @MisuseCase
And let's add another layer:
Certain industries (*cough* FDA-regulated ones for example) risk having to shut down production for revalidation (including functional testing) if you are not careful...
@hacks4pancakes ok so why the fuck are these not on an isolated subnet? Just standard business opposition to downtime risk? Are they and the attackers are already in the network?

@Jetengineweasel@hackers.townVastly increased process efficiency and cost-savings. The vast, vast majority of "secure" modern industrial networks vaguely follow the Purdue Model, meaning there's some kind of DMZ between the Enterprise network and industrial process network. It can vary in crunchiness and number of holes. So there is some segmentation, but there's a lot of necessary stuff going between networks and subnets, such as telemetry, distributed control, system updates, file transfers, and remote access. This is hugely important to business in modern process environments. I see maybe 1 or 2 airgapped networks a year doing this full time.

Are attackers already in industrial networks? Absolutely! It's hard to conduct meaningful attacks against a process (not just a device, which has redundancies and safety controls). So lots of resourced adversaries are building footholds and learning.

@hacks4pancakes I'm surprised it's that high.

Though my definition of "airgapped" has been warped by DoE people who are reassuringly extreme in their interpretation of the term.

@Name_Too_Long I mean, my interpretation is pretty loose, and I do specifically only this full time.

@hacks4pancakes Theirs is basically "if anything on the network can read removable media, the network is not airgapped." And now I'm quoting (because it burned into my brain) "its' connection to the outside world is just very high latency."

I'm not sure if I'd go *that* far, but it's nice to know the people responsible for the network running the nuclear reactor <100 miles from where I sleep do (and that it meets their definition).

@Name_Too_Long 😂​that's the greatest quote I've heard yet from those folks. And I work adjacent to DOE/DOD nukes all the time.
Just as good as "never underestimate the bandwidth of a box truck"
@Jetengineweasel @hacks4pancakes from personal experience: because folks want data/interface/control and didn’t bother to put appropriate and secure middleware in place.
@hacks4pancakes how many networks do you see that are properly segmented? Do they leverage flow logs for forensics? Do they have functional asset inventory? Just curious. 🙂

@c0nsid3rate How many networks do I see that are properly segmented? Quite a few in better resourced orgs... it's just that there is unfortunately almost always some pass-through via the DMZ as well as vendor / out of band connections that break the Purdue Model. So you have a firm crunchy candy outside with a very soft and squishy candy center.

Do we leverage flow logs for forensics - sometimes, but they are not very useful when we're talking attacks that primarily occur internal > internal, and potentially leveraging specific industrial protocols. Actual network threat detection and packet capture is really important in industrial, and it has to be done differently than in IT.

Who has an asset inventory... I don't have firm metrics. Lots of orgs have one in OT but it has not been updated in 5, 10, 20 years and there have been system updates and additions since then. It's a big problem everywhere.

@hacks4pancakes thanks for your full reply! Flow data vs. actual packet capture is a nice distinction to call out. Thanks for your thoughts on that. Interesting about asset inventory and segmentation. I guess that's what I would expect.

@hacks4pancakes Have you ever come across a ModBus network in the field that doesn't make you want to tear your hair out?

At least with DNP and 61850 you can *try* to do something reasonable. :sigh:

@ve7fim They don't make me want to tear my hair out, honestly. The purpose of modbus is to function very simply and effectively with no complexity. It's not great that we are encapsulating something that was meant to be serial and shooting it over modern networks, but when you hit the big red button, the big red button goes. Can't have delays or overhead in that.
@hacks4pancakes FYI might be worthwhile to note that the National Institute of Standards and Technology refers to these types of systems, along with IoT systems/devices, as “Operational Technology” (OT), in contrast to solely “Information Technology” (IT). OT includes sensors that read conditions in the physical world and/or actuators that do things in the physical world.
@MisuseCase I try to avoid using “OT” when I’m talking to anyone who isn’t specifically in the space because it’s definitely not a well known term. It doesn’t mean a lot to most plant staff and it’s rarely known to broader IT
@hacks4pancakes That makes sense. Also people might look at you funny if you said the systems in oil refineries aren’t that different from their Nest thermostat. (Are they? Well, it’s complicated…)
@hacks4pancakes as someone who doesn't do DFIR, but might help build systems. Where do you find forensics most valuable? Low level control, HMIs, ERP/MES? How much is usually helpful? Full disk imaging, basic collection or something else?

@blackfell Documentation, honestly. We have to reverse engineer a ton of low level devices just to get logs because the physical interfaces and the memory architecture are not documented anywhere and we just need some simple diags and logging. HMIs and Engineering Workstations typically run Windows so other than figuring out where stuff is stored in the application, forensics isn't in outer space.

We tend to triage & memory collect on Windows and Linux systems, but due to the lack of agents sometimes we have to grab full disk images. We have to be very flexible due to process uptime, legacy systems, and lack of agents.

@hacks4pancakes ah shit, anything but docs 😂 is that config docs, or vendor docs mostly? Sounds like a bit of the latter maybe?

What's the typical scale like? Single site, multi site, bigger?

@blackfell every doc, all of the docs. You would be amazed what I have to SE out of places. I do IR from everything from two person municipalities to f10s.
@hacks4pancakes excellent! cheers 🤩 do more docs, know your shit, got it.

@hacks4pancakes why are these systems externally connected? With the critical nature of the systems, why not keep them isolated and manage offline?

I’m guessing that’s just not possible but I don’t know why :)

@Adman @hacks4pancakes how do you get stuff like production numbers from the plant to the ERP system so someone knows how much feedstock to order if you don’t have a connection between the systems? How do you link batch numbers of products to the conditions at the time of manufacture without a link? How to you process simulations of production in a digital twin and have the twin feed the changes needed to optimise production without a connection?

@Adman @Wil had some really good answers here. Networking systems has enabled vastly more efficient processes across facilities and enterprises, and incredible cost-savings. Staff can be centralized in a single facility, or work from home. It's no longer necessarily required to dispatch repair techs to remote sites. Process data can be shared across multiple facilities to ensure synchronization. Telemetry can be used to identify failures and inefficiencies proactively.

And, it's all a lot cheaper to do this using existing technologies like TCP/IP and vendors like Microsoft, Juniper, and Cisco.

@hacks4pancakes @Wil all makes sense, was just thinking about the cybersecurity challenges for critical infra and wondering how the risks stacked against the benefits :)
@Adman @Wil it’s pennies to dollars in most cases, surprisingly.
@hacks4pancakes how would you suggest IT SOC analysts skill up for handling OT needs, including sharing a box of doughnuts or whatever with the people that run the OT stuff?
@whereisthespai I talked a little about open and healthy communication being key (yes, bribery with food is a great tactic). You need to listen to the operators and engineers because they know more about the process than you ever will, and lives and safety are on the line. You also need to start truly understanding the process at a high level and consequences of concern, as well as operational life cycles.
@hacks4pancakes Is there a lot being done to shore up security against attacks to key infrastructure in the U.S., EU, other places? What should governments be doing that they aren’t? (Or organizations if that’s more your area?) Hope questions aren’t frighteningly off base.