emptywheelI've pulled together what we can learn from the October 22, 2020 CYA memo on the Hunter Biden laptop. Hoping some tech folks, esp @malwarejake and @matthew_d_green can review it to see if they can figure out why FBI had to install laptop hard drive in new laptop to get an image of it.
Jake Williams@emptywheel @matthew_d_green After a cursory review of the whistleblower transcript (item 20), there is no reason I can see why you'd do this. It's honestly a bit perplexing to me. But this drive seems to have been mishandled at every turn - at least this is consistent...
Alternate theory: it's inarticulate wording?
Note that he doesn't know how to spell Cellebrite.
But there was "computer guy" in the room.
@matthew_d_green
@malwarejake Also, is the observation that iMessages were encrypted (on the drive) but there was a business card with the password consistent with what we've seen elsewhere? @matthew_d_green
@emptywheel @matthew_d_green I don't remember that there were encrypted iMessages, but their investigation is very different (both in goal and scope) than mine. I'm sure there's data on that drive that I've never encountered, even across multiple analyses of the system.
Heck just on email, the reporters understand even just the email contents FAR better than me. I was the guy saying "yes, we can validate this email is authentic." The FBI can just subpoena Google. Whole different scope.
Random Nunes Parody@malwarejake @emptywheel @matthew_d_green It seems like the last thing you'd want to do is boot it. I'd use something like a gparted boot disk or Puppy Linux to boot from usb and image the whole drive that way.
i assume there are more professional equivalent tools.
@RandomNunesParody @emptywheel @matthew_d_green Definitely, though we may be seeing the telephone game effect at play here.
@malwarejake I think I'm pretty close to convinced there are material inconsistencies between thta laptop and what has been released. @RandomNunesParody @matthew_d_green
@emptywheel
The copies I've looked were all live booted from the drive image at different times. This is consistent with the FBI doc, only that I didn't expect the FBI to be doing that too. @RandomNunesParody @matthew_d_green
The copies I've looked were all live booted from the drive image at different times. This is consistent with the FBI doc, only that I didn't expect the FBI to be doing that too. @RandomNunesParody @matthew_d_green
Thomas A. Fine 
@emptywheel @malwarejake @RandomNunesParody @matthew_d_green
Every boot adds to the log files that forensic people should be looking at (instead of dick pics). Once booted, computers periodically "rotate" log files – meaning they move current logs to a different filename daily or weekly or monthly, keep N old versions, and purge anything older.
In short, booting will modify forensic data, and depending on timing can destroy forensic data.
@emptywheel @malwarejake @RandomNunesParody @matthew_d_green
So the logs would show recent boots. You'd be able to tell that it happened.
But in theory the FBI should have never done this.
In fact, it seems like a good lawyer could have all laptop data thrown out as evidence if the FBI actually did this, though you are probably more familiar with that part of things.
@emptywheel @malwarejake @matthew_d_green What they should have done is booted to a different hard drive either by physically removing the drive and plugging it into a different computer or booting to a usb drive, and then copy the whole physical drive in one big piece.
I can't believe the FBI doesn't know that.
DoctorRobofinger@RandomNunesParody @emptywheel @malwarejake @matthew_d_green
The charitable interpretation is that they're saying "boot" here instead of "mount," but this whole thing is such a mess that nothing would surprise me now
The charitable interpretation is that they're saying "boot" here instead of "mount," but this whole thing is such a mess that nothing would surprise me now
Bruce@emptywheel @malwarejake @RandomNunesParody @matthew_d_green
Nunes' Parody beat me to it. You absolutely would not want to boot the thing directly. Every time you boot, the OS scribbles something on the drives. (Especially if there's additional stuff [e.g. malware] on it.) So you really do want to put it on a separate system that treats it as read-only data. One shouldn't need to be a computer forensics expert to know this.
Compuguy 🐱@RandomNunesParody @malwarejake @emptywheel @matthew_d_green Agreed. The one class on computer forensics I took almost a decade ago taught me not do do those things and to maintain the integrity of a forensics image. As @wpoland says in a later post, they should of been using a write-blocker in order to make a clean forensics image.
The whole Hunter laptop thing is a great example of what not to do when forensically examining a computer!
@compuguy @malwarejake @emptywheel @matthew_d_green @wpoland I refuse to believe the IRL FBI just booted up a computer in evidence. It's either defense lawyer misinformation or just plain BS, probably the later.
@RandomNunesParody It doesn't come from defense attorneys. It comes from the IRS Agent himself, and was written 3 years ago. @compuguy @malwarejake @matthew_d_green @wpoland
@emptywheel @compuguy @malwarejake @matthew_d_green @wpoland Got it. Thanks for the reply. Good to see you here.
Byron@emptywheel forensic practice would be to pull the drive and connect it via a write blocking cable/device over usb to a forensic tech’s computer possibly with another usb drive to then image the drive with out altering its contents.
@emptywheel it should have them been indexed with a forsensic program like Nuix. Using the image to make a clone to then use in a new computer just is bad practice and does not make sense. As soon as computer boots files on drive could change.
Daniel Schwarz 
@wpoland @emptywheel This exactly. You want to be absolutely certain you’re not changing the data in any way at all while you recover an image.
Stefan Dembowski@wpoland @emptywheel THIS.
Forensics 101: work off a sector by sector copy. Unless it was encrypted, the drive format is well understood.
BlueDot🇺🇦This.
As opposed to what they always do on TV cop shows, which is just fire it up.
This.
You don't even need a write-blocking cable (although it's the safe way). You can just set a Mac to not mount disks at plug in, and then mount it by hand read-only. After that, the SSD sits sealed in a bag, and everybody works from locked read-only disk image copies (dmg or iso files) that are easy to share and faster than an SSD.
(You do also need an Apple proprietary enclosure for the non-standard Apple SSDs. Widely available.)
@thomasafine So why would they do it? What would the effect have been? @wpoland
@thomasafine ANd might that explain why they were resorting to validating files using a CSV download? @wpoland
@emptywheel @wpoland
I'm not going to harp on them about this because I don't know what tools they were using. Creating a CSV database of files for easier searches might be totally fine. And in fact, since with Apple metadata, no one single command lists all of it, a real forensic analysis would benefit from something like this (although there is zero indication here that this is what they actually meant).
I'm not going to harp on them about this because I don't know what tools they were using. Creating a CSV database of files for easier searches might be totally fine. And in fact, since with Apple metadata, no one single command lists all of it, a real forensic analysis would benefit from something like this (although there is zero indication here that this is what they actually meant).
For what they were talking about you could run:
find / -ctime -250 -ls
or just
ls -alctR /
And get a very quick answer. Sticking stuff in a database would be to set up for a deeper forensic analysis. (e.g. comparing filesystem metadata to the metadata Apple stores outside of the core filesystem structure).
John Panzer@thomasafine @emptywheel @wpoland I’m not looking up the command line flags but won’t that just give you the metadata for files and not a hash of the contents of each file, so if used for forensics de-duplication, won’t tell you if the contents are true duplicates?
@jpanzer @emptywheel @wpoland this was to answer the question of when files were changed, where they had proposed making a CSV listing to accomplish that.
@thomasafine Oh. Interesting point. The memo says that someone from CART imaged the hard drive.
It doesn't say CART was involved in the laptop image.
@wpoland
@emptywheel @wpoland
But that could just be sloppy language about "the hard drive". Once the one from the laptop was out of the laptop, people might have referred to it the same way.
It demands a different question to me: wouldn't a basic step to see what Mac Isaac had done be to simply compare the original ssd and the copy on the external drive and see if they differed?
@thomasafine And instead they deduped once they got the image of the laptop. @wpoland
@emptywheel @wpoland
I read that as people produced reports, listings of files, and these reports had redundancies in them. Unclear if these were files where there were duplicates on the laptop or filenames that were repeatedly reported. Would lean towards the latter.
I read that as people produced reports, listings of files, and these reports had redundancies in them. Unclear if these were files where there were duplicates on the laptop or filenames that were repeatedly reported. Would lean towards the latter.
@emptywheel @wpoland
OK I do see that they specifically said "imaged the external hard drive in Delaware". That does sound less like confusion about which drive - though it's still possible.
OK I do see that they specifically said "imaged the external hard drive in Delaware". That does sound less like confusion about which drive - though it's still possible.
@thomasafine With hard drive CART came in and imaged it--BUT NOT without problems. (See the March email.)
No mention of CART w/the laptop, and CART wouldn't do what was done.
@emptywheel @wpoland
It's unclear if that statement means they had problems imaging it, or if they had problems with what they found in the image.
The email might have said that Mac Isaac forensically ruined the data. (Which he probably did.) You might have a better guess at why they'd redact stuff from agents though.
@emptywheel @wpoland
Hmm. Shapley's version of the timeline says that "hard drive" is imaged by 12/19/2019 (#21,22) but on 3/6 the FBI received the image of the *laptop*. (#35,36)
So did the laptop take three months to image? Item #37 "First evidence ... from laptop" seems to corroborate that.
The message about "quality and completeness" says "the hard drive".
So maybe your interpretation of deduping was the right one?
Can't say I know the tech -- but then, I can spell Cellebrite properly and he can't -- but that is how it reads: immediate turnaround on the hard drive, four months for the laptop.
@emptywheel @thomasafine using cloned drive in another computer maybe so someone non technical could look around quickly 🤷🏼♂️ definitely not forensically sound. I work at a law firm and do email/data collections, and this isn’t something we would even consider.
@wpoland @emptywheel
It also seems pointlessly hard compared to just using a disk image file that could be stored on a thumb drive or transferred around your network.
It also seems pointlessly hard compared to just using a disk image file that could be stored on a thumb drive or transferred around your network.
I should probably note that most Macs have "target disk mode" in which the laptop can be booted as if it is nothing but a hard drive (although if you're pulling the disk out and putting it in something new, an actual enclosure is vastly cheaper and a better choice than using a laptop as an enclosure).
It's _possible_ they meant this, but it still sounds like a stupid method.
@emptywheel @wpoland
So for example if they just logged in and poked around, then ever "poking around" action would be updating access times.
If they used Apple Mail to read email (as Mac Isaac did), this alters the email files (because Apple Mail stores metadata at the end of email message files, like last read time).
@emptywheel @thomasafine @wpoland
> So why would they do it?
Conspiratorial: so they can insert stuff
More likely: incompetence
> What would the effect have been?
Probably nothing important, unless they wanted to twiddle the data. However, following proper procedures would mean being able to prove the data were not twiddled. That would have been a good thing.
Alan Miller 
🇺🇦@thomasafine @wpoland @emptywheel the big thing a write blocking cable preserves is admissibility. It's basically as important as chain of custody which in this case is already a bad joke.
"[Some guy who might or might not have been Hunter Biden dropped off this laptop for repair then never came back to pick it up and pay for the work. Once it was legally deemed abandoned and became my property I started it up and started snooping through it, then called some people who hate Hunter Biden's father and let them have unfettered and unrecorded access to it as well. There's some very incriminating stuff on there!]"
Steve Hersey@emptywheel @malwarejake @matthew_d_green
Imaging a hard drive generally requires that it NOT be the one running the operating system. There are several reasons for this, which all boil down to "you can't get a trustworthy, consistent snapshot of the disk hosting the operating system you are running at the time:"
- A drive hosting an active operating system will have numerous files open (applications, logs, etc), preventing the imaging software from reading them, and may also play games with the partition table, for either good or bad reasons;
- Any malware on the drive being imaged may well interfere with the imaging process, potentially hiding its presence if that's the drive running the operating system; and
- the disk imaging tools may be running on a different operating system from the one that is present on the target hard drive.
@emptywheel @malwarejake @matthew_d_green
"FBI determined in order to do a full forensic review a replacement laptop had to be purchased so the hard drive could be installed, booted and imaged."
The instant you boot from that SSD, you've altered forensic data. Log files are added to at least, and if rotated, data may be lost before you've done a thing.
This is BASIC. What they did is clown school forensics. Did they do this themselves without cyber?
@thomasafine @emptywheel @matthew_d_green I'm trying to give this the most generous reading possible and think it's a description of a non-technical individual reciting what they saw the techies do.
@malwarejake You say my observation that CART was definitely involved with the hard drive, but that's not recorded for the laptop? @thomasafine @matthew_d_green
@malwarejake @emptywheel @matthew_d_green
I just can't find a reading generous enough to make sense out of things.
If they were working from a disk image copy and not the original, there'd be no confusion about people needing to "buy a laptop to put the hard drive in" (from 43.d).
But 20. also is quite specific. "FBI determined in order to do a full forensic review a replacement laptop had to be purchased so the hard drive could be installed, booted and imaged".
Chris Bussard@emptywheel @malwarejake @matthew_d_green
Well, the short answer is likely that the original laptop was broken.
But moving the HD to another laptop and then *booting* it is absolutely bonkers. When you boot from a drive, you're going to change its contents. New logs will be written. Old logs might be archived or deleted. Temp files might be deleted. Checks for automatic updates may run, write logs, and download updates in the background. Programs configured to run at startup may check for automatic updates, synchronize to clouds, download e-mails, etc. And these new writes may happen in sectors containing deleted-but-theretofore-recoverable files, making them unrecoverable.
The correct approach is to mount it as a read-only secondary drive (likely in a desktop with ample SATA ports) and use dd to do a sector-for-sector copy of the drive to a file. Then set the original HD aside in case you later discover the copy has flaws. From here on out you should be working on copies of copies.
[Continues]
@emptywheel @malwarejake @matthew_d_green
[Continued]
Is it possible that the memo author took it for granted that "everyone knows" the above and skipped over it? Mounting a copy of a copy of the original hard drive in a new laptop and then booting *that* would probably be a convenient way to browse the user's files.
The only other possibility I can fathom is that the FBI was profoundly incompetent here.
@cwbussard The memo author doesn't know how to spell Cellebrite and calls the one computer expert in the room "computer guy."
And the memo notes real difficulties for actually browsing files--plus there were concerns about the Cellebrite report access.
@emptywheel @malwarejake @matthew_d_green
Hmmmm.... Is it possible that the memo author couldn't understand what "computer guy" was saying, and so the memo reflects its author's wildly inaccurate understanding of what "computer guy" did?
(Playing devil's advocate here. The "profoundly incompetent" hypothesis seems more likely.)
@cwbussard Absolutely! That's why I keep harping about the way he spells Cellebrite.
But that's why AUSA Lesley Wolf's apparent shared understanding of the process is important. She knows more details of the process and probably knows how to spell Cellebrite.
@emptywheel @malwarejake @matthew_d_green I’m not a forensic expert but that sounds absolutely nuts to me. Not how you preserve chain of custody since that is guaranteed to change the laptop’s data.
Also: “The investigators didn’t get content from the laptop until April, and it was deduped from the hard drive (though there seems to have been stuff on the laptop that was not on the hard drive).” Huh. De-duped how? Was the hard drive image partial? Lots of questions.