Major incident: Hypervisor Outage | Status Page

[Investigating] **What We First Observed** We were initially alerted to the incident when our monitoring systems detected that several VMs lost network connectivity. Upon investigating, we found ransom messages being displayed at boot on all of the affected VMs. Our engineering teams immediately isolated the affected servers and began analysis. During the investigation, we confirmed that the boot sectors of impacted VM disks had been overwritten with the ransom message. We are attempting to recover the data by various means including examining raw block devices, reconstructing partition tables, and searching for intact filesystems. **How the Attack Was Executed** Meanwhile, the team investigating the breach discovered that a remote bash script (which is no longer accessible) had been executed across all affected nodes. Shell histories on those hosts had also been cleared. We performed a thorough review of authentication activity using system journals, rotated log files, login records and auditing data and found no evidence of unauthorized SSH access. All recorded user logins matched known internal accounts. At this point, we started looking into other infrastructure that could have facilitated this attack and discovered that logs of one of our Virtualizor instances had been cleared from around the time of the incident. This is the Virtualizor instance that all of the affected nodes are connected to. At this time, based on the available evidence, we believe that the attackers used the "Server Terminal" functionality within Virtualizor to gain shell access to connected nodes and execute the malicious script. This access method does not use SSH, which explains the lack of evidence relating to SSH connectivity, and we also discovered that this doesn't leave any login records on the nodes (all root level logins are also alerted via emails), explaining why we didn't find anything out of the ordinary earlier. **Scope of Impact** We use Virtualizor instances to support our VPS services. At this time, we have confirmed that only nodes connected to a single Virtualizor instance were impacted. Nodes attached to our other platforms were not affected. We also do not store personal or billing information of our users within virtualization platforms such as Virtualizor. Our investigation has found no evidence that customer databases or billing systems were accessed or compromised. We are currently working on the way forward, and all affected clients shall be emailed, and we apologize for the inconvenience this has caused to all our affected clients.