Mastodawn

Lemmy just had its first major hack. What happens next:

https://lemmy.world/post/1299831

Lemmy just had its first major hack. What happens next: - LemmyWorld

Hi all, If you’re just now signing in for the first time in 12+ hours, you may just now be finding out that Lemmy World and other instances where hijacked. The hijackers had the full abilities of hijacked user, mod, and admin accounts. At this time, I am only aware of instance defacing and URL redirections to have been done by the hijackers. If you were not forced to sign back in this morning, contact your instance admin to verify mitigations were completed on your instance. ## How? This occurred due to an XSS attack [https://owasp.org/www-community/attacks/xss/] in the recently added custom emojis. Instance admins should follow the issue tracker on the LemmyNet GitHub [https://github.com/LemmyNet], as well as the Matrix Chat [https://matrix.to/#/#lemmy-space:matrix.org]. Post-Incident Activity is still on-going. Currently, it is likely that just your session cookie was stolen, with instance admins being targeted specifically by checking for navAdmin, an HTML element only instance admins had. I do not believe this to affect users across instances, but I have yet to confirm this. ## What happens next? As I am not the developers or affected instance admins, I cannot make any guarantees. However, here is what you’ll likely see: 1. Post Incident investigation continues. This will include inspecting code, posts, websites, and more used by the hijackers. An official incident writeup may occur. You should expect the following from that report: - Exactly what happened, when. - The incident response that occurred from instance admins - Information that might have helped resolve the issue sooner - Any issues that prevented successful resolution - What should have been done differently by admins - What should be improved by developers - What can be used to identify the next attack - What tools are needed to identify that information 2. A CVE is created. This is an official alert of the issue, and notifies security experts (and enthusiasts), even those not using lemmy, about the issue. 3. A code security audit is done. This will likely just be casual reviews by technical lemmy users. However, I will be reaching out to the Mozilla Foundation and Cure53 as they recently did an audit of Mastodon. If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

Dreamer_joy Jul 10, 2023

Was lemmy.ml affected ?

Pekka Jul 10, 2023

Only Lemmy instances with custom emoticons were affected based on the Recap of the Lemmy XSS incident. So if Lemmy.ml doesn’t have these it should not have been affected.

Recap of the Lemmy XSS incident & steps for mitigation - LemmyWorld

# UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading. Original post follows: ---- This post is intended as a central place that admins can reference regarding the XSS incident from this morning. ### What happened? A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis. Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users. ### Am I vulnerable? If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe. ### I had custom emojis on my instance, what should I do? This should be enough to mitigate now: 1. Remove custom emoji DELETE FROM custom_emoji_keyword; DELETE FROM custom_emoji; 2. Rotate your JWT secret (invalidates all current login sessions) -- back up your secret first, just in case SELECT * FROM secret; -- generate a new secret UPDATE secret SET jwt_secret = gen_random_uuid(); 3. Restart Lemmy server If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus) ### Legal If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402 [https://lemmy.world/comment/1064402] ##### More context: https://github.com/LemmyNet/lemmy-ui/issues/1895 [https://github.com/LemmyNet/lemmy-ui/issues/1895] https://github.com/LemmyNet/lemmy-ui/pull/1897 [https://github.com/LemmyNet/lemmy-ui/pull/1897]

ghostwolf Jul 10, 2023

Well, I’m glad I didn’t bother to add them.

dudebro Jul 11, 2023

Less is more.

dan1101 Jul 10, 2023

I think generally the less flashy features (pun intended) the better. Text and links (and well sanitized) is all we need.

dudebro Jul 11, 2023

Yeah. Hopefully the devs will learn to wait before pushing new technologies that don’t really matter. (they won’t)

monerobull Jul 10, 2023

Good thing we don’t have custom emojis on monero.town and the admin account isn’t used for things outside of the local community :D

🔻-_AnoN_-🔻Jul 10, 2023

Good opsec

0x4E4F Jul 10, 2023

If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

It could get VERY, VERY expensive… depends on code complexity.

RCMaehl [Any]Jul 10, 2023

Yeah, specifically why I mentioned “affordable”.

static Jul 10, 2023

Not at this stage.
Lemmy grew too fast, got many more eyes.
Step 1 is getting a security focus group selected from the people who contribute code to lemmy.

0x4E4F Jul 10, 2023

Agreed, this is wise.

kakes Jul 10, 2023

Agreed. It might be hard to swing right now, but imo this is going to be a crucial step moving forward.

0x4E4F Jul 10, 2023

The examples below might actually do most of the work for free.

redcalcium Jul 10, 2023

This incident made me realize not to use an admin account for my primary lemmy account in my private instance. I setup another account for instance admin purpose (with 2FA enabled) and keep it logged out, then remove my primary account from the instance admin list.

Renacles Jul 10, 2023

This is a good mindset in general, when working in AWS you are not supposed to use your root account unless it’s absolutely necessary even if you are the only user. Hosting a Lemmy instance should be no different.

kakes Jul 10, 2023

Yeah, even in Windows or what-have-you, you should always keep your admin account separate from your daily driver account for exactly this reason.

Vilian Jul 10, 2023

most, if not every, linux distro work that way

Trainguyrom Jul 11, 2023

I just setup a VPS for a Minecraft server with friends and did exactly that. I was under a bit of a time crunch, but still took the time to think through those challenges in access for everything. Created an unprivileged user to run the server as, created a seperate unprivileged user for another service.

Crackhappy Jul 10, 2023

I work with 2 factor, Oath, SAML, etc. all the time for work, and for the life of me I can’t get it working properly with Lemmy.

redcalcium Jul 10, 2023

I use bitwarden, so it’s pretty simple: just copy the link from the 2fa button in Lemmy into the TOTP field in your account editor in bitwarden, and it’ll automatically recognize the format.

Crackhappy Jul 10, 2023

Yeah, I’m using Google Auth, and I’m getting codes just fine, but none of them work, regardless of how I try to use it.

redcalcium Jul 10, 2023

Google Authenticator doesn’t seems to support SHA256: github.com/google/…/11

Maybe try FreeOTP instead: freeotp.github.io

Plans to support SHA256? · Issue #11 · google/google-authenticator-libpam

From @ThomasHabets on October 10, 2014 8:7 Original issue 393 created by synikal on 2014-06-19T07:29:02.000Z: Hi all, Just wondering if there were plans to support SHA256? I'd really like to use ex...

GitHub

BarterClub Jul 11, 2023

Yup. Basics of running a server for anything. Never use your admin account and make a default backup with 2 factor.

flyaway1359 Jul 10, 2023

oh, that’s why I had to sign-in again 😮‍💨

Timception Jul 10, 2023

So if you dont have to resign-in?

Mantiz_Shrimp Jul 10, 2023

Timception you should contact the admin for your lemmy server. If you didn’t need to log back in, your server likely was not fixed. This means someone could hack it and steal the session cookies to impersonate server members.

curiosityLynx Jul 10, 2023

That pun flew over your head, didn't it?

Timception Jul 13, 2023

I was able to scroll through content, but had to relogin to upvote. However, today I was forced to re-login since I entered the Memmy app. So I guess, all good? Thx for the tips.

luis123456 Jul 11, 2023

No problems from me

ZuckerbergIsCancer Jul 10, 2023

Friendly remind that Lemmy.world is considering federating with Meta while Lemmy.ml will not.

I would strongly suggesting ditching Lemmy.world for Lemmy.ml before they can even get started with that stupidity.

goat Jul 10, 2023

Lemmy.ml is filled with Rankies and Stalinists who regularly remove any content critical of communist states

Tankies on lemmy.ml removing comments highlighting genocides from Communist states. - sh.itjust.works

frozen Jul 10, 2023

Or a smaller instance! lemmy.ml wasn’t meant to be the main Lemmy hub, and it would help the load on the larger instances to spread users out.

simple Jul 10, 2023

Dumb take. Lemmy.world devs simply said it’s not likely Threads will federate with Lemmy anytime soon anyways, and they’ll make a decision when there is actually a decision to be made.

ZuckerbergIsCancer Jul 10, 2023

“Dumb take” lol

The “dumb take” is defaulting to federate with Threads rather than NOT federating with Threads.

The only people defending this stupidity are .world users.

Their “wait and see” approach should be “don’t federate until we know it’s safe”.

They’ve taken the opposite (foolish) route.

simple Jul 10, 2023

I’ve made many comments saying why we should defederate with Threads but I still agree with the admins that there’s no point in doing a knee-jerk reaction on a threat that

A) won’t happen until at least a few months later

B) likely won’t federate with Lemmy anyways

C) isn’t actually a threat to Lemmy, but could pose a threat to microblogging websites like Mastodon

So yeah, waiting and seeing isn’t stupid.

ZuckerbergIsCancer Jul 10, 2023

“Wait and see” is fine.

Federating in the meantime is not.

Stop trying to dance around the heart of the issue.

simple Jul 10, 2023

Federating in the meantime is not.

Dude, Threads isn’t federating with anything right now. That’s the point, we’re not federated so there’s no reason to make a decision on something that won’t happen yet.

ZuckerbergIsCancer Jul 10, 2023

I’m not sure you understand the takeaway from the admin post last night.

ElectroNeutrino Jul 10, 2023

lemmy.world/post/1274909

Conclusion:

From the points discussed above, the possible lack of moderation alone justifies considering defederation from Threads. However, it remains to be seen how Meta will handle moderation on such a large scale. Additionally, the inability of individuals to block an instance means we have to do what is best for the community.

Where does it say in the admin post that they are default federating with Threads?

TheInsane42 Jul 10, 2023

They haven’t made a decision, as it’s no issue as there is no federation to threads and won’t be for the forseeable future.

Making a fuss about it now is as usefull as making a fuss over the sun dying in several million years.

NullDrive Jul 11, 2023

You’re missing the forest for the trees. You need to stop and listen to your fellow federation users.

TheSpookiestUser Jul 10, 2023

Correct me if I’m wrong, but I don’t think Threads even has the ability to federate yet?

RCMaehl [Any]Jul 10, 2023

While this is definitely a discussion to be had (I’ve created a few accounts on different instances). Posting that here is just adding onto the drama.

ZuckerbergIsCancer Jul 10, 2023

Not sure how you view it that way.

It’s just good advice for people who want to stay the hell away from a company like Meta that was willing to hand over private data to assist authorities in arresting a woman for exercising her reproductive rights.

Really not thrilled with the Lemmy.world users trying to downplay just how ridiculous the idea of federating with Meta is.

Is it possible to block .world entirely yet?

Baron Von J Jul 10, 2023

User-level instance blocking is not yet implemented. You could use an instance that is defederated from lemmy.world, such as beehaw.org.

Allow a user to block an instance · Issue #2397 · LemmyNet/lemmy

For front end issues, use lemmy-ui Is your proposal related to a problem? When looking at the "all" feed and even sorting by "top week" , it is basically all communist propaganda/advocacy ,obviousl...

GitHub

Mereo Jul 10, 2023

Look, Lemmy.world takes the rational way where it says there is no use to panic. Let’s be rational and analyze the situation rationally and go from there.

I support Lemmy.world. Let’s see how the situation develops and let’s make a rational decision with cool heads.

RxBrad Jul 10, 2023

“Ma’am… this is an Arby’s.”

Threads is not a Lemmy issue. Mastodon is where the concern lies.

NullDrive Jul 10, 2023

Not the time or place for this.

mrmanager Jul 10, 2023

Maybe ditching Lemmy.world makes sense but signing up on Lemmy.ml doesn’t. Pick smaller instances and spread out guys. That’s the entire point. :)

notenoughbutter Jul 10, 2023

here are some great instances github.com/maltfield/awesome-lemmy-instances

GitHub - maltfield/awesome-lemmy-instances: Comparison of different Lemmy Instances

Comparison of different Lemmy Instances. Contribute to maltfield/awesome-lemmy-instances development by creating an account on GitHub.

GitHub

CodeMonkeyDance Jul 10, 2023

I ran screaming from FB. Like, hair on fire running. That place is a roiling pot of piss. I would 100% not interact with ANY instance that has federated with them. They are a cancer that will infect this next big thing

dudebro Jul 11, 2023

I would strongly recommend making decisions for yourself.

CifrareVerba Jul 10, 2023

The way the hack was utilized is honest very creative and interesting; either way, if all big communities could crowdsource money for security audits, I believe that it could help prevent something like this in the future.

graphite Jul 11, 2023

The way the hack was utilized is honest very creative and interesting;

That’s often the case with exploits.

CifrareVerba Jul 11, 2023

Your not wrong, however, most of the time when these events happen it’s meant for something more insidious than lemon party.

graphite Jul 11, 2023

Sure. Not sure how that’s relevant though?

In general, finding an exploit requires looking for little tiny details that could exist in, really, any area of a given system; looking for a bug, and then exploiting that bug by understanding how input data can be used to create a deterministic chain of events.

This almost always requires thinking outside of the box.

There are people who are also paid to find these before malicious actors do.

It’s always going to be creative in some way, at least in the beginning.

nikt Jul 11, 2023

HTML injection / XSS vulnerabilities tend to be a sign of amateur hour to be honest. Made me a bit worried that I’m hitching my reddit escape wagon to the wrong technology. But sounds like this was due to instance-specific customizations rather than the core Lemmy tech, so hopefully we’re still on solid ground.

P03 Locke Jul 11, 2023

No, Bobby Tables is really not that creative at all. It is the most basic, entry-level exploit.

Exploits of a Mom

xkcd

Coreidan Jul 10, 2023

So much drama