While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.
That’s something a hacker would say.
🤣
Very impressed by how quickly action has been taken by this and other instances to patch the issue.
Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.
I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD
So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear
What I did was add my account (again)
There was no warning or anything, and it populated the list with two of me.
At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?
I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.
I logged back in, and now everything is back to normal
I found I didn’t actually have to log out, just go into account settings and reconfirm everything without changing it
Very, seems like great work.
uh, why did you have negative one dislike?
Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.
huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances)
Nope, just like Reddit it’s a value that ranges between negatives and positives. If I get two thousand upvotes, positive 2k. If I get two thousand downvotes, negative 1999 (because iirc you start with one by default).
Not exactly sure I understood what you meant by “either positive or zero”.
see your comment rn, it has 1 upvote (from yourself by default) and 0 dislike (so it’s not shown)
but in the screenshot I sent above you got 287 upvote and minus -1 downvote (making your total 288) which is mathematically correct but seems like an unintended behavior
for example this comment of mine normally have 9 upvote and 2 downvote (which is shown as a positive integer 2, not negative), making my total upvote 7
Just occurred to me that the app I use also shows separate counters. I fooled myself into thinking it was a single counter.
That’s interesting. Remember it’s a very new platform, minor bugs aren’t out of the ordinary.
I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.
Deleting hospital fees/debt is very dangerous… In many HUGE regions in the US there’s only one hospital and if that hospital suddenly can’t pay its bills it could shut down, leaving a whole lot of completely innocent people in a very sad, people-are-dying sort of state.
In fact, something like this already happened:
https://www.cbsnews.com/chicago/news/st-maragrets-health-central-illinois-hospital-closing/
Hospitals are special in that they’re often evil organizations (not all though) that are some of the easiest to hack but also provide critical services to the most vulnerable. One should tread lightly. Political solutions are better (hack some politicians that are against healthcare reform instead).
Clearing credit card debt via hacking is nearly impossible but I agree it would be a much more ethical choice for hackers to target. I used to work for the credit card industry. My unique insider perspective, deep industry knowledge, and personal experience is here to let you know they suck. They are just as evil and unethical and unnecessary as everyone thinks they are! Seriously: If Visa, MasterCard, American Express, and all the lesser players suddenly disappeared the world would be a better place.
Before that can happen though people need a backup payment method that doesn’t go through their systems and no: Cash won’t work (there’s not enough in circulation and it’s dangerous to carry large amounts of it). The credit card companies know this threat exists which is why they lobbied Florida (and probably other states) to outlaw alternative, government-run forms of payment (e.g. central bank currency).
As soon as people have a widely accepted payment option that doesn’t go through Visa and MasterCard’s middlemen (e.g. First Data) then hackers can take their gloves off! Until then though… Let’s keep the payment infrastructure working, OK? Thanks!
There’s no limit to the amount of good deeds hackers can do though. So let’s encourage that! For example, there’s plenty of cartels and evil religious organizations (e.g. Taliban, ISIS, Mormon Church, Prosperity Gospel scam artists) that have plenty of money to spare and enormous attack surfaces 👍
Hospitals are special in that they’re often evil organizations
Just want to state the obvious and say, this is pretty much only the case in the US.
I think the alternative payment systems in the developing countries are actually good. UPI in India is very utilitarian. China also has the wechat thing. I guess the issue with these are that they are not universal and limited to a single country.
Ribbit
clearing credit card debt, deleting hospital fees, or something else that actually serves the public good,
Inflation does very clearly not serve the public good. That aside, causing havoc in banks and medical institutions would have other unpleasant effects.
Having a dedicated sub for bad understanding of economics seems stupid, it’s already spread over all subs, it’s normal.
Of course, the extremes of bad economics would be usually found someplace with “soc” in the name.
I’m saying your statement is bad economics. Debts get discharged all the time and they have no impact on inflation. It’s called the Bankruptcy System and it’s been a part of American economic reality since the mid-1800s.
Debts get discharged all the time
Yes, so?
and they have no impact on inflation
Measured by whom? Logically they should.
It’s called the Bankruptcy System and it’s been a part of American economic reality since the mid-1800s.
So in your idea of good economics it doesn’t matter for inflation if debt of NxM total gets discharged per month or of NxK total per month where K is much bigger than M?
I just don’t get that pretentious acting.
Logically they should.
No, they shouldn’t. The money supply is unaffected by discharges.
So in your idea of good economics it doesn’t matter for inflation if debt of NxM total gets discharged per month or of NxK total per month where K is much bigger than M?
Discharge does introduce short-term shocks but it’s not the doomsday scenario you’re painting it to be. We did it in the 1800s and it was mostly fine compared to the regular bank panics before the greenback was adopted.
The money supply is unaffected by discharges.
Ah, OK. Maybe “inflation” is the wrong word, but there’s a response. Insurance becomes more expensive, loans become more expensive, basically everybody for whom such an event is a risk reacts to its probability growing.
but it’s not the doomsday scenario you’re painting it to be
Well, I’m not saying it’s literally a doomsday scenario, just that it likely wouldn’t benefit the person dreaming about it more than it would harm them.
How about cleaning the bottom 10%'s debt, with the earings from one week of the top 0.1%?
I already know I’m gonna be downvoted for this, but the top 1%/0.1% spending isn’t gonna change, whereas the bottom 10% will cause inflation… That’s why there’s no magic bullet.
The bottom 10% don’t have enough money to “cause” inflation, not even the bottom 90% have that much money. Inflation is driven by the top 5-10%, representing 70% of the wealth; the rest just get taken for the ride.
You’re right the top 1%'s spending won’t change, it’s already 1000x above a person’s basic needs, so what’s the difference between 1000x and 900x (10% inflation).
Exactly, the bottom 10% don’t have enough money, meaning that any money you give them will go towards consumption. The top bracket’s spending as % of income or wealth is tiny and is mostly independent of their income. Their money is spent on investments, not basic goods and services. They practically don’t affect inflation.
I think money should be printed during periods of low inflation. E.g. Japan could have benefited from that. After this bout is over, governments can return to printing, carefully.
Ah, you mean unauthorized “redistribution”, not unauthorized “vanishing debt”.
Technically should do less harm in terms of inflation, but money lying around is different from money being used, so there’ll still be an increase in inflation.
The part about causing havoc - kinda same, there may not be direct inconsistencies as in the initial variant, but there’ll still be some confusion due to the “top 0.1%” possibly being petty and trying to get their money back.
I frankly prefer changing the rules so that there’d be fewer artificial barriers for competition and economic efficiency to this. Say, patent law and trademark laws and IP laws have basically outgrown their usefulness and are now just a plague. Same with various licenses and practices for medical/pharmaceutical stuff (I know that things should be tested and an average person can’t tell a hoax from a normal thing, just entities doing certification shouldn’t be able to block stuff which would then be used to create oligopolies). Same with telecom. And so on.
Except for air traffic, water traffic, road traffic and radio, of course. Not regulating those would mean, eh, real havoc.
you mean unauthorized “redistribution”
Fine, let’s do taxes: how about cleaning the bottom 90%'s debt, with the income from 4 months of the top 0.1%.
…and that’s just 30% income tax, it used to be 90% for the rich right after WWII: History of taxation in the United States
It’s not that simple, there’s a response of the “top 0.1%” moving their property elsewhere or distributing it by various legal means so that they’d have to pay less.
In dumb terms, you have to design a system where 4 people collectively owning 4bln$ would pay the same as 1 person owning 4bln$. Not even mentioning that they can have N friends abroad.
Also there are still “rich” people in Scandinavian countries, who may not directly own nearly as much as Bill Gates, but still have enormous power.
Also this will, in fact, affect inflation.
My point is - money represents power, which is convertible into other means, you can tax money or property, but you can’t universally tax power.
Money-wise (as a universal equivalent in a non-coercive system) you can at least somewhat clearly evaluate that power. If you scare powerful people off to convert their power into more obscure media, you won’t have that clarity.
So I don’t see this as a problem one can solve, but I see other problems more accessible, like patent\IP\trademark\certification laws.
I have no idea why you got down voted, but you’re right
Well, I honestly in a way write such comments in a tone more likely to irritate people. Maybe not consciously. I just happen to have grown in a family and in groups where disagreeing and arguing was not considered disrespectful, and I am ironically not very tolerant to the other way of looking at this.
(Should think about this more often when I want to complain about life - some people were not that lucky.)
Seems that a chunk of this platform’s userbase are people generally angry at the establishment who upvote everything that wants to bring it down and downvote everything that rejects the idea. Happy that there’s many reasonable people here too :)
I’m generally angry at the establishment too, it’s just that I see that establishment being pretty friendly to leftist ideas on economics in everything but direct admission of it.
You’re ruining the circlejerk with your realism! 😠
Edit: I think Mr. Robot gave a good glimpse what would happen if all debts were wiped. It sounds fun on paper, but in the end, the people with the least money would suffer the most.
I personally just lose any interest in conversation when I realize that my counterpart doesn’t want a working system or a better world or really some justice, they simply want to rob someone who has more than they do. No deeper purpose or something, just plain envy.
It’s like certain moments in sex. So bloody frustrating.
And, of course, the only leftists I’ve encountered who wouldn’t be what I describe were book characters. Yeah, nice characters, fascinating, really making me wish something like this was possible, but even with the depth limitations for describing an entire person on paper they were still deeper that RL leftists, FFS!!!
I have at least met living sincere good-willing ancaps and living sincere good-willing fascists (sic) even. The only people I know in person I could possibly call a real sincere good-willing leftist would be my sister, and maybe one of my cousins, and one DM (though from a few conversations I suspect he just has, eh, a leftist background, but is more literate in economics than such people usually are).
Please don’t confuse wealth and money.
What about earnings and income?
Top 0.1% don’t have earnings and income.
“Marty we started this journey together!”
“It was a prank, Cos’!”
So what happened:
- Someone posted a post.
- The post contained some instruction to display custom emoji.
- So far so good.
- There is a bug in JavaScript (TypeScript) that runs on client’s machine (arbitrary code execution?).
- The attacker leveraged the bug to grab victim’s JWT (cookie) when the victim visited the page with that post.
- The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.
Am I right?
I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
- User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
- JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
- How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
- The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.
Am I right? Correct me if I’m wrong.
Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.
Marek.
I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
I’m a modern web developer who used to be an old-school one.
User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
Yeah - pretty much, though there are some mitigating factors.
Strictly speaking, it was the alt text for the emoji. Alt text is HTML, and rather than allow arbitrary HTML they allowed another language called Markdown. Markdown is “a plain text” language with human readable syntax specifically designed to be converted into HTML.
Markdown is the right format to use for emoji alt texts, but you do need to be careful of one thing - the original purpose of Markdown was to allow HTML content to be easier to write/read and it is a superset of the HTML language. So arbitrary HTML is valid markdown.
Virtually all modern Markdown parsers disable arbitrary HTML by default, but it’s a behaviour which can be changed and that leaves potential for mistakes like this one here. Specifically the way Lemmy injected emojis with alt text into the Markdown content allowed arbitrary HTML.
This wasn’t an obvious mistake - the issue over on Lemmy’s issue tracker is titled “Possible XSS Attack” because they knew there was an XSS Attack somewhere and they weren’t immediately sure if they had found it in the emoji system. Even now reading the diff to fix the vulnerability, it still isn’t obvious to me what they did wrong.
It’s fairly complex code and complexity is the enemy of security… but sometimes you have to do complex things. Back in the “old-school” days, nobody would have even attempted to write something as complicated as a federated social network…
JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS.
Yeah - the Lemmy developers made a mistake there. There are a few things they aren’t doing right around cookies and JWT tokens.
Hopefully they fix it. I expect they will… It was already actively being discussed before this incident, and those discussions have been seen by a lot more people now.
How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
There are several levels of isolation that could have blocked this:
- Users should not be able to inject arbitrary HTML.
- A flag on the page should be set telling the browser to ignore JavaScript in the body of the page - this is a relatively new feature in the web and disabled by default for obvious backwards compatibility reasons, but it should be set especially on a high value target like Lemmy, and I expect once it’s been around a little longer browsers will enable it by default.
- A flag should have been set to block JavaScript from contacting an unknown third party domain. Again, this isolation is a relatively new web feature and currently disabled by default.
- As you say, JavaScript shouldn’t be able to access the JWT token or the cookie. That’s not a new feature in the web, it’s just one Lemmy developers didn’t take advantage of (I don’t know why)
- Even if all of those previous levels of isolation failed… there are things Lemmy should be doing to mitigate the attack. In particular instance admins have had to manually reset JWT tokens. Those tokens should have expired somehow on their own - possibly the moment the attacker tried to use them.
The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.
Yep - the modern best practice is for admins to manage the site via a completely different system. That adds considerable complexity and cost though, so it’s rarely done unfortunately. But you know, Lemmy is open source… so if someone wants to take on that work they can do it.
I’ll add one more - it should have taken less time to close the exploit… but given this is the first serious exploit I’ll forgive that.
Ultimately several of failures contributed to this attack. I expect many of those failures will be corrected in the coming weeks, and that will make Lemmy far more secure than it is right now - so that next time there’s a bug like the one in the Markdown parser it isn’t able to cause so much disruption.
The good news is no harm was done, and a lot of people are going to learn some valuable lessons as a result of this incident. Ultimately the outcome is a positive one in my opinion.
Awesome write up for someone who doesn’t have much experience at all in coding to understand. Thanks, it was a good read
JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
JavaScript needs access to the cookies, they are the data storage for a given site.
To protect them, the browser silos them to the individual site that created them, that’s why developers haven’t been able to easily load cross domain content for years, to mitigate XSS attacks.
The security relies on the premise that the only valid source of script is the originating domain.
The flaw here was allowing clients to add arbitrary script that was displayed to others.
You’re dead right that only the way to fix this is to do away with JavaScript access to certain things, but it will require a complete refactor of how cookies work.
I haven’t done any web dev in a few years, this might even be a solved problem by now and we are just seeing an old school implementation. 🤷this might even be a solved problem by now
Yes, it is called
HttpOnlyand is decided by the server who is sending the cookie to you in HTTP response header. I believe there are also HTTPS-only cookies that when received via HTTPS, cannot be used from HTTP, but I cannot find it right now.
“never happen in a desktop/server application” Just don’t run mystery floppys and CDs right :)?
Oh I forgot another line of defense / basic security mitigation. If a server produces an access token (such as JWT or any other old school cookie / session ID), pair it with an IP address. So in case of cookie theft, the attacker cannot use this cookie from his computer (IP address). If the IP changes (mobile / WiFi / ADSL / whatever), the legitimate user should log-in again, now storing two auth cookies. In case of another IP change, no problemo, one of the stored cookies will work. Of course limit validity of the cookie in time (lets, say, keep it valid only for a day or for a week or so).
To be fair, it’s not an issue with “the web.” A bug in JavaScript is not a problem for the web. It’s a problem for people who use JavaScript.
Thank the heavens the meme community stayed safe through this without my daily dose of cybersecurity memes idk how I would function ;)
Ugh, people should not go after systems trying to give a free service to the internet. It just ruins everything.
On the other hand, the “attack” seemed to just be mostly harmless griefing and it’s good for these sorts of vulnerabilities to be found early.
Yes, but responsible disclosure would’ve been even better.
That doesn’t surprise me. Especially the “homemade” instances. The documentation is severely lacking and I had to fix lots of stuff in the instructions with try&despair to make my instance run.
There’s not a great focus in security if your application starts with “step 1: install docker”
people often assume if they run their own instances it will be more secure. from one perspective it’s true: everything is in your own hands; from the other perspective, they are rarely capable of doing it correctly.
Well that’s just great it really is a shame though how some people would actively want to ruin something free like this just because they can.
“Some people just want to watch the world burn”
On a positive note. It’s much better that these things happen and vulnerabilities are discovered while we’re still a small-ish community.
One wonders how much spez paid for that hack.
lots of
It seems there is no way in Lemmy to invalidate all your session cookies? Without that, how can you secure an account which has a stolen session cookie?
The ‘secret’ in the database was changed, causing all existing cookies to become invalid.
Very good. I think a feature where a user can revoke all their cookie sessions is still worthwhile, and maybe I’ll look at raising a feature request for that, but it is good to know that cookies stolen during the recent hack have already been addressed.
what steps are being taken to ensure it doesn’t happen again? was any personal data compromised for users?
Good point, I’ll update the post.
So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I’m sure they were only interested in admin cookies, so most others were “useless” to them? I see nothing wrong with my posts so I should be safe, right?
Prior to the JWT secret being rotated, yes, they could have authenticated as you. The tokens are now all invalid and useless
If you think they could change your password:
YES, they could.
They could have changed the email => “Forgot PW” and with that you lost ur account.
I think I’ve lost my account, I clicked Forgot Password and nothing came into my mailbox. This account is the one I made just now.
My old account:
If you see that account post or comment on anything, please report it
Edit: Nvm, I use another email to sign up for Lemmy and forgot about it
Report it directly to Ruud or otherwise he will just delete it.
actually nevermind, I forgot that I use a different email for Lemmy, I can log back in now
Also I am curious, what’s the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?
Mail: info@lemmy.world Mastodon: @mwadmin@mastodon.world Matrix: https://matrix.to/#/#lemmy-support-general:discuss.online
Why wasn’t there an info on /lemmy-world.statuspage.io ?
I think the admins that were on it didn’t think of updating the status page…
That explains why I had to clear my browser cache, I was unable to login until I did.
Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.
For capable people this is a minor annoyance but whenever there’s an “everyone needs to login again” issue, we will lose mere mortal users. In this case it wasn’t even clear that was needed - I appeared to be logged in but nothing worked. Ordinary users give up over things like this. I’ve seen it happen many times on sites where I had access to the analytics. I hope we regard this as a really bad thing to be avoided at all costs and not a “no big deal, just log in again.” Easy for you, easy for me, many others will just bail.
Further 3rd party heads up – for us nontech refugees:
If it looks like you are logged in, you may not be. I use Connect, and at your reminder, I clicked my acct and it says I was logged in. I tried to comment that Connect login was working, and my comment didn’t show up.
I tried again, only to see an ”error: not logged in” message pop up.
Signed out, signed in again manually, and all is well.
So do a double check, Lemurs. Trust in your actions, not your eyes.
Seem to have a hard time loging in
I’ve been struggling to login this morning.
You guys really have my highest respect for spending so much time to keep this running, despite all the recent trouble and now even an attack.
Thank you very much <3 You guys are awesome and I really appreciate how publicly you deal with this.
