Mastodawn

Self-host Reddit – 2.38B posts, works offline, yours forever

https://lemmy.dbzer0.com/post/61554734

Self-host Reddit – 2.38B posts, works offline, yours forever - Divisions by zero

Reddit’s API is effectively dead for archival. Third-party apps are gone. Reddit has threatened to cut off access to the Pushshift dataset multiple times. But 3.28TB of Reddit history exists as a torrent right now, and I built a tool to turn it into something you can browse on your own hardware. The key point: This doesn’t touch Reddit’s servers. Ever. Download the Pushshift dataset, run my tool locally, get a fully browsable archive. Works on an air-gapped machine. Works on a Raspberry Pi serving your LAN. Works on a USB drive you hand to someone. What it does: Takes compressed data dumps from Reddit (.zst), Voat (SQL), and Ruqqus (.7z) and generates static HTML. No JavaScript, no external requests, no tracking. Open index.html and browse. Want search? Run the optional Docker stack with PostgreSQL – still entirely on your machine. API & AI Integration: Full REST API with 30+ endpoints – posts, comments, users, subreddits, full-text search, aggregations. Also ships with an MCP server (29 tools) so you can query your archive directly from AI tools. Self-hosting options: - USB drive / local folder (just open the HTML files) - Home server on your LAN - Tor hidden service (2 commands, no port forwarding needed) - VPS with HTTPS - GitHub Pages for small archives Why this matters: Once you have the data, you own it. No API keys, no rate limits, no ToS changes can take it away. Scale: Tens of millions of posts per instance. PostgreSQL backend keeps memory constant regardless of dataset size. For the full 2.38B post dataset, run multiple instances by topic. How I built it: Python, PostgreSQL, Jinja2 templates, Docker. Used Claude Code throughout as an experiment in AI-assisted development. Learned that the workflow is “trust but verify” – it accelerates the boring parts but you still own the architecture. Live demo: https://online-archives.github.io/redd-archiver-example/ [https://online-archives.github.io/redd-archiver-example/] GitHub: https://github.com/19-84/redd-archiver [https://github.com/19-84/redd-archiver] (Public Domain) Pushshift torrent: https://academictorrents.com/details/1614740ac8c94505e4ecb9d88be8bed7b6afddd4 [https://academictorrents.com/details/1614740ac8c94505e4ecb9d88be8bed7b6afddd4]

PLEASE SHARE ON REDDIT!!! I have never had a reddit account and they will NOT let me post about this!!

El Barto Jan 14

Anyone doing this will be banned.

We can’t share this on Reddit, but we can share this on other platforms. Basically, what you have done is you scraped tons of data for AI learning. Something like <<create your own AI Redditor>>. And greedy Reddit management will dislike it very much even if you will tell them that this is for the cultural inheritance. Your work is great anyway. Sadly, that I do not have enough free space to load and store all this data.

And only a 3.28 TB database? Oh, because it’s compressed. Includes comments too, though.

Yes! Too many comments to count in a reasonable amount of time!

douglasg14b Jan 13

Yeah, it should balloon out to 15TB or more I think

muusemuuse Jan 14

If only I had the space and bandwidth. I would host a mirror via Lemmy and drag the traffic away.

Actually, isn’t the a way to decentralize this that can be accessed from regular browsers on the internet? Live content here, archive everywhere.

psycotica0 Jan 14

Someone could format it into essentially static pages and publish it on IPFS. That would probably be the easiest “decentralized hosting” method that remains browsable

breakingcups Jan 13

Just so you’re aware, it is very noticeable that you also used AI to help write this post and its use of language can throw a lot of people off.

Not to detract from your project, which looks cool!

Yes I used AI, English is not my first language. Thank you for the kind words!

mustlane Jan 13

It’s not an excuse. You are just lazy.

MadMonkey Jan 13

Brush, you do not seem like a nice person to be around.

Spread love and kindness, not hate.

I hope you have a better rest of your day.

idealism_nearby Jan 13

Would love to see you learn an entire foreign language just so you are able to communicate with the world without being laughed at by people as hostile as yourself.

fartographer Jan 14

I can’t even learn my own language!

potustheplant Jan 14

They said it wasn’t their “first” lanugage. Which leads me to believe that they do speak English. If that’s the case, then they indeed are kind of lazy. There have already been studies in the impact of AI when used for communication and the results are not positive.

This isn’t something I’d personally point out and criticize, just something I wouldn’t do. Take the time to express your own ideas in your own words. The long term cost is higher than the short term gains.

I have A1 and A2 level in a couple of non-first languages, technically I can speak those, realistically I don’t and will not be able to communicate something more complex than ‘here, take a look’

So I don’t agree with your absolutistic stance

potustheplant Jan 15

There’s nothing “absolutistic” about my “stance”. If you’re rusty using a language, you won’t get better if someone else does the homework for you. Make an effort, make mistakes, write in a way that sounds weird, who cares. But practice. If you only take the easy way out, that’ll be your only option in the future.

Although, like I already said, that’s MY way of thinking about it. If you want to use ai to write your stuff, you do you. It doesn’t negate the fact that, whle it’s not “wrong”, it’s the lazy (or minimum effort) option. Don’t know why it bothers you so much.

Hey I drove to the library, picked up all these things you needed, got dinner here ya go, free!

You drove? man that’s lazy…

He used AI to clean up translation and save time after he spent a fuck ton of time curating and delivering us a helpful product. Calling him out as lazy is an awful take.

there are the so called activists that complain alot then there are the activists that deliver projects and code… enough said

potustheplant Jan 15

“Activists”? What are you even talking about?

Regardless, I specifically said that what you did wasn’t wrong or anything likw that. I simply think that it’s going to do you more harm than good in the long run. You’re free to do whatever you want though, obviously.

Another piece of advice. When someone simply shares an opinion, don’t get instantly butthurt over nothing. Otherwise this might as well be reddit.

potustheplant Jan 15

First, that’s an awful analogy.

Second, you’re assuming (for some unknown reason) that they “cleaned up” the “translation” using ai. You have literally no idea exactly how they wrote the post. It’s kinda weird to make up a random scenario but ok.

Third, no, it’s not an awful take. You can code something that requires a ton of effort but write awful documentation. One thing does not make the other impossible.

Fourth, I already explained that there have already been studies that concluded that using AI to write stuff for you has a negative impact on your communication skills. This is not an opinion or me being ingrateful or whatever. I was just sharing information.

If that documentation was awful, I’d REALLY like to see your take on NixOS :)

potustheplant Jan 15

That was a hypothetic example, not my opinion on this post.

I mean I can’t see what the comment was and I’m assuming it must’ve been downright hateful, but that person almost certainly has learned a foreign language just to communicate with the world and in fact had to learn another foreign language in school because their name is Estonian for “gypsy” and learning two foreign languages (usually English and Russian, sometimes German or something else for the second foreign language) is required. Likely they speak 2.5 languages as is common here (my German is so bad I count it as half a language - native speakers speak too fast for me, but I can kinda get my point across if needed), but could be more.

Just pointing out that even when trying to be accepting of others, subtle anglo-defaultism can show up in your comment, not that I necessarily agree with whatever the comment was.

irmadlad Jan 13

Yu mussi bawn backacow

Shut the fuck up loser.

Melvin_Ferd Jan 13

You’re awesome. AI is fun and there’s nothing wrong with using it especially how you did. Lemmy was hit hard with AI hate propaganda. China probably trying to stop it’s growth and development in other countries or some stupid shit like that. But you’re good. Fuck them

Yup, if there was ever a decent use for AI, this is it. Lemmy can (and will) hate the shit out of it, but it took a little burden off the shoulders of someone doing us a great service.

Melvin_Ferd Jan 13

I fucking hate lemmy sometimes.

UnderpantsWeevil Jan 13

I would sooner download a tire fire.

thanks anyway for looking at my project 🙂

irmadlad Jan 13

I use Reddit for reference through RedLib. I could see how having an on-premise repository would be helpful. How many subs were scrapped in this 3.28 TB backup? Reason for asking, I’d have little interest in say News or Politics, but there are some good subs that deal with Linux, networking, selfhosting, some old subs I used to help moderate like r/degoogle, r/deAmazon, etc.

the torrent has data for the top 40,000 subs on reddit. thanks to watchful1 splitting the data by subreddit, you can download only the subreddit you want from the torrent 🙂

irmadlad Jan 13

Sweet! I’ll check it out.

Say what you will about Reddit, but there is tons of information on that platform that’s not available anywhere else.

UnderpantsWeevil Jan 13

:-/

You can definitely mine a bit of gold out of that pile of turds. But you could also go to the library and receive a much higher ratio of signal to noise.

pixeltree Jan 14

This one specific bug in this one niche library has probably not been written about in a book, and even if it has I doubt that book is in my local library, and even if it is I doubt I can fucking find it

mirisgaiss Jan 14

obscure problems almost always have reddit comments as search results, and there’s no forums or blogs with any of it anymore. be nice to have around solely for that… though I’m sure if shit like /pics or whatever else was removed it could get significantly smaller…

Howlinghowler110th Jan 13

I think this is a good use case for AI and Impressed with it. wish the instructions were more clear how to set up though.

thank you! the instruction are little overwhelming, check out the quickstart if you haven’t yet! github.com/19-84/redd-archiver/…/QUICKSTART.md

redd-archiver/QUICKSTART.md at main · 19-84/redd-archiver

A PostgreSQL-backed archive generator that creates browsable HTML archives from link aggregator platforms including Reddit, Voat, and Ruqqus. - 19-84/redd-archiver

GitHub

Wow, great idea. So much useful information and discussion that users have contributed. Looking forward to checking this out.

thank you!!! i built on great ideas from others! i cant take all the credit 😋

Tanis Nikana Jan 13

Reddit is hot stinky garbage but can be useful for stuff like technical support and home maintenance.

Voat and Ruqqus are straight-up misinformation and fascist propaganda, and if you excise them from your data set, your data will dramatically improve.

the great part is that since everything is built it is easy to support any additional data! there is even an issue template to submit new data source! github.com/19-84/…/submit-data-source.yml

redd-archiver/.github/ISSUE_TEMPLATE/submit-data-source.yml at main · 19-84/redd-archiver

A PostgreSQL-backed archive generator that creates browsable HTML archives from link aggregator platforms including Reddit, Voat, and Ruqqus. - 19-84/redd-archiver

GitHub

Avid Amoeba Jan 13

How does this compare to redarc? It seems to be similar.

GitHub - Yakabuff/redarc: Reddit archiver

Reddit archiver. Contribute to Yakabuff/redarc development by creating an account on GitHub.

GitHub

redarc uses reactjs to serve the web app, redd-archiver uses a hybrid architecture that combines static page generation with postgres search via flask. is more like a hybrid static site generator with web app capabilities through docker and flask. the static pages with sorted indexes can be viewed offline and served on hosts like github and codeberg pages.

Avid Amoeba Jan 14

Is there difference in how much storage space is needed between the two approaches?

redd-archiver will take up more disk space because the database exists along with the static html

😈MedicPig🐷BabySaver😈Jan 13

Fuck Reddit and Fuck Spez.

muusemuuse Jan 14

You know what would be a good way to do t? Take all that content and throw it on a federated service like ours. Publicly visible. No bullshit. And no reason to visit Reddit to get that content. Take their traffic away.

El Barto Jan 14

Where would it be hosted so that Conde Nast lawyers can’t touch it?

muusemuuse Jan 14

What would they say? It’s information that’s freely available, no payment required, no accounts to simply read it, no copyrights, where’s the legal in hosting a duplicate of the content?

limelight79 Jan 14

It might fall under the same concept that recipes do - you can’t copyright a recipe, but a collection of recipes (such as a book) is copyrightable.

In any case, they have a lot more money to pay lawyers than you or I do, I’ll bet, so even if you are right, that doesn’t mean you’ll have the money to actually win.

muusemuuse Jan 15

So distribute it and n a fault tolerant way. They can’t sue all of us.

El Barto Jan 15

Oh I agree with you, friend. The problem is that they’ll say that they’re losing ad revenue. So they’ll try and sue, even if they’re in the wrong.

muusemuuse Jan 15

Fine, decentralize it then. And fuck your ad revenue, nobody likes you, Spez!

a1studmuffin Jan 13

This seems especially handy for anyone who wants a snapshot of Reddit from pre-enshittification and AI era, where content was more authentic and less driven by bots and commercial manipulation of opinion. Just choose the cutoff date you want and stick with that dataset.

What is the timing of the dataset, up through which date in time?

douglasg14b Jan 13

It’s literally says in the link.

Oh I didn’t see it. I’m sorry I asked.