Gotosocial.social AdminsHi everyone, some users can’t access their home or local timelines since the last update (other features are fine). I’m looking into it and will keep you posted in this thread.
I think I've found the issue. And I’m working on a fix now.
Unattributed 👤 ☑@admin Glad to know it's not just me....was baffled why notifications worked, but not the home timeline.
@Unatributed Sorry for the trouble, I've mitigated the issue. You can try again and see if the home timeline loads now :)
@admin Yup, it's working now... Thanks. From the mitigation time sounds like this was a doozy...any extra support needed?
@Unatributed Yeah, this really took me a lot longer than usual to debug and mitigate. Currently I don’t need extra support, but I really appreciate the offer and your initial feedback 
I’ve already reported this to the GoToSocial developers. You can track the progress at Issue #4757 and Issue #4758.
As for why the debugging took so long, here are some technical details (feel free to skip if you’re not interested, it's a performance issue, and no data was lost):
At first, I saw a lot of slow database queries in the logs, which I assumed were just post-migration background tasks. However, timelines still wouldn't load and eventually started throwing 500 errors. That’s when I saw this: ERROR: could not resize shared memory segment ... No space left on device (SQLSTATE 53100).
Realizing it was a shared memory exhaustion, I allocated more shm to the DB, and the errors just turned into request timeouts. Since other users were reporting issues too at this time, I started digging into the recent migration changes.
I initially misidentified the cause as index failure due to inconsistent table aliases, which is a relatively simple fix, so I posted that I was working on a fix. But the problem turned out to be more complex. A simple hotfix on our DB wasn't enough; I either had to manually build a custom version and push it to the server, or submit a PR and wait for gts dev to review and merge.
Since we always use official builds, I didn’t have a workflow ready for custom builds, meaning everything had to be done manually. I was pretty exhausted by then, so I submitted the details and headed to bed.
When I woke up, the GTS devs had already pushed a fix. I updated and tested it, but the issue persisted. Finally, I discovered that by combining the new fix with some temporary manual indexing, I could get the timelines loading again. That’s how I finally managed to mitigate the issue.
[bug] Home timelines and local timelines timing out in PostgreSQL deployments
### Describe the bug with a clear and concise description of what the bug is. After #4752, some users on my instance experience request timeouts when opening their home timeline or local timeline. The problem is especially visible on accounts that follow many other accounts and therefore have a ...
@admin @unattributed Yikes... I would have been lost after upping the shm.
I was worried that things were resource (I/O, CPU, memory) bound, and was thinking if you needed to throw some additional resources at things I could potentially have helped out financing getting the resources.
@Unatributed @unattributed aww, thank you for the kind offer 🥺 GtS is quite lightweight, so CPU and RAM aren't really the issue. The real bottleneck for an instance like ours is storage, because GtS is designed to cache all remote media locally, and the database as well, since remote post pruning hasn't been implemented yet.
I managed to reorganize my resources before the recent price hikes, so I still have some headroom. My total costs only went up by less than 10%, so we’re all set for now. The only potential hurdle would be if I need to move to a 3-node storage cluster later on, but that's a worry for another day.