Carlton Gibson 🇪🇺PSA: During @djangoconeurope, managed to persuade @be_haki — “that database guy” — that he should join the fediverse. You should follow him! And check out the posts on his site if you’ve not seen it before. Utter gold mine.
Jake Howard
Haki Benita@be_haki Consider yourself nerd-sniped 😉 ❤️
very bad performance with too many DBTaskResult entries · Issue #140 · RealOrangeOne/django-tasks
I had the db_worker running all of yesterday but noticed today that it was orders of magnitudes slower than yesterday. I changed quite a lot of code so it was not clear right from the start, but ev...
@jake looks like you already found the culprit, get_locked()
I saw issue 118 that added the index on status. I personally would do it differently - I would add a partial index on ID where status = ready. This should speed up the process of looking for the next task to execute.
Also, splitting pending/ongoing tasks from completed tasks should eliminate some of the issues you mentioned (at the expense of having to check two tables to get the status of a given task). Have you ever considered it?
@be_haki Separating out tasks from results could be interesting, especially in making the tables smaller.
Adding an index for the ordering for new tasks seems to be the easiest way to use an index, plus a few on useful columns. Reliably testing that indexes are used is as always, a hard problem 😭