partizan

Hey people, what are you using to run your background tasks in #Python?

I was using #celery for a long time, and even contributed some bugfixes to it, but today on freelance project we had some restrictions - like "it runs on pythonanywhere", so we cannot use rabbitmq or redis as celery brokers.

So, i started looking for other solutions and found #dramatiq. With dramatic-pg it can use postgres as a broker.

#django

Nov 29, 2023 at 9:16pmWeb
Show threadGabriel PettierNov 29, 2023
@partizan we use apscheduler, it's fine, and it also supports postgres. We didn't try the new v4 series yet, which is apparently an important break from previous versions.
Show threadJesuis  JedničkaNov 30, 2023
@partizan only worked with twisted
Show threadJohn Sturgeon (pdx) has movedDec 1, 2023

@partizan I've tried celery, but there was just too much infra to maintain for the many projects I have.

Various cron schedulers was next, but that was not powerful enough for what I was after.

Oddly, I have now landed on https://www.prefect.io

I love the ability to monitor and control my tasks / flows remotely.

Workflow Orchestration Made Simple | Prefect

Prefect offers modern workflow orchestration tools for building, observing & reacting to data pipelines efficiently.

Show threadpartizanDec 1, 2023
@johnsturgeon Whoa, looks interesting, thanks!
Show threadJohn Sturgeon (pdx) has movedDec 1, 2023

@partizan You bet, also .. there's a bit of a learning curve and it took me a while to get my head around it and start using it effectively, so if you have questions, feel free to shoot them my way.

I also ditched Doppler for my secret management and use Prefect Secret blocks and Prefect variables. Very happy with that as well