Разбор threading vs multiprocessing vs asyncio в Python

При работе с Python да и другими языками программирования часто возникает необходимость ускорения выполнения кода, масштабирования обработки данных или работы с большим количеством сетевых запросов. Именно в Python для решения этих задач существуют три базовых метода. Это: threading, multiprocessing и asyncio. На первый взгляд – механизмы схожие. Но при детальном разборе ясно, что они решают принципиально разные задачи, опираются на разные модели исполнения и обладают своими ограничениями. В статье расскажу об особенностях каждого метода – будет интересно и познавательно.

https://habr.com/ru/articles/991478/

#threading #multiprocessing #asyncio #Python #параллельность #конкурентность #CPUbound #event_loop #многопоточность

Very interesting read about how ~1970‘s C-based synchronous timeslice hardware architecture doesn‘t fit +2000’s networked GUI #multiprocessing anymore.

It‘s true, even if we live the paradigm change in #programming we need to bolt it on the previous stuff using async or dispatcher libs, message handlers, etc. and then we say „look mum, I made it event-driven and distributed“ 😅

It is even difficult to imagine an AS/400‘s memory model if you lived in a C landscape.

https://programmingsimplicity.substack.com/p/hardware-stockholm-syndrome

Мониторинг Celery. Pull-модель

В этой статье рассмотрим возможность получать метрики Celery непосредственно от самих воркеров, хитрости, на которые придётся пойти, чтобы решить эту задачу, и, самое главное, какие преимущества от этого можно получить по сравнению с классическим подходом к мониторингу Celery. Также продемонстрирую небольшой Django-проект и пример его конфигурации. Особое внимание будет уделено режиму мультипроцессинга и тому, как та или иная конфигурация запуска Celery будет влиять на сложность решения.

https://habr.com/ru/companies/domclick/articles/942584/

#celery #celery_worker #celery_flower #celery_beat #metrics #multiprocessing #multithreading #monitoring

There was something interesting going on on one of my systems:

If a certain function in #Python was called as a separate process with the #multiprocessing library, then the sort_values function of #Pandas would just hang (and therefore the process would never produce the output I was waiting for). Called from the main process was OK.

The solution was to change the sorting algorithm by the `kind="stable"` parameter. Weird.

#ArchLinux #Linux #programming #parallelization

Почему multiprocessing.Queue() тормозит и как обойти это с помощью shared_memory

Привет, Хабр! Вы запускаете многопроцессную задачу, кидаете данные в multiprocessing.Queue() , а потом вдруг замечаете... что всё тормозит. Муторно. Медленно. Местами прям отвратительно. Вы смотрите в монитор, на top, на htop, на код — и не понимаете: ну ведь должно же летать! А не летит.

https://habr.com/ru/companies/otus/articles/913200/

#python #multiprocessing #очередь #производительность #shared_memory #межпроцессное_взаимодействие #кольцевой_буфер

https://kitfucoda.medium.com/telegram-chatbots-evolution-decoupling-parallel-python-s-shared-state-for-clarity-e76880ce9b1f

My recent refactoring journey began with a take-home assignment's hidden pitfall: a hard-coded queue in settings.py. This seemingly small detail quickly grew into a global state headache, making testing and modularity a real challenge.

To tackle this, I focused on explicitly managing inter-process communication. The solution involved leveraging multiprocessing.Manager to centralize shared synchronization primitives, allowing for much cleaner dependency injection across parallel processes.

This refactoring delivered significant benefits. It drastically improved testability by eliminating the need for global patching. It also brought consistency and enhanced modularity, laying a robust foundation for future scalability.

It's been a valuable lesson, directly applicable to my own operational chatbot. While not a performance optimization, the structural clarity gained is immensely rewarding. I'm looking forward to diving deeper into the testing aspects of this setup soon!

#Python #Refactoring #Multiprocessing #CleanCode #SoftwareArchitecture #getfedihired #fedihire #OpenToWork

I've been working on a #space #visualization tool for our operators. It basically needs to always know, and be ready to plot, where every single one of 60k+ objects is down to millidegree/meter/second resolution just in case the sensor suddenly slews there

My own constraint is that it has to be 1) a single 2) #python executable because otherwise I'm not interested

Earlier this year, I found a great 30x faster technique for determining which #satellites are above the horizon. (In fact, it's far more general than that, but that's all the help it gives me to this problem.)

I also realized I could spawn a #multiprocessing child to do lookahead on data and then pass a huge #numpy array to my graphing process. (Investigated ~9 different ways, chose the best)

But there I was stuck.

At any given moment, there are ~4500 space objects above the horizon (at our latitude). Putting 4500 points with little persistence trails and labels and then updating all that at 1Hz let alone the 10Hz I'd like was taking too long, even using the amazing #pyqtgraph

So there I was stuck. Until this week.

https://kitfucoda.medium.com/concurrency-vs-parallelism-achieving-scalability-with-processpoolexecutor-c366bfcc6207

Concurrency and parallelism are often confused in async programming discussions. Go's goroutines highlighted the difference: concurrency is doing many things at once, while parallelism is doing many things at the same time.

AsyncIO handles concurrency well for I/O, but CPU-bound tasks need parallelism. Python uses AsyncIO for concurrency, and ProcessPoolExecutor for parallelism, distributing work across CPU cores.

Process communication is harder than thread communication. AsyncIO's task cancellation differs from ProcessPoolExecutor's, requiring workarounds like event objects for reliable cancellation and shutdown.

Essentially, ProcessPoolExecutor enables parallelism for CPU-bound tasks, scaling them across multiple cores, while AsyncIO handles I/O concurrently.

#python #asyncio #concurrency #parallelism #multiprocessing #opentowork #getfedihired #fedihire #opentowork