APScheduler + requests 遇到 OSError: [Errno 24] Too many open files 的問題

前陣子試著在 Python 上面定時跑 crawler 抓一些東西寫進資料庫。 因為是偏實驗性質,選擇用 PyPy (PyPy3) 而非 CPython (就是官方版本的 Python),然後搭著 Gunicorn (跑 Flask) + APScheduler + requests,專案的程式碼可以在 app/app.

Gea-Suan Lin's BLOG
TIL: #fail2ban läuft ganz offiziell mit #pypy3!

Wrzuciłem dzisiaj nową weresję #PyPy3.10 do #Gentoo, która poprawia, ekhm, "iterację po współprogramach przy pomocy API rozszerzeń C". Albo, mówiąc prościej, naprawia rozszerzenie yajl2_c w paczce ijson.

Chciałbym z całego serca podziękować rtobar za włożenie wielkiego wysiłku, by rozkminić ten problem, naprawić PyPy i przy okazji poprawić kilka innych problemów po stronie ijson. Naprawdę taką współpracę chciałbym widywać codziennie!

https://github.com/ICRAR/ijson/issues/101
https://foss.heptapod.net/pypy/pypy/-/issues/3956

#Python

yajl2_c backend crashes on PyPy3 · Issue #101 · ICRAR/ijson

Describe the bug When running the test suite with yajl2_c backend on PyPy3 (tried 3.9 and 3.10, v7.3.12), it crashes with a segmentation fault: $ pypy3.10 -m unittest -v [...] test_basic_parse (tes...

GitHub

I've pushed a new #PyPy3.10 build to #Gentoo today with a fix for, err, "iterating over coroutines in C extension API". Or, to put it simpler, it fixes problems with yajl2_c extension in ijson.

I'd like to wholeheartedly thank rtobar for going above and beyond the call of duty, figuring out where the problem is, fixing PyPy and making a bunch of fixes to ijson as well. This is really awesome kind of interaction I'd love to see every day!

https://github.com/ICRAR/ijson/issues/101
https://foss.heptapod.net/pypy/pypy/-/issues/3956

#Python

yajl2_c backend crashes on PyPy3 · Issue #101 · ICRAR/ijson

Describe the bug When running the test suite with yajl2_c backend on PyPy3 (tried 3.9 and 3.10, v7.3.12), it crashes with a segmentation fault: $ pypy3.10 -m unittest -v [...] test_basic_parse (tes...

GitHub

I've done a benchmark between Rust, C, Common Lisp and Python on my SBC which has ARM processor and here are the results:

https://far.chickenkiller.com/computer/benchmarking-common-lisp-rust-c-and-python-on-arm/

#benchmark #c #lisp #cl #common_lisp #rustlang #arm #armv7 #pi #python #cpython #pypy #pypy3

Benchmarking Common Lisp, Rust, C and Python on ARM

I have previously benchmarked C++ against Common Lisp and the result was that unlike the common belief about Lisp, it can have performance which is almost as fast as C++. After that I did another benchmark for computing fibonnaci series using Rust, Common Lisp, C and Python. But both of these benchmarks were done on x86_64. In this benchmark, I’m using the same codes from the last benchmark but run them on an ARM device instead. Unfortunately, my device is old and not ARM64 but still it can be of interest.

Farooq's