Google for Developers (@googledevs)

Google Colossus를 PyTorch에서 더 빠르게 활용할 수 있도록 Rapid Bucket과 fsspec(GCSFS) 기반 개선을 소개한다. 읽기 4.8배, 쓰기 2.8배, PyTorch Lightning에서 전체 학습 시간 23% 단축 등 대규모 학습 성능 향상이 핵심이다.

https://x.com/googledevs/status/2049534176173383982

#pytorch #googlecolossus #fsspec #gcsfs #pytorchlightning

So much cool #TIL stuff in there: https://csvbase.com/blog/7

#csvbase (https://csvbase.com/) is a site for sharing (and getting) csv files

#fsspec (https://github.com/fsspec/filesystem_spec/) and how to use it in your code (change `open()` with `fsspec.open()`) and how to extend it with your own custom url schemes let it work in pandas/polar/...

#python

Client libraries are better when they have no API: https://csvbase.com/blog/7

#CSV #pandas #polars #dask #curl #fsspec

fsspec и вообще зачем оно нам нужно

Привет! Сегодня я расскажу вам про fsspec, его киллер фичи и почему он является незаменимым инструментом любого python разработчика.

https://habr.com/ru/articles/781804/

#fsspec #filesystem #s3 #cloud_storage