Mastodawn

netrom

Mar 10

Oh, my #ccache is filling up:

Cache size (GiB): 40.3 / 100.0 (40.25%)

Note that this is #zstd compressed.

#dev

Show thread

Sakarepmu Weh Mar 3

@davidculley so what? Actually, many contributions from #meta #facebook to #opensource world including #MySQL. #RocksDB born there. And don't forget #zstd.

Yuka Mar 3

My new daily backup script, pg_dump with zstd compression level 19.

docker exec \
   ak-postgres-1 \
   pg_dump -U umeyashiki umeyashiki_akkoma | \
   nice -n 19 \
   ionice -c 3 \
   chrt --idle 0 \
   zstd -T0 -19 --rsyncable -q > "$BACKUP_DIR/db_latest.sql.zst";

Today I learned something new about the scheduling priority. Since zstd compression is very CPU-intensive, set it to low priority so it doesn’t slow the entire system down during compression.

Commands that precede zstd here are:

nice -n 19 [cmd]
ionice -c 3 [cmd]
chrt --idle 0 [cmd]

By chaining nice, ionice, and chrt together before the zstd command, the script forces the compression process to run with the absolute lowest possible priority for both the CPU and the disk.

nice 19 is the lowest priority CPU priority.
ionice with class 3 means idle. A program running with idle I/O priority will only get disk time when no other program has asked for disk I/O for a defined grace period.
chrt --idle 0: Set scheduling policy to SCHED_IDLE (scheduling very low priority jobs).

References:

man 1 chrt
man 1 nice
man 1 ionice

#linux #docker #zstd #postgresql #sched

Seth Larson Feb 12

Interesting blog post about text classification using compression, specifically the new "compression.zstd" module contributed by @emmatyping

https://maxhalford.github.io/blog/text-classification-zstd/

#compression #zstd #zstandard #classification

Text classification with Python 3.14's zstd module

Python 3.14 introduced the compression.zstd module. It is a standard library implementation of Facebook’s Zstandard (Zstd) compression algorithm. It was developed a decade ago by Yann Collet, who holds a blog devoted to compression algorithms. I am not a compression expert, but Zstd caught my eye because it supports incremental compression. You can feed it data to compress in chunks, and it will maintain an internal state. It’s particularly well suited for compressing small data. It’s perfect for the classify text via compression trick, which I described in a previous blog post 5 years ago.

Max Halford

Habr Jan 28

[Перевод] Пишем свой git: минимальная реализация на Rust

Контроль версий долгое время был для меня «чёрным ящиком»: я не понимал, как именно хранятся файлы, как формируются diff’ы и из чего состоят коммиты. А поскольку я люблю изобретать велосипеды, почему бы не попробовать реализовать git самому?

https://habr.com/ru/companies/cloud4y/articles/990052/

#git #rust #контроль_версий #sha256 #zstd #хеширование #petproject #системы_контроля_версий #объекты_git #commit

Пишем свой git: минимальная реализация на Rust

Хабр

mort Jan 22

Many people seem weirdly suspicious about Google's #Brotli compression while being weirdly chill about Facebook's #ZSTD, to the point of commenting on posts about Brotli compression being added to things to the effect of, "This is a conspiracy by Google, they clearly should've chosen ZSTD instead". What's up with that? Is Google really so much less scary than 𝘍𝘢𝘤𝘦𝘣𝘰𝘰𝘬?

(This is a subtweet about a certain HN post about Brotli compression coming to #PDF)

#compression #zstandard #computers