Mastodawn

Title: P2: HF tokenizers from sources [2025-05-24 Sat]

I didn't find a working solution to manage "cfg" and, so
I just patched the Cargo.toml files of dependencies by
commenting out lines.
(∠・ω )⌒ #dailyreport #tokenizers #huggingface #rust #gentoo #ebuild #secops #cargo

anoncheg Apr 27

Title: P1: HF tokenizers from sources [2025-05-24 Sat]

tokenizers require HF hub for testing, that I disabled.

It was a hard but good experience with Cargo package
manager of Rust. The main problems was due to strange
cfg flags that Gentoo should have set automaticly, for
ex. target_os=linux was not set. "cfg" is an
abomination that you can't add change this safely. #dailyreport #tokenizers #huggingface #rust #gentoo #ebuild #secops #cargo

anoncheg Apr 27

Title: P2: P0: HF tokenizers from sources [2025-05-24 Sat]
I removed optional dependencies and disabled HTTP
requirements to enhance security.

I wrote very simple tests for tokenizers, safetensors,
transformers and integration test for them, because #ebuild #secops #cargo #dailyreport #tokenizers #huggingface #rust #gentoo #ebuild #secops #cargo

anoncheg Apr 27

Title: P1: P0: HF tokenizers from sources [2025-05-24 Sat]

#ebuild #secops #cargo
I compiled the HF 🤗 tokenizers library from sources and
enhanced Gentoo ebuild file to allow reproducible
installation from sources. #ebuild #secops #cargo #ebuild #secops #cargo #dailyreport #tokenizers #huggingface #rust #gentoo #ebuild #secops #cargo

anoncheg Apr 27

Title: P3: Safetensors and Rust cargo [2025-05-19 Mon]

12) try to build/emerge, some dependencies still may be
missed, but downloaded with “cargo vendor”.
- may help: rustup update nightly, cargo +nightly -Z
unstable-options build --release --build-plan

I removed Cargo.lock from bindings/python. For me
optimization has greater priority than stability. #dailyreport #rust #gentoo #ebuild #safetensors #huggingface #sci

anoncheg Apr 27

Title: P2: Safetensors and Rust cargo [2025-05-19 Mon]

- Generate Cargo.toml dependencies from “cargo tree”
8) Copy output to [dependencies] in Cargo.toml, in
original ones change version only.
9) (script) Remove “features = ["std"],”
10) check that count of dependencies not reduced, you may
repeat step (7)
11) (script to create) CRATES= for Gentoo Ebuild: from
“cargo tree”. #dailyreport #rust #gentoo #ebuild #safetensors #huggingface #sci

anoncheg Apr 27

Title: P1: Safetensors and Rust cargo [2025-05-19 Mon]

2) tar xpf v0.5.3.tar.gz
3) cd safetensors-0.5.3/safetensors
4) comment [dev-dependencies] section in Cargo.toml
5) “cargo tree -e normal,build”
6) specify subdependencies in Cargo.toml from script
7) (script) run “cargo tree -e normal,build”
again. Some packages require “std” some not, remove
“std” if they not require. #dailyreport #rust #gentoo #ebuild #safetensors #huggingface #sci

anoncheg Apr 27

Title: P2: P0: Safetensors and Rust cargo [2025-05-19 Mon]
for one reason: exclude defaults and optional
dependendies from "so called" transitive dependencies
(subdependencies) from fxxxing Rust.

Steps:
1) wget https://github.com/.../v0.5.3.tar.gz #huggingface #sci #dailyreport #rust #gentoo #ebuild #safetensors #huggingface #sci