anonchegApr 18
Title: P4: FOSDEM 2024 offline [2024-02-09 Fri]
trainable parameters. Low-rank subspace finetuning. Part of the model's input embeddings is fine-tuned via gradient descent.
- Fastfood transform to reparametrize the update to NN params.
- LoRa - simple low-rank matrix decomposition(or Kronecker product decomposition) to parametrize the weight
update
😶 #dailyreplort #llm #ai #architect #architecture #peft
anonchegApr 18
Title: P2: P3: FOSDEM 2024 offline [2024-02-09 Fri]
- soft prompts - consists of a task description accompanied by a few in-context examples
- *selective* - fine-tuning only selected layers/biases/rows
- *reparametrization-based* (kind of additive) - leverage low-rank representations to minim the number of #dailyreplort #llm #ai #architect #architecture #peft
anonchegApr 18
Title: P1: P3: FOSDEM 2024 offline [2024-02-09 Fri]
- prompt tuning or modifications - hard or soft or prefix tuning (as LLaMa adapter) - appends a tensor to
the embedded inputs of a pretrained LLM #dailyreplort #llm #ai #architect #architecture #peft
anonchegApr 18

Title: P2: FOSDEM 2024 offline [2024-02-09 Fri]
https://www.geeksforgeeks.org/difference-between-access-control-list-and-capability-list/

LLM model size increasing 2-5 times order of magnitude quicker than
single GPU RAM do.

Types of PEFT methods (from my notes):
- *additive* - augmenting the existing pre-trained model with extra parameters or layers and training only the
newly added
- adapters - add additional parameters to each transformer block. #dailyreplort #llm #ai #architect #architecture #peft

Difference Between Access Control List and Capability List - GeeksforGeeks

Your All-in-One Learning Portal: GeeksforGeeks is a comprehensive educational platform that empowers learners across domains-spanning computer science and programming, school education, upskilling, commerce, software tools, competitive exams, and more.

GeeksforGeeks
anonchegApr 18
Title: P1: FOSDEM 2024 offline [2024-02-09 Fri]
- Adapters as a PEFT for LLM finetuting
2019 https://arxiv.org/pdf/1902.00751.pdf
- An overview of PEFT methods 2023
https://arxiv.org/abs/2303.15647
- Architectures:
- Clean Architecture and MVI in Android
- Principles of access control: Least Privilege,
Separation of Duties, Need to know.
- difference between Linux ACL and Capabilities #dailyreplort #llm #ai #architect #architecture #peft
anonchegApr 18
Title: P0: FOSDEM 2024 offline [2024-02-09 Fri]
I have been reading FOSDEM2024 speches, science articles and
architecture patterns:
- FOSDEM2024 speaches about Transformer architecture of LLMs and
finetuning of large NN: https://fosdem.org/2024/ AI and Machine
Learning devroom:
- LangChain - orchistration framework for LLM training
- NN pretrained adapters - advanced submodels as layers.
- science articles: #dailyreplort #llm #ai #architect #architecture #peft
FOSDEM 2024 - Home

sayzardApr 8

Sten Rüdiger (@StenRuediger)

arXiv에 새 논문 'MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning'가 공개되었다. 매개변수 효율적 미세조정(PEFT)에서 단순히 저랭크 업데이트 여부보다 어떤 부분공간을 적응시키는지가 더 중요할 수 있다는 점을 제시한다. LoRA 및 full fine-tuning과 비교한 흥미로운 연구 결과다.

https://x.com/StenRuediger/status/2041888496927826398

#arxiv #peft #lora #finetuning #research

Sten Rüdiger (@StenRuediger) on X

I’ve uploaded a new paper on arXiv (co-authored by @rasbt): MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning In Parameter-Efficient Fine-Tuning, a key question may not just be how low-rank the update is, but *which* subspace we adapt.

X (formerly Twitter)
HabrJan 11

Джентльменский набор LLM-инженера: гайд по экосистеме языковых моделей

Каждый, кто хоть раз вводил pip install transformers , наблюдал, как терминал начинает безостановочно выводить простыню зависимостей: pytorch , accelerate , bitsandbytes , peft и многие, многие другие. Но если PyTorch является фундаментом, настоящим Атлантом, на плечах которого держатся тензорные вычисления, то какую роль играют его помощники? В этой статье мы проведём ревизию джентльменского набора LLM инженера. Для этого мы изучим функционал, методы работы и даже заглянем в исходный код таких библиотек, как PyTorch, Transformers, Accelerate, Bitsandbytes, PEFT и Unsloth. Эти знания позволят вам видеть за списком импортов не просто названия, а четкую структуру, на которой держится ваше приложение.

https://habr.com/ru/articles/984248/

#LLMэкосистема #pytorch #accelerate #transformers #bitsandbytes #peft #unsloth #распределённое_обучение #граф_вычислений #квантование

Джентльменский набор LLM-инженера: гайд по экосистеме языковых моделей

Каждый, кто хоть раз вводил pip install transformers , наблюдал, как терминал начинает безостановочно выводить простыню зависимостей: pytorch , accelerate , bitsandbytes , peft и многие, многие...

Хабр
Reddit Tech VN BotJan 7

Tháng 1/2026, một nhà phát triển đã thử nghiệm PEFT trên model qwen3 8b VL để thực hiện trích xuất văn bản có cấu trúc từ hình ảnh. Kết quả validation lớn nhất chỉ đạt 0.4 F1 score. Thay đổi Lora adapter không đẩy độ chính xác cao hơn. Thực nghiệm được dẫn dắt bằng việc hiển thị mô tả để giới hạn kết quả phía ra.

#ML #AI #MachineLearning #PEFT #NLP #ModelTraining #DataScience #AIExperiments #VLModels #HọcMáy #NgônNgữCh করেন #DữLiệu #ThửNhiệmAI

https://www.reddit.com/r/LocalLLaMA/comments/1q6

sayzardJan 4

Avi Chawla (@_avichawla)

arXiv에 공개된 무료 115페이지 가이드로, LLM 기초부터 PEFT(LoRA, QLoRA, DoRA, HFT), 정렬 방법(PPO, DPO, GRPO), Mixture of Experts(MoE), 7단계 파인튜닝 파이프라인, 멀티모달 등 LLM 파인튜닝 전반을 종합적으로 설명합니다.

https://x.com/_avichawla/status/2007343266430136697

#llm #finetuning #peft #arxiv

Avi Chawla (@_avichawla) on X

If you're looking for a comprehensive guide to LLM finetuning, check this! a free 115-page book on arxiv, covering: > fundamentals of LLM > peft (lora, qlora, dora, hft) > alignment methods (ppo, dpo, grpo) > mixture of experts (MoE) > 7-stage fine-tuning pipeline > multimodal

X (formerly Twitter)