Title: P4: FOSDEM 2024 offline [2024-02-09 Fri]
trainable parameters. Low-rank subspace finetuning. Part of the model's input embeddings is fine-tuned via gradient descent.
- Fastfood transform to reparametrize the update to NN params.
- LoRa - simple low-rank matrix decomposition(or Kronecker product decomposition) to parametrize the weight
update
😶 #dailyreplort #llm #ai #architect #architecture #peft

Title: P2: P3: FOSDEM 2024 offline [2024-02-09 Fri]
- soft prompts - consists of a task description accompanied by a few in-context examples
- *selective* - fine-tuning only selected layers/biases/rows
- *reparametrization-based* (kind of additive) - leverage low-rank representations to minim the number of #dailyreplort #llm #ai #architect #architecture #peft

Title: P1: P3: FOSDEM 2024 offline [2024-02-09 Fri]
- prompt tuning or modifications - hard or soft or prefix tuning (as LLaMa adapter) - appends a tensor to
the embedded inputs of a pretrained LLM #dailyreplort #llm #ai #architect #architecture #peft

Title: P1: FOSDEM 2024 offline [2024-02-09 Fri]
- Adapters as a PEFT for LLM finetuting
2019 https://arxiv.org/pdf/1902.00751.pdf
- An overview of PEFT methods 2023
https://arxiv.org/abs/2303.15647
- Architectures:
- Clean Architecture and MVI in Android
- Principles of access control: Least Privilege,
Separation of Duties, Need to know.
- difference between Linux ACL and Capabilities #dailyreplort #llm #ai #architect #architecture #peft

Title: P0: FOSDEM 2024 offline [2024-02-09 Fri]
I have been reading FOSDEM2024 speches, science articles and
architecture patterns:
- FOSDEM2024 speaches about Transformer architecture of LLMs and
finetuning of large NN: https://fosdem.org/2024/ AI and Machine
Learning devroom:
- LangChain - orchistration framework for LLM training
- NN pretrained adapters - advanced submodels as layers.
- science articles: #dailyreplort #llm #ai #architect #architecture #peft