RT @NeurIPSConf
We are happy to announce the #NeurIPS2023 Creative AI Track. This track will invite ML researchers to showcase their work during the conference in the form of visual, language, musical, and performing arts. Proposals are due by *June 15*.
https://blog.neurips.cc/2023/05/02/call-for-neurips-creative-ai-track/
RT @DrJimFan
In the age of foundation models, designing the data pipeline is actually more important than tweaking the model architecture.
Here's a fun competition: curate an image-text dataset that yields high performance on downstream tasks, while keeping CLIP model & training *fixed*.
RT @du_yilun
Check out our work on recycling diffusion models!
We show how to combine different diffusion models together, to form new probability distributions which can solve a variety of new tasks– without any need for training!
Webpage:
https://shorturl.at/nrQ39
(1/7)
RT @harmdevries77
Surprised by the loss of LLaMA-7B still going down after 1 trillion tokens?
In a new blogpost, I explain why you shouldn't be and argue we haven't reached the limit of the recent trend of training smaller LLMs for longer:
https://www.harmdevries.com/post/model-size-vs-compute-overhead/
Analysis in 🧵👇
Very cool work on diffusion models for constrained data. The fact that the inverse of a reflected diffusion is another reflected diffusion seems magical🤯 this enables DDIM sampling with thresholding "baked in", which is awesome.
---
RT @aaron_lou
Presenting Reflected Diffusion Models w/ @StefanoErmon!
Diffusion models should reverse a SDE, but common hacks break this. We provide a fix through a general framework.
Arxiv: http://arxiv.or…
https://twitter.com/aaron_lou/status/1646528998594482176
RT @karpathy
Common Q: Can you train language model w diffusion?
Favorite A: read this post (the whole blog is excellent)
(Roughly speaking state of the art generative AI is either trained autoregressively or with diffusion. The underlying neural net usually a Transformer.) https://twitter.com/sedielem/status/1612459398005235716
“New blog post about diffusion language models: https://t.co/uMF2BZNCqZ Diffusion models have completely taken over generative modelling of perceptual signals -- why is autoregression still the name of the game for language modelling? And can we do anything about that?”
This is definitely a problem with AR waveform models, which produce very long sequences (~10^6 steps) and are prone to "going off the rails".
It's clearly not been much of an issue with language models so far, but I suppose it could be in the long run!
Diffusion it is, then?😁
---
RT @ylecun
I have claimed that Auto-Regressive LLMs are exponentially diverging diffusion processes.
Here is the argument:
Let e be the probability that any gene…
https://twitter.com/ylecun/status/1640122342570336267
“I have claimed that Auto-Regressive LLMs are exponentially diverging diffusion processes. Here is the argument: Let e be the probability that any generated token exits the tree of "correct" answers. Then the probability that an answer of length n is correct is (1-e)^n 1/”
Denoising Diffusion models have demonstrated their proficiency for generative sampling. However, generating good samples often requires many iterations. Consequently, techniques such as binary time-distillation (BTD) have been proposed to reduce the number of network calls for a fixed architecture. In this paper, we introduce TRAnsitive Closure Time-distillation (TRACT), a new method that extends BTD. For single step diffusion,TRACT improves FID by up to 2.4x on the same architecture, and achieves new single-step Denoising Diffusion Implicit Models (DDIM) state-of-the-art FID (7.4 for ImageNet64, 3.8 for CIFAR10). Finally we tease apart the method through extended ablations. The PyTorch implementation will be released soon.
