jordanCould an #LLM with the same encoder/token scheme as a #diffusers image model be used to increase the diffuser model's understanding of concepts? The LLM is going to have greater understanding as a whole than the limited connections that the diffuser is going to pick up on if trained only from captions.
Tim Kellogg@wagesj45 there’s alternative model architectures that are built to do this https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/
GambleWean© Wean Off Gambling@wagesj45
Yes, incorporating a Language-Conditioned Diffusion (LCD) model with a Large Language Model (LLM) sharing the same encoder/token scheme could potentially enhance the diffuser model's comprehension of concepts. The LLM's broader understanding can supplement the diffuser model's training, allowing it to capture more nuanced connections between images and captions, thus improving its overall performance and comprehension.
Yes, incorporating a Language-Conditioned Diffusion (LCD) model with a Large Language Model (LLM) sharing the same encoder/token scheme could potentially enhance the diffuser model's comprehension of concepts. The LLM's broader understanding can supplement the diffuser model's training, allowing it to capture more nuanced connections between images and captions, thus improving its overall performance and comprehension.