Mastodawn

Will Reed Apr 25, 2023

Grace Lindsay Apr 25, 2023

"Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry....Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods"
#ML #MachineLearning #AI #DataScience
https://arxiv.org/abs/2304.12210

A Cookbook of Self-Supervised Learning

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

arXiv.org

Will Reed Feb 6, 2023

arXiv Comp. Linguistics📚Feb 6, 2023

📝 Multimodal Chain-of-Thought Reasoning in Language Models 📚👾🔭

"We incorporate vision features in both stages of rationale generation and answer inference to improve chain-of-thought reasoning in a decoupled training framework and demonstrate its effectiveness on science question answering." [gal30b+] 🤖 #CL #AI #CV

⚙️ https://github.com/amazon-science/mm-cot
🔗 https://arxiv.org/abs/2302.00923v1 #arxiv

GitHub - amazon-science/mm-cot: Official implementation for "Multimodal Chain-of-Thought Reasoning in Language Models" (stay tuned and more will be updated)

Official implementation for "Multimodal Chain-of-Thought Reasoning in Language Models" (stay tuned and more will be updated) - amazon-science/mm-cot

GitHub

Will Reed Jan 31, 2023

Nicholas Thompson Jan 30, 2023

"It’s only natural to peer into the dark unknown and ask what could possibly go wrong. It’s equally necessary—and more essentially human—to do so and envision what could possibly go right." https://www.theatlantic.com/ideas/archive/2023/01/chatgpt-ai-technology-techo-humanism-reid-hoffman/672872/

Technology Makes Us More Human

When techno-optimists use ChatGPT, they see <em>Star Trek</em>: a future in which opportunities for personal fulfillment are as large as the universe itself.

The Atlantic

Will Reed Jan 31, 2023

Aran Komatsuzaki Jan 31, 2023

Looped Transformers as Programmable Computers

Presents a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop.

https://arxiv.org/abs/2301.13196

Looped Transformers as Programmable Computers

We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including embedding edit operations, non-linear functions, function calls, program counters, and conditional branches. Using these building blocks, we emulate a small instruction-set computer. This allows us to map iterative algorithms to programs that can be executed by a looped, 13-layer transformer. We show how this transformer, instructed by its input, can emulate a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Our work highlights the versatility of the attention mechanism, and demonstrates that even shallow transformers can execute full-fledged, general-purpose programs.

arXiv.org

Will Reed Jan 30, 2023

Gabriel Arana Jan 30, 2023

Omg Population III stars! (Imma space nerd.) These had only hydrogen cuz there was no metal in the universe then.☝️ https://www.quantamagazine.org/astronomers-say-they-have-spotted-the-universes-first-stars-20230130/ 👁❤️ @QuantaMagazine @7homaslin

Astronomers Say They Have Spotted the Universe’s First Stars | Quanta Magazine

Theory has it that “Population III” stars brought light to the cosmos. The James Webb Space Telescope may have just glimpsed them.

Quanta Magazine

Will Reed Jan 30, 2023

arXiv Computer Vision🔭Jan 30, 2023

📝 Rethinking 1x1 Convolutions: Can We Train CNNs with Frozen Random Filters? 🔭👾🧠

"Works by rethinking pointwise convolutions as a linear combination of fixed random filters and adding a novel weight sharing mechanism that allows sharing of a single weight tensor between all spatial convolution layers to massively reduce the number of weights." [gal30b+] 🤖 #CV #AI #LG

🔗 https://arxiv.org/abs/2301.11360v1 #arxiv

The Power of Linear Combinations: Learning with Random Convolutions

Following the traditional paradigm of convolutional neural networks (CNNs), modern CNNs manage to keep pace with more recent, for example transformer-based, models by not only increasing model depth and width but also the kernel size. This results in large amounts of learnable model parameters that need to be handled during training. While following the convolutional paradigm with the according spatial inductive bias, we question the significance of \emph{learned} convolution filters. In fact, our findings demonstrate that many contemporary CNN architectures can achieve high test accuracies without ever updating randomly initialized (spatial) convolution filters. Instead, simple linear combinations (implemented through efficient $1\times 1$ convolutions) suffice to effectively recombine even random filters into expressive network operators. Furthermore, these combinations of random filters can implicitly regularize the resulting operations, mitigating overfitting and enhancing overall performance and robustness. Conversely, retaining the ability to learn filter updates can impair network performance. Lastly, although we only observe relatively small gains from learning $3\times 3$ convolutions, the learning gains increase proportionally with kernel size, owing to the non-idealities of the independent and identically distributed (\textit{i.i.d.}) nature of default initialization techniques.

arXiv.org

Will Reed Jan 30, 2023

arXiv Artificial Intelligence👾Jan 30, 2023

📝 A Comparison of Tiny-Nerf Versus Spatial Representations for 3d Reconstruction 👾🔭🦾

"Neural rendering is based on a continuous representation of the environment, where the 3D scene is encoded into a neural network that can be sampled at any resolution to synthesize the scene." [gal30b+] 🤖 #AI #CV #RO

🔗 https://arxiv.org/abs/2301.11522v1 #arxiv

A Comparison of Tiny-nerf versus Spatial Representations for 3d Reconstruction

Neural rendering has emerged as a powerful paradigm for synthesizing images, offering many benefits over classical rendering by using neural networks to reconstruct surfaces, represent shapes, and synthesize novel views, either for objects or scenes. In this neural rendering, the environment is encoded into a neural network. We believe that these new representations can be used to codify the scene for a mobile robot. Therefore, in this work, we perform a comparison between a trending neural rendering, called tiny-NeRF, and other volume representations that are commonly used as maps in robotics, such as voxel maps, point clouds, and triangular meshes. The target is to know the advantages and disadvantages of neural representations in the robotics context. The comparison is made in terms of spatial complexity and processing time to obtain a model. Experiments show that tiny-NeRF requires three times less memory space compared to other representations. In terms of processing time, tiny-NeRF takes about six times more to compute the model.

arXiv.org

Will Reed Jan 29, 2023

PartHairCut Jan 29, 2023

I used midjourney and now I feel a bit dirty.

I finished a new #song and am thinking about the thumbnail image for Bandcamp, soundcloud etc. and a couple of people have mentioned they used #midjourney #ai

Even if I think I don’t agree with something, I believe I should make the effort to understand it before coming to a more educated decision, so I gave it a go.

To be clear; I was never going to pay anyone for this piece of art. I don’t make any money from the #Music, I do it for me.

Definitely the results are good enough to use, and maybe I will but… I’m not sure it feels right. When I #draw the #art myself I learn a little each time. Like the music, there is a journey and a satisfaction in bettering myself. At the same time, a thumbnail for a song is not exactly the most groundbreaking or meaningful thing in the world.

It’s a visual index for audible things.

I may have saved myself time, but have I missed out in the process? Have I indeed up with something that looks better but I value less?

Does anyone other than myself care if I drew it, or a machine did?

Will Reed Jan 29, 2023

arXiv Artificial Intelligence👾Jan 29, 2023

📝 Reflective Artificial Intelligence 👾

"Reflective AI can be viewed as a form of self-awareness in AI agents, and is an essential component to the goal of Artificial General Intelligence (AGI)." [gal30b+] 🤖 #AI

🔗 https://arxiv.org/abs/2301.10823v1 #arxiv

Reflective Artificial Intelligence

Artificial Intelligence (AI) is about making computers that do the sorts of things that minds can do, and as we progress towards this goal, we tend to increasingly delegate human tasks to machines. However, AI systems usually do these tasks with an unusual imbalance of insight and understanding: new, deeper insights are present, yet many important qualities that a human mind would have previously brought to the activity are utterly absent. Therefore, it is crucial to ask which features of minds have we replicated, which are missing, and if that matters. One core feature that humans bring to tasks, when dealing with the ambiguity, emergent knowledge, and social context presented by the world, is reflection. Yet this capability is utterly missing from current mainstream AI. In this paper we ask what reflective AI might look like. Then, drawing on notions of reflection in complex systems, cognitive science, and agents, we sketch an architecture for reflective AI agents, and highlight ways forward.

arXiv.org

Will Reed Jan 29, 2023