| Github | https://github.com/japonophile/ |
| Blog | https://chopp.in/clj/ |
| Google Scholar | https://scholar.google.com/citations?user=GIEiSNEAAAAJ&hl=en |
| ORCID | https://orcid.org/0000-0003-4108-7395 |
Looking forward to seeing what they do
the tiny corp raised $5.1M
https://geohot.github.io/blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html
My space interests are the unimaginably far-future sort, well beyond the timeframes where there's reasonable expectation of intelligent life still operating with anything near the energy appetites and timescales of our own...universe has long gone dark and entropy is slowly unmaking it, etc.
And I love people and the optimism we insist on having even at this point, in a sea of blackened stars no longer even spitting out residual heat, that maybe by then there will be some form of life out there working to figure out how to spin off new universes, who will escape entropy. Sure! Why not.
It's funny, I had the same idea everyone else did as a kid that we'd break the light barrier someday and find our galaxy full of life, but now that I'm team "we're probably alone in this galaxy and limited to sublight," I mean your great tools for anything you can possibly build then become massive energy capture and unimaginable timeframes, concerted efforts of generations well beyond the number we've had so far. And still it won't be us or anyone even remotely like us who needs to solve these problems.
But we're still gonna have faith someone will be out there and they'll pull it off. Because we live in the first flush of the universe where time stretches on and on and on with limitless potential and unimaginable resources. Sure, at some point along the line we expect there won't even be matter left but hey, they've got time to get it all figured out.
I love us. I love how with no expectation of ever leaving our own solar system we can firmly believe others will escape the end of all things.
My current job just gave everyone a 4-15% raise for inflation and because they REALLY don't want to do anymore hiring after all of the pandemic turn over (they had a bunch of people retire early rather than deal with remote teaching. )
4% went to the highest paid the 15% to the lowest. They offered us a flat raise at first. But *we* negotiated for the progressive spread, including the most highly paid. I love working here everyone cares about each other.
Work calendars are the wrong way around.
They should assume people are busy all day, and only create space for people to book meetings in slots explicitly marked as being available.
The assumption of availability created by an empty space in the calendar is backwards and contributes to people being permanently distracted and dragged into endless meetings which look and sound like work, but which often are anything but.
"Can a large language model be conscious?
Can a #LLM think?"
Join us
Fri Jan 20, 4 PM ET
at the #LearningSalon
to listen to & discuss with prominent philosopher of mind
David Chalmers @davidchalmers42
John Krakauer, @melaniemitchell , Claire & I are excited to discuss with David!
https://crowdcast.io/e/learningsalon/58
* The #LearningSalon is an online interdisciplinary forum, with brief talks & 2-hour discussions on #neuroscience #AI #psychology #philosophy
So far, it has been extremely fun & rewarding to explore the fediverse, almost like peeking into the unknown and letting it suprise you by its diversity, randomness and somewhat unhinged collective wisdom.
I hope this feeling remains & grows as does the number of accounts I follow, and I wish it is experienced by more of us as we create (or better even, get adopted by) the mastodon community
“A wise old owl lived in an oak;
The more he saw the less he spoke;
The less he spoke the more he heard:
Why can't we all be like that bird?”
(Edward Hersey Richards)
#BuboBubo #EurasianEagleOwl
#birdsinlatin
#Birdphotography #Birds #Birdsofthefediverse
#BirdsofMastodon #Wildlife #Wildlifephotography #Nature
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss. Motivated by that construction, we show empirically that when training self-attention-only Transformers on simple regression tasks either the models learned by GD and Transformers show great similarity or, remarkably, the weights found by optimization match the construction. Thus we show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass. This allows us, at least in the domain of regression problems, to mechanistically understand the inner workings of in-context learning in optimized Transformers. Building on this insight, we furthermore identify how Transformers surpass the performance of plain gradient descent by learning an iterative curvature correction and learn linear models on deep data representations to solve non-linear regression tasks. Finally, we discuss intriguing parallels to a mechanism identified to be crucial for in-context learning termed induction-head (Olsson et al., 2022) and show how it could be understood as a specific case of in-context learning by gradient descent learning within Transformers. Code to reproduce the experiments can be found at https://github.com/google-research/self-organising-systems/tree/master/transformers_learn_icl_by_gd .
Max von Webel
pamela 
myrmepropagandist
Stuart Frisby
Ida Momennejad
Danai Riga
BirdsinLatin
Dan Stowell
Anjiolina Nakaya