Disney songs, sung by a neural network with a very short memory

I train neural networks, which are a type of computer program that try to learn to copy human things by looking at examples. By giving them just a list of names and no further instructions, I can get neural networks to invent names for paint colors, guinea pigs, craft beers, and more. The way I set up a neural network will control how well it does at its task. One important parameter I can control is the length of its memory. Let’s see what happens when I make its memory really, really short. Dory Image spiced up by Neural net-generated deep dream Here’s why memory is important for a neural network: As it’s generating the letters of new words, it’s looking at the letters it has already generated, trying to figure out what comes next. If it can look back over several words or even sentences, it will be more memory-intensive to run than a neural network that can only see a couple of words at a time, but it will also be better at learning long phrases. Here’s how this plays out when I train a neural network to write lyrics to Disney songs. I started with a list of 224 Disney songs from a lyrics dataset posted by GyanendraMishra on Kaggle, about 8200 lines in all. 50 character memory At first, the neural network had a modest memory length of 50 characters - that is, 50 letters. Its view of the dataset, and of the text it generates, looks something like the this: ow you the world Shining, shimmering, splendid Tel Humans who know the “I can show you the world” song from Aladdin can probably recognize this bit and fill in what comes next. The neural network can work with this too. It memorizes whole lines of songs, although it sometimes jumps between different songs - I’ve built a small degree of randomness into its choice of which letters to use next so it doesn’t always go with its top choice. A whole new world (A whole new world) A new fantastic point of view No one to tell us no Or where we will live Oh so surprise Every time to spare Let it snow, let it snow, let it snow, let it snow, let it snow, let it snow But I couldn’t do that it’s all around the sun With all the colors of the wind Can you paint with all the colors of the wind Can you paint with all the colors of the wind Can you paint with all the colors of the wind Can you paint with all the colors of the wind Can you paint with all the colors of the wi You’ll notice that it tends to get stuck on certain repetitive lines - that’s because those lines repeat in the original dataset, so the neural network learns that one “paint with all the colors of the wind” line is followed by a second. What it doesn’t know, however, is how many “colors of the wind” lines it’s already seen at a given point. Its view of these lines at any given time is something like: paint with all the colors of the wind Can you pain And it has no idea whether to move on to the next line, or to keep singing the same one. Usually, like Dory, it just keeps singing. 20 character memory With less than half the memory that it had before, the neural net is struggling a little harder. Its view of the training data looks something like: ow you the world Shi It still can memorize phrases (it can spell supercalifragilisticexpialidocious like a champ), but now gets lost and jumps between songs more easily. It tends to string together songs with some similarity, so Heigh-Ho (we dig dig dig dig dig dig dig in our mine the whole day through) gets mixed up with drip drip drop little April shower, and Mama Odie’s song from Princess and the Frog (you gotta dig). And once it starts singing Heigh-ho, Heigh-ho, it basically doesn’t stop.  A new fantastic point of view No one to tell us no Or where to go Or say in the wood Is the savage one is mine! We dig dig dig dig dig dig dig dig dig dig dig dig dig dig dig dig dig dig up with your pretty little song Greatest are seppation for you You paint with your pretty little song Gay little song right The savage one is way to sing with your pretty little song It’s gonna be a change Shake you are the sky is cloudy You gotta dig Choir: (dit) Mama Odie: Dig and dig and diggety-dig Heigh-ho, heigh-ho Heigh-ho, Heigh-ho Heigh-ho hum Heigh-ho, heigh-ho Heigh-ho, heigh-ho… 10 characters A memory this short is seriously unfair to give a neural network that’s trying to learn whole phrases. That line from Aladdin now looks something like: ow you the But it does its best. [Troy] If you say that must be true When you leave the world and the same place in the sky You can fly You can fly You can fly You can fly You can fly You can fly When you’re staring straight back at me? Why a true!! We’re breaking free. [Troy] Oh the most wonderful time of the year Winnie the Pooh Willy nilly silly old bear Winnie the Pooh, Winnie the Pooh Willy nilly wearing, when you’re welcome, she’s going the bare necessities of life will come to you I’ve been searching to find the bare necessities of the wind I’ve seement her dont and bring you can fly You can fly You can fly You can fly You can fly You can fly You can fly You can fly 5 characters Evil laughter. It’s honestly a surprise that a neural network with a 5-character memory can manage to function at all. ow yo That’s what the line from Aladdin looks like now. This what the neural network has to work with. This is what songs are to it now. It attempts “Under the Sea” or possibly “Part of Your World” from The Little Mermaid. Wish the sky What’s the street too I wonder the sea Under the sea Under the sea Under the sea she wants to dreamed to me Un the shore here where the world in by me about the river bend I wanna be so far I could be to place I’ll see it there to start And the shore shore Underer the sea When the sea Under the sea Under the sea Under the sea It is going to try to be king. Or a wolf. I’ve been in the sun It’s longing with a s This, dilly, dilly sofver who do it once a smiling A whole new world to the told me so If think with you And I wonderful from the wolf cry to the blue corn mountain This where who one souls So be king So good in a staring as the beginning with your cat I’ll never thing thing the start A whole new world with new the wolf cry to the tole on the otter that the same That down tomb told me so I’ll be king, dilly, dilly dain ine told me soy And I’ll never paint with new world with all the colors of the cores the sun I’ll be king Oh, so chue the wolf cry to the colors of the beginning with your to I wanna be king is the wolf cry to the storm to tell your corn who want the roll me soy I’ll never seen the blue here, and the chain as old as the sun start And I’m fine I’ll try I’ll try It has a seriously metal-sounding version of Beauty and the Beast. Something to the beast And and the beast The door When the beast The door When will bare necessities The beast When the most wonderful time to spart of the house your learn and black as the sun And the beast Tale as old as time Song as old as rhyme Beauty and be soul It’s the sun We’re only friends and the door When will sell you see her fleight and and bring about your heart don’t say a way the most wonderful time of the time of the shock The beast And a secret, winnie the Pooh Willy all the same I saw There’ll be you feel the Hokey the beast And then it rears up, gathers all its neurons, and flounces out an impressive: Birdriillagilisticexpialidocious Want more strange songs from this last neural network? It really likes to do metal versions of Beauty and the Beast. Sign up here to get them. Want to help with neural network experiments? For NaNoWriMo I’m crowdsourcing a dataset of novel first lines, after the neural network had trouble with a too-small dataset. Go to this form (no email necessary) and enter the first line of your novel, or your favorite novel, or of every novel on your bookshelf. You can enter as many as you like. At the end of the month, I’ll hopefully have enough sentences to give this another try.