That is actually pretty much what happened.

The language they speak in Madagascar is a Polynesian language. Polynesia is north of Australia. Apparently the people in mainland Africa had not come to sail far enough to the sea in order to notice there’s a big-ass island to inhabit. Then they got visitors going “hi, wanna join us on our new place?”