@pedromj No, that's not the main point of difference. That only affects vocabulary, and for the most part, the push for locally sourced words is a nineteenth-century one, when rapid advances of technology caused rapid inflow of new Greek/Roman loanwords into everyday languages.
Finnish is not an Indo-European language. Its ancestral languages came from somewhere in Siberia, possibly (although that part is fuzzier) Far East. As a result, it has a substantially different phonetics and vastly different grammar than most other European languages, yet has nesting sets similarities to various circumboreal languages possibly all the way to Yakutia, and (more disputably) Kamchatka. Hungarian is a (distant) relative to Finnish.
Basque is weird; scientists don't really know where it came from, but it might possibly be a remnant branch of some indigenous Stone Age language spoken in Europe since before the Indo-European farmers arrived. There's been some arguments made that it might be a (very distant) relative to some likewise weird Caucasian languages, but, well, Caucasus is a linguistically so diverse area that one should suspect cherry-picking until the arguments get really good.
As for cultural influence, Russian influence on Finnish culture was relatively small, and the net flow might possibly have gone the other way, sort of. Present-day Finnish language contains a very small number of Russian loanwords, and most of the Russian cultural artefacts in present-day Finland are of administrative origins rather than cultural per se. Russia governed Finland for two centuries, but treated it mostly as a peripheral colony.
But, OTOH, at the time the city of Sankt-Petersburg was built, the inhabitants of the area — and so, the people whom Pyotr Pervyy hired in large numbers to build his shiny new city and to live in it — used to be Fenno-Ugric people relatively culturally and ethnically similar to Finnish people of the same time. From modern genetic studies, we know that there has been a substantially shared genetic background to people living in present-day Finland, Estonia, and Leningrad Oblast (the Russian region surrounding the city of Sankt-Petersburg: the city used to be called Leningrad, but got named back, but the oblast around it was not named back). This has likely been a source to some noted cultural differences between people of Leningrad/Sankt-Petersburg and people of Moscow, and this, in turn, has contributed to some speculation that if Russia were to break up, Leningrad Oblast plus Karelia might find a somewhat easier time to joining EU as an independent republic than, say, the enclave of Kaliningrad / Königsberg, which was entirely repopulated in the late 1940s.
For some added complications, the genetic heritages and the linguistic heritages in this part of the world do not strongly mesh. The Sami and Finnish languages, for example, are close relatives, but there's a marked genetic difference between the Sami and the Finnish people, and the haplogroup distribution patterns can plausibly be interpreted in a number of different ways. The first human inhabitation of Finland is about 9000 years old or so, as in, humans showed up very shortly after the Ice Age receded, and a few tidbits are known about the earliest human cultures, from dig sites like the ones at Kunda, but nobody in the present Finland corresponds directly to the 'original indigenous Finns', which would probably be a clade of what we nowadays call the 'Early Scandinavian Hunter-Gatherers, themselves a genetic mixture of different waves of early human migration into Northern Europe. At the same time, the Sami people seem to have the highest direct descendancy in Finland of the ESHG population. But there have been several subsequent waves of migration into this corner of the world between then and the modern times, and genetic studies show at least two or maybe three major admixture events, including one that likely correlates with adopting agriculture without the Proto-Indo-European culture; these two seem to have entered most of the more southernly parts of Europe relatively close together. The ancestor of the modern Finnish language, however, probably arrived to Finland only somewhen between 2000–3000 years ago, by which time humans had already been living there for millennia, both nomadically and in settlements, and had been presumedly speaking some languages, of which we know very little. (The current consensus threshold for 'Behaviourally Modern Humans' is ca. 50_000 years ago. It is believed that humans have been "definitely" speaking, with languages and syntax and recursive grammars and semantics, at least since then. But the genes that likely made it possible for human brains to process speech like modern humans do apper to have become common much earlier than that, and there's some fierce current research into what the 'early human language' or 'proto-language' might have looked like. At the same time, the earliest we know about human languages with some definiteness is about 5000 years old, and even the very wildest scholarly claims of historic reconstruction — the very bleeding edge between 'speculative fringe science' and 'raving pseudoscience' — date to no earlier than about 15_000 years.)
And, because that's not messy enough, some Original Nazi Racists(tm) were so charmed by "Nordics" that they made extra bullshit up about the Finnish people. Nazi pseudoscientists like simple models that a person stupid enough to be racist would understand; one of the recurring themes of these simplifications was mixing up Swedes, who are Germanic people speaking a Germanic language only about 1500 years apart from modern-day Germans, and having cultural artefacts that over-obsessed fanboys of the Nibelungenlied would easily recognise, and Finns, who are a somewhat genetically distinct clade, speak (and spoke) a very different language from Swedish or German, and may have had ancient deities that had to be renamed so they would make any sense to somebody like Guido von Fucking List.
Even the whole issue of whether ancient Finns had deities is genuinely questioned; the ancient Finnish folklore likely had tales of heroes, and some of the alleged "Ancient Finnic Deities" may have been those ancient mythical heroes reimagined by Christian zealots looking for "pagan gods", as though they had been those pagan gods. I mentioned how Finland kicked the Pope out centuries ago (well, Sweden did it, actually; Finland didn't have political self-determination at the time), but when Catholicism first arrived to Finland, there's some evidence to suggest that the newly popular patron saints may have been deliberately mixed with, and supplanted upon, previously popular animist fae or guardian spirits, whose lore might have somewhat resembled the many kami of present-day Shintô.