Just for some background info and timelines:
-
Yamashita Cave Man (~32,000 BCE; Naha). No assumed origin.
-
Pinza-Abu Cave Man (~26,000 BCE; Miyakojima). Assumed from Malaysia.
-
Minatogawa Man (~21,000 BCE; Naha). Assumed Australo-Melanesian.
-
Shiraho Saonetabaru Cave People (~20,000 BCE). Two skeletal remains identified as belonging to haplogroup M7a, which is attributed to both East Asia and Southeast Asia.
-
Jōmon People (~14,000 BCE - present; pan-Japan). Attributed to multiple groups living throughout Japan, but primarily modern Ainu people. Japanese people are said to have between 10-20% Jōmon ancestry, while Ryukyuan are said to have somewhere around 28% Jōmon ancestry. They are said to have come from East Asia or Continental Asia.
-
Yayoi People (~1,000 BCE - present; pan-Japan). Attributed to current Japanese and Ryukyuan people. Said to have come from the Korean Peninsula.
-
Hayato and Kumaso People (~400-800 CE; Kyushu). Evidence points to them being Yayoi people or very closely related.
It's pretty well known that modern Japonic peoples are descended from the Yayoi People and that they largely displaced the Jōmon People. Despite this and despite some mixing, we don't actually have much evidence of borrowings, besides a few words found in modern Japanese that appear to have been borrowed from Ainu (such as トナカイ tonakai "reindeer", ラッコ rakko "sea otter" and a few others).
In Kyushu specifically, there are a lot of myths surrounding the Hayato and Kumaso people (e.g. that they come from Oceania or Indonesia), but evidence suggests they were a Yayoi people (or closely related). It's believed based on historical accounts that the Hayato likely spoke a Japonic language, but slightly different from the local people. (see pp 80-84 of this document) If that's the case, then it's doubtful that we would able to identify any loans from this language.
In the Ryukyus, there's evidence of different waves of people living throughout the islands over time. Despite this, and even despite the Jōmon genetic ancestry being greater in the Ryukyus than in mainland Japan, no one has been able to identify loans within the Ryukyuan languages coming from languages other than Japanese, Chinese and more recently European languages. Maybe the one exception is the word for "alocasia" in Miyako languages, which appears to be borrowed from an Austronesian (possibly Philippine) language (see wiki).
With that said, keep in mind that linguistic studies on Ryukyuan and Kyushu varieties is still in its infancy and we might find evidence of other loans over time. But, as you stated, it appears that most of these other peoples were displaced, assimilated or killed, and so it's unlikely that we'll find much evidence at all.
And for a last food-for-thought: A lot of linguists are still trying to pinpoint the ultimate origin of the Japonic languages. Some of these proposals try to connect them to Austronesian or Austroasiatic languages. If you're interested in that, I'd suggest going through the sources in this Wiki article section.
