On screen it's a passing sentence, in print it is a cluster of consonant (e.g. Mxyzptlk from DC comics).
This is done to make the alien appear more alien, but it is a really lazy gimmick. On Star Trek the unpronounceability is used to justify why the Andorians speak Andorian, live on the moon Andoria (with capital city of Andor) in the Andorian sector. So a lazy gimmick to cover for lazy scriptwriting.
The Wikipedia article Alien language lists extensively the presence of alien languages in cinema. Here I want to reflect on the unpronounceability cliché. Namely, humanoid aliens that sound like humans, but have unpronounceable words, which they apologetically whisk away.
In fact, given that the voice of most singers on MTV are subjected to autotune, one would expect that the alien voices would be altered. There are several examples of really alien alien sounds, but few of subtle alterations to make speech not quite human. In Star Trek TNG:11001001 the Bynars have a high-pitch voice which is only matched in lameness by the dateness of the episode, while apparently the Nausicaan on TNG:Tapestry had an altered voice, but not on Enterprise episodes. The Nausicaan accent is very subtle in TNG, a bit echoy, but that's all. It is a shame that one is overdone and the other underdone. I could not find any other examples from Star Trek.
At first one might argue that it is a matter of technology. In fact, most apps and programs that alter voice simply change the whole spectrum (speed and pitch). Sounds are not simply frequency amplitude and its change over time (timbre). They have fundamentals and harmonics, and are not a single frequency but a pattern across frequency. The difference between various vowels can be broken down into formants.
There are pages and pages dedicated to formants and formant synthesis. So it is not technologically impossible to do so. So it is a shame that it not applied to better alter voices.
So what could a scientifically valid impossible sound be? Human sounds are a combination of various factors, such manner and place of articulation, voicing and so on. The IPA chart (consonants pictured below) covers the main forms of articulation. The link between acoustics and phonetics is really interesting (e.g. link), but is rather complex as it does not have a simple relationship.
In the IPA chart there are greyed out areas for impossible sounds, but most of these are impossible because they are nonsensical, not because of human physiological constrains. However, there are many sounds that humans can make that are not used in any/many languages, for example linguolabial, bidental and "super-retroflex" consonant. So there are many non-English sounds that can be made without cheating electronically.
However, there is catch called diaphonic identification that reduces the novelty. Klingon is a constructed language that features /x/ (H), /q/ (q), /q͡χ/ (Q) and /t͡ɬ/ (tlh) sounds which are not found in English, making it sound very harsh. Dothraki features /x/ and dental t d (t and d pronounced with the toungue like for th —þ and ð) and sounds harsh, but without the hard to pronounce sounds. That is because the /q͡χ/ sound is hard to distinguish from /x/ (arabic kh, ch in loch) to an untrained ear. This is a phenomenon called diaphonic identification. In Mandarin the Pinyin b, d and g (and a few more) are unaspired p, t and k, but as they sounds similar to the voiced equivalents they are mistaken by English speakers for b, d and g (which are not aspired). Consequently, there are diminishing returns when it comes to alien sounds as they will sound odd, but not as extreme as hoped.
So in conclusion, the unpronounceable alien cliché is just a piece of lazy writing that could be enriched by phonology, expect that it requires extra effort and the results are not as substantial as the effort required due to mistaking alien sounds for native ones.
❧ A wee parenthetical tangent…
The printed form, namely a strange clusters of consonants, is different matter and worth only a cursory note or two. Firstly, some consonants can act as vowels (syllabic consonants). English has syllabic consonants, such as l in animal or n in even —the last vowel isn't pronounced. Norse had syllabic r as encountered in several names (e.g. Seiðr). So a name like Mxyzptlk could be pronounced faithfully assuming it was known which letters were syllabic (M, y, tl is my guess) and what the letters meant (in this case x and y). Cthulhu is an example of unpronounceable due to spelling issues. Lovecraft wrote it is an approximation and the h is to mark a "guttural tone", which is very ambiguous, except that it hints that lh is probably not a Welsh double l (/ɬ/) or that the c is an Irish c (/c/) or a c pronounced like an s. Most people pronounce it /k̩'θu:lu:/ as a result.
