What makes demons really terrifying in a movie?

The actor/actress that plays the demon, a well written part, and the makeup/costume artists doing a good job with the makeup.

Focus on what differentiates demons from other monsters.

The demons that look, act, and talk exactly like us are the demons that terrify me the most.

thank god it was an angel that told me to start drinking beer . . .

I second @Blondesjon – it’s that they resemble us, and also that they can invade and possess us.

Bob is the best: “Catch you with my death bag.”

Hooves.

I find the demons that aren’t visible the scariest, but they have to do a very good job of presenting it.

I think of demons as the embodiment, or a particular aspect, of evil. There’s no reasoning with it, no appealing to its “humanity” – despite appearances and actions they are not human and we can not truly understand or identify with them, there can be no dissuading or sating its desires. It can take human form (among others), perhaps it can be driven off but it doesn’t die, it feeds off us, it ferments its evil in us – whispering over and over – twisting us until our desires can only be fulfilled by become it. Above all, a powerful demon is unique – as much as any character, it has its own persona. Nothing positive should come from the end of a demons machinations, a victory is pyrrhic at best and bleakly temporary, leaving desolation, death, pain, and suffering in its wake.

Check out Hellriaser, Devil’s Advocate, Demons, Prince of Darkness, Pumpkin Head, Rosemary’s Baby, Insidious they don’t all use “demons” exactly but use interesting ways of portraying what could easily be a demon. Even The Thing particularly when trying to determine who’s human and who isn’t, that sense of tension, distrust, and dread.

In general, horror is about suggestion and atmosphere; nothing is more terrifying than what the viewer conjures in their own imagination. Make it believable, at least enough to make the audience suspend their disbelief, think “what if”, and see themselves as vulnerable. It could be anyone, anywhere just plant the seed, provide the setting, and allow them scare themselves silly. One of the things a demon allows for is straddling the border of human and monster. Some find it all the more horrifying to discover it was a person that was the root of the evil others an inhuman monster – a demon can be portrayed either way. Hinting at its otherworldliness, providing just the suggestion that it may not be human always making the viewer wonder. Or by exposing it’s unrelenting monstrousness in paralyzing, heart stopping, fear as it consumes its prey.

Sorry for making that sort of a “nebulous” answer but I think all that goes to what makes a demon terrifying – it’s not always the same thing – it really depends on how its being used and portrayed.

The people in the movie never seem to believe in demons. Or even consider the possibility that they’re dealing with something outside of their world view. The nipple head in Paranormal Activity, even after the evidence that he was dealing with something out of his range of understanding, insisted that he could “handle it”.
I feel like the fact that they don’t know what they’re dealing with is part of it.
There is the idea that there is no real motivation for the demon, it seems to be evil for no reason, just for the sake of evil.
Also, one doesn’t know what it is capable of, what the limitations are. Does it have unlimited power? Can it only manipulate objects? Does it gain in strength? Is it subject to rules or limitations?
Not knowing is a huge fear factor.

The control they can have over you, and your entire fate, even after death. They can also know you, so that gives them a great upper hand against you. In a lot of movies there are very specific and effective ways to destroy, or at least push back a demon, but if you don’t know what it is, even God Himself can’t do anything for you if the demon drags your soul to Hell.
Demons are supposed to be pretty smart…but I hear tell even the strongest demons would rather have nothing to do with ancient vampires…of course, those could be considered demons too…

That their breath smells like a wildebeest’s nutsack.

Demons are scariest when they are just hinted at. The tension and the fear can build remorselessly. The more we see of them the less scary they become however good the special effects are.

@ucme Apparently, Dracula’s breath smells like a charnel house. That makes me wonder how exactly did vampires get the whole romantic and sexy thing…I wouldn’t want to make out with some dude who smells like old bones.

@Symbeline Exactly, maybe he used a whole thing of tic-tacs for that minty fresh breathed feeling.

Gotta do somethin bout that eternal death smell, ya know.

I was watching the Rite yesterday. Didn’t finish it yet, but man did it scare the shit out of me. I think what makes demons/devils (TYPING THIS IS SO SCARY) is that the idea of them actually exists. Or, like, (if you believe in this) they exist. Before ‘The Rite’ started, they actually wrote ‘Inspired by true stories…’ You can imagine how much I peed my pants.
Then there comes the crack-noises and the creepy stiff movements, and the blank eyes. Oh gosh the blank eyes. Oh and that harsh raspy voice, and the names; ‘Lucifer, Judas’ etc.
Sometimes religious references tend to be scary, especially when paired with Latin songs and whispers.

Possession is real I think. So watching movies about it is frightening.