Why do people resist taking medication?

People seem to have an almost-universal distaste for taking medication long-term, especially psychiatric medications. Why? Would it make a difference to you if you only took the medication for a few weeks, and the effects were permanent? Are naturopathic medications less offensive, even taken long-term? Why are are medications to treat mental illness more offensive than those to treat physical illness? (What’s the difference, really?)

(And there’s no judgment intended in the tone of this question—just curious for some perspectives.)