Is "dropping in" on friends or family always rude?

If you happen to know for a fact that they don’t mind, is it still rude?

I have no problem with people dropping in on me. My house is rarely spotless, but it’s always presentable. If I’m busy I say I’m busy, but they’re free to talk or whatever, while I continue doing whatever.
My kids, of course, just walk in without even knocking. If, for some reason, we didn’t want them to we just lock the door for a while!

When my friends and I were stay at home moms we dropped in on each other all the time.

There was a time, before phones, where that’s all people ever did, was just go by someone’s house.

Generally speaking, do you mind drop ins?

I have a story to tell of the one horror story involving a drop in that I experienced, but I’ll put it in as an answer.