I think that the problem you have here, @dopeguru, is that you tend to take people’s words literally. (I do this, too, so I often have the same problem – but on the other hand I’m also much older and have a lot more experience, and I grew up in this language and culture, so I understand that language is not always to be taken as literally as it might appear. But I still miss the hidden meanings sometimes, even so.)
The problem that you face is that you can’t really tell without knowing the person, knowing his intent and his own familiarity with language and idiom, custom and culture, to know with any certainty whether “Come up to my apartment for a drink” means exactly and only that or… “Come up to my apartment and let’s screw our brains out.” And if you can’t pick up the subtle clues to his intent in the social (public) interaction between the two of you, then you’re at a double disadvantage.
For example, and to use another current idiom, how about “Come on over and let’s Netflix and chill.” The first time I heard it (not directed at me – I wish!) I thought it meant to watch movies on Neflix (literally) and “to relax” as the idiomatic use of “chill” means. But that’s not it at all, or at least, that’s not how it usually relates these days in a boy-girl and I’ve-got-the-place-all-to-myself context. In that case, it generally means the same as “Come on up for a drink.”
If you don’t know (or trust) the one making the invitation, and if you have the least doubt about what the invitation means behind the actual words used, then it’s probably best to decline the invitation. And if Bill Cosby asks you to “come up for a drink” then it’s probably best to both decline and run.
The panel members you mention know this from experience and knowledge of the culture, which is why they make the inference they do: that invitation is a precursor to sex. It’s not inevitable, and it’s not mandatory, and rape is never excused – ever – but there’s a general understanding that sexual intimacy will be on the agenda, and if that woman accepts that invitation but then doesn’t clearly say – and maybe repeatedly, and with no ambiguity, “No, I don’t want this,” then an aggressive man will generally continue doing whatever it is that he’s doing. “Because she never said ‘no’.”
(I presume that this is in regard to the ongoing Aziz Ansari issue, which I have barely heard about, but seems to be “the talk” these days in this regard. I don’t know the details of that story beyond the invitation to “come on up for a drink” and a later accusation of sexual assault, apparently based on just this kind of misunderstanding – or rape – as the case may be.)