Is it rude to double-look at someone passing by?

I was at the mall and I saw a guy pass by and I looked at him two times. I wasn’t checking him out or anything, it was just brought up to my attention.

And to clarify, he wasn’t doing anything wierd or bad, he just passed by me and I forgot to mention he gave me the look