Why is being blue the color of saddness?

Maybe because when someone ‘sings the blues’ it’s usually a sad song?

Blame it on the Navy… Back in the good old days, if a ship lost her captain or any of the officers during the voyage, she would fly blue flags and have a blue band painted along her entire hull when returning to her home port. It was a signal for any family to come check on their loved ones.

This is an interesting graphic.
https://www.npr.org/sections/health-shots/2013/12/30/258313116/mapping-emotions-on-the-body-love-makes-us-warm-all-over

Blue is calming for me. I love playing Blues Guitar. This makes sense to me as I am an emo.
To answer your question however, Blue can be felt as cold. It is one of several Sad colors like Black and Gray.