Do large buildings have less trouble with high winds?

I recently saw a TV series on skyscrapers. It was really interesting, they covered different skyscrapers being built around the world, different materials being used, all the consideration that need to be taken to account.

They mentioned only briefly the idea of mega large buildings being built where people could work, play, shop, and live; basically, an indoor city. I read about this concept in a sci-fI novel when I was a teenager. People would be commuting more vertically than horizontally.

The TV series I recently saw mentioned aerodynamics around the buildings, and my question is, if the building was very large, let’s say 3 city blocks square, then could it be built taller than a typical building, or is the opposite true? Which building is more easily engineered not to sway at high heights?