Why do so many people here act as if men walk around topless everywhere?

This is usually brought up in conjunction with the fact women aren't usually allowed to walk around topless. The prevailing idea seems to be men can go wherever they want topless. Who sees this kind of thing? The only place I ever see it happen in public is at the beach, the lake, the swimming pool...basically anything revolving around swimming. You don't exactly see lots of shirtless men walking up and down Main Street do you? Or is this really commonplace everywhere but here?