Why do people look at me like they're mad when me and my friends speak spanish in public?

I can be out anywhere with my latino friends or my family and when we speak spanish there is always somebody there to give us a dirty look.Its like I’m speaking a forbidden language or something.It almost makes me want to go gangster on them.I am I the only one that has experienced this?