Excellent article here from Frankie Boyle about free speak and offending people. I'm not a big fan of his normally but he talks a lot of sense here.
His last paragraph kind of says it all for me
"We don't live in a shared reality, we each live in a reality of our own, and causing upset is often the price of trying to reach each other. It's always easier to dismiss other people than to go through the awkward and time consuming process of understanding them. We have given taking offence a social status it doesn't deserve: it's not much more than a way of avoiding difficult conversations."