Consider me black
Several times in the past few days, people I've been talking to have said things or expressed opinions that I would consider racist. When I pointed out that I thought their comments unfair, they tried to justify themselves, always beginning "it's not racist, it's just an observation" or whatever. So what's the definition of racist? Here's mine: say, for the purposes of example, that we're discussing something to do with black people. If a black person was in the room, would you say what you were about to say? Or would you rephrase it a little? If so, then your comment was probably racist, or amiguous enough to be offensive anyway. And just because I'm not black, or Jewish, or Pakistani, or Indian doesn't mean it doesn't offend me when you make unfair generalisations about those groups. And don't expect me not to "make an issue of it" just because there's nobody nearby who you think should be offended by it: by staying silent, I would be implicitly accepting your world-view, with which I disagree strongly, and I have no wish to bolster your ghastly assumptions.
Being a member of an "invisible" minority, I know what it feels like to be the butt of one of those jokes that people make when they think it's okay, because there's nobody around who will be offended. And yes, offensive is relative, and there are very seldom any amish in the room. But there is a difference between risqué jokes and bigoted statements that aren't intended to be amusing: anybody can make a joke, but you shouldn't make statements unless you're prepared to stand by those statements in front of a large crowd of the group you are denigrating. Preferably, while said crowd is holding many sharp implements. Just to, you know, focus your mind.