Why aren't Irish jokes considered "racist"?

Yes, I suppose you could point out that people who make such jokes are being racist, but you know what? Many Irish aren't Catholics and a fair few aren't white either. And quite frankly we are concerned with far more important things and most of us are done being the victim, so you can call us whatever you like because we don't really care anymore.