Wasps in my porch column

I've been seeing wasps or yellowjackets crawling into my brick porch column thru a tiny hole. I'm guessing this is a problem. Anyone know a wasp removal specialist? And does this mean I have to repoint? Yipes.