This is something that the nimblest standup comedians learn, over time, to handle gracefully. They'll go between prepared material, then they'll respond to what's happening in the room and weave it back into the prepared material and so on.
One of the things I've always loved about New York is there is so much precedent for ornament on industrial buildings.