Director Jason Byrne brings his avant-garde rehearsal strategy to a dark comedy about paternity, requiring players ‘to be extremely vulnerable,’ Gayle MacDonald reports