The actor insists that he has finished with musicals and has taken on his most difficult role in years