The new Sam Wanamaker Playhouse is beautiful, but this production fails to capture the chilling thrills of The Duchess of Malfi, says Charles Spencer