As Andrew Lloyd Webber prepares for his fifth television collaboration to find a musical theatre star, Matthew Hemley asks how the industry's perception of the genre has evolved over the years