The term “Button-down” is often used to describe a shirt generally. Originally the term described a shirt with a collar that buttoned down.  Buttoning the collar points to the shirt originated when English polo players wanted to stop their shirt collars from flapping in their faces.  

Post World War II, the button-down shirt, alongside cardovan shoes and the sack suit became essential elements of the American Ivy League dress.

According to Hoyle, button up your button down with a sport coat or a tweed, and feel free to skip the tie.

Fact Credit: Wikipedia and Nicholas Antongiavanni, The Suit, p. 102 

Photo Credit: Barbour.com