The holiday has become an unofficial signal to ditch the white clothes filling our summertime wardrobe.