Down relies on air between its clusters to retain warmth. When the face fabric of your garment becomes dirty, body oils and dirt attract water which can then transfer into the down, causing it to clump. Without the pockets of air between the down clusters, the garment loses its insulation properties.
To maintain performance, wash your garment when it shows signs of dirt or sweat, or when the down is starting to lose loft or clump. It is best to use a cleaning agent made specifically for technical down products.