baby bathwater

a shortened version of "don't throw the baby out with the bathwater", meaning, while you are getting rid of something bad (ie dirty bathwater) make sure you keep the good elements (ie a cute little baby)