Broadband spectral control of quantum light with frequency conversion