The spectrum of light that an animal can see – from ultraviolet to far red light – is governed by the number and wavelength sensitivity of a family of retinal proteins called opsins. It has been hypothesized that the spectrum of light available in an environment influences the range of colors that a species has evolved to see. However, invertebrates and vertebrates, which often live in similar habitats, use phylogenetically distinct groups of opsins in their retinae to see. It is unclear whether habitat similarly affects visual sensitivity across animal phyla, particularly when comparing across lineages that use different types of opsins. To close this gap in our knowledge, we conducted a systematic literature review that compared the bluest, reddest, and range of colors seen by >200 species of invertebrates and vertebrates to the habitats in which they live. We found that aquatic animals see bluer and redder light than terrestrial animals, and that invertebrates see bluer light than vertebrates; however, controlling for phylogeny removes the effects of habitat and lineage on visual sensitivity. We found that closed and open habitat terrestrial species have similar spectral sensitivities, but that deep water animals see bluer light than shallow water animals. These results suggest that, while ecological factors influence animals’ visual sensitivities, evolutionary history a strong effect on which colors a species can see.