Sexual color dimorphism is largely attributed to sexual selection, where different sexes will broadcast distinct displays to conspecifics to convey information such as mating availability and fitness for reproduction. This is balanced by the effects of natural selection, where colors that allow an organism to mitigate conspicuousness to potential predators may also contribute to fitness. Colors themselves contribute to conspicuousness, but so can overall pattern through the juxtaposition of adjacent or other co-occurring colors. Among species that exhibit sexual color dimorphism, disparity in color pattern geometry contributes to differences in relative conspicuousness towards both predators and conspecifics among sexes. We quantify color pattern geometry and complexity across males and females of dichromatic and non-dichromatic species of wrasses (Labridae) using charisma, a novel color classification method that automatically detects the number and proportions of color classes on each individual to best capture overall pattern diversity. While previous work has examined the evolution of color dimorphism in terms of the presence or absence of discrete colors across labrids, the evolution of the relative proportion and distribution of colors that comprise patterns among sexually dichromatic species has not been previously evaluated. We also use visual models and phylogenetic comparative methods to test how males and females diverge from one another in terms of statistics that capture color pattern geometry. We demonstrate that not only color composition, but also relative proportion and orientation of colors in patterns have important implications for different life histories among sexes of wrasses.