For structures lacking homologous landmarks, limited approaches are available to compare 3D shapes and discern evolutionary patterns. Instead of comparing shape variation, we used alpha shape analyses to quantify vaginal complexity in 40 marine mammal specimens. Additionally, we explored phylogenetic signal and the potential roles of natural selection and sexual selection on 3D vaginal shape evolution. Complexity scores were consistent with qualitative observations. Cetaceans had a broad range of alpha complexities, while pinnipeds were found to be comparatively simple, and sirenians were found to be complex. Surface shape complexity was driven by invaginations and protrusions of the vaginal wall. Despite intra-specific variation in vaginal shape, there was some species-level grouping in complexity within PCA morphospace. Significant phylogenetic signal, however, was absent from the data. Metrics of natural selection (relative neonate size) and sexual selection (sperm competition risk, sexual size dimorphism, and penis tip morphology) did not explain complexity patterns. Additional metrics, such as penile shape complexity, may yield interesting insights into marine mammal genital coevolution. We advocate for the use of alpha shapes to discern patterns of evolution that would otherwise not be possible in 3D anatomical structures lacking homologous landmarks.