In this project, we consider the problem of fish species classification where, given an external image of a fish specimen, our goal is to identify the species class of using machine learning (ML) methods. Fish species classification is an important task that is the foundation of many industrial, commercial, ecological, and scientific applications involving the study of fish distributions, dynamics, and evolution. While conventional approaches for this task have used off-the-shelf ML methods such as Convolutional neural network (CNN) architectures, there is an opportunity to inform the CNN architecture using our knowledge of biological hierarchies among taxonomic classes. In this work, we propose infusing some metadata in the form of phylogenetic information into the model’s training. Namely, in a mix of supervised and unsupervised multi-task learning formulations, we use the genus of the fish to guide the structure of our model’s hidden layers and relationships among the extracted features. The proposed model, named Hierarchy-Guided Neural Network (HGNN), outperforms conventional CNN models in terms of classification accuracy even with scarce training data in our extensive experimental analyses. We also observe that HGNN shows better resilience to adversarial occlusions, where some of the most informative patch regions of the image are intentionally blocked and their effect on classification accuracy is studied. Additionally, we examine HGNN from several other angles including the interpretability of the extracted features (using saliency map visualizations in the input image space) and the mapping of extracted features to biological traits, the diversity of these features, and the model’s ability to generalize beyond seen species, over a large database of more than 23,000 images that we have recently gathered from several museums.