Meeting Abstract
Deep learning is a method by which complex mathematical functions such as artificial neural networks are trained to make accurate predictions using a limited set of examples. Deep learning has proven wildly successful for tasks ranging from object detection to control of self-driving cars and has the potential to eliminate the need for extensive manual video digitization, which is one of the slowest bottlenecks in studies of animal behavior and biomechanics. Motivated by the growing availability of frameworks (e.g. Keras) to more easily construct and train neural networks, we tested these methods on the task of tracking six points on the body and wings of freely maneuvering blue orchard bees (Osmia lignaria) as they carried mud loads during nest construction. We filmed 43 flights of seven individuals and manually annotated 11 flapping cycles in each video to obtain ~10,000 training images across two camera views. Once trained, we deployed our network on 113 trials to automatically track six points in ~183,000 frames across 212 continuous flight segments, with a median of 32 tracked wing strokes per segment. Neural network predictions were robust to shadows and focus blur, and reprojection error of triangulated points (2% of wing length) was comparable to that of manually digitized data. We examined how the number of training examples and their allocation among individuals and trials affected neural network tracking performance. Finally, we explored the effect of mud loading on the relationship between fast wing motions and slower body dynamics during flight maneuvers. Deep learning methods greatly improve the speed and consistency with which data are extracted from long videos and allow us to unravel processes operating at multiple timescales over complex behavioral sequences.
