The Datta lab studies how natural behavior supports cognition. Here we describe a method that combines 3D machine vision with unsupervised machine learning, to characterize the underlying structure of mouse behavior. We refer to this approach as Motion Sequencing (MoSeq). Using MoSeq we have discovered that mouse behavior can be segmented into a fundamental set of components that we call “behavioral syllables.” Each behavioral syllable is a brief and well-defined motif of 3D behavior that the brain places in into specific sequences via definable transition statistics (or behavioral “grammar”) to flexibly create complex patterns of action. By characterizing mouse behavior in terms of its component parts, we can use our behavioral characterization technique to identify subtle differences in the pattern of motor output under different experimental conditions with an unprecedented level of sensitivity. By combining MoSeq with in vivo imaging of neural circuits in behaving animals, we have identified context-dependent neural correlates for the sub-second structure, and have identified the dorsolateral striatum as a key node for implementing naturalistic behavioral sequences. We have developed a closed-loop version of MoSeq that enables us to trigger optogenetic interventions based upon the expression of targeted syllable or sequences; using this system to pulse dopamine during the expression of targeted behavioral syllables reveals rules and principles that constrain naturalistic learning. We have also developed extensions of MoSeq that enable characterization of multiple animals in complex environments. These experiments demonstrate that MoSeq can serve as a quantitative prism useful for characterizing relationships between neural circuit activity and naturalistic behavior.