The structure and behaviour of many social systems are shaped by the interactions among their individuals. Representing them as complex networks has shed light on the mechanisms that govern their formation and evolution. Such representations, however, focus on interactions between pairs of individuals (represented by the edges of the network), although many social interactions involve instead groups: these can be represented as hyperedges, that can comprehend any number of individuals, leading to the use of higher-order network representations. While recent research has investigated the structure of static higher-order networks, little is known about the mechanisms that govern their evolution. How do groups form and develop? How do people move between different groups? Here, we investigate the temporal dynamics of group change, using empirical social interactions collected in different settings. We leverage proximity records from two data-collection efforts that have tracked the temporal evolution of social interactions among students of a university and children in a preschool. We study the mechanisms governing the temporal dynamics both at the node and group level, characterising how individuals navigate groups and how groups form and disaggregate, finding robust patterns across contexts. We then propose a dynamical hypergraph model that closely reproduces the empirical observations. These results represent a further step in understanding the social dynamics of higher-order interactions, stressing the importance of considering their temporal aspect. The proposed model moreover opens up research directions to study the impact of higher-order temporal network patterns on dynamical processes that evolve on top of them.