How do isolated quantum systems approach an equilibrium state? We experimentally and theoretically address this question for a prototypical spin system formed by ultracold atoms prepared in two Rydberg states with different orbital angular momenta. By coupling these states with a resonant microwave driving we realize a dipolar XY spin-1/2 model in an external field. Starting from a spin-polarized state we suddenly switch on the external field and monitor the subsequent many-body dynamics. Our key observation is density dependent relaxation of the total magnetization much faster than typical decoherence rates. To determine the processes governing this relaxation we compare with different theoretical approaches which treat quantum effects on initial conditions and dynamical laws separately. This allows us to identify an intrinsically quantum component to the relaxation attributed to primordial quantum fluctuations.