We describe intermediate Jacobians of Gushel-Mukai varieties $X$ of dimensions 3 or 5: if $A$ is the Lagrangian space associated with $X$, we prove that the intermediate Jacobian of $X$ is isomorphic to the Albanese variety of the canonical double covering of any of the two dual Eisenbud-Popescu-Walter surfaces associated with $A$. As an application, we describe the period maps for Gushel-Mukai threefolds and fivefolds.