We study the second fundamental form of the Siegel metric in $\mathcal A_5$ restricted to the locus of intermediate Jacobians of cubic threefolds. We prove that the image of this second fundamental form, which is known to be non-trivial, is contained in the kernel of a suitable multiplication map. Some ingredients are: the conic bundle structure of cubic threefolds, Prym theory, Gaussian maps and Jacobian ideals.