Can you roll a basketball on the floor so that the ball comes back to the same position, but the orientation of its pattern has changed? Yes, you can, and after a few tries you will find out that the final orientation of the pattern is determined by the path traced by the ball on the floor. Quantum systems behave pretty much the same way: when subjected to a slowly varying external field, they acquire a phase that depends on the path they trace in their Hilbert space, that is, a geometric phase. In this work we have studied what happens when the external field is itself a quantum system, namely, a harmonic oscillator which can be populated with any number of photons. Instead of varying the state of the field, we prepare the field in a given state and slowly vary its coupling to the system. What we find is that a geometric phase is acquired even when the field is in the vacuum state, a result that has no semiclassical explanation but is correctly predicted by a full quantum theory. Our ability to control this phase opens new possibilities for the coherent manipulation of hybrid atom-cavity systems, with direct implications for quantum information processing. The results are published in Science Advances.