Magnetic vortices occur in soft magnetic nanodisks with suitable dimensions. The spin structure curls in the plane around the center where it points out-of-plane either up or down, defining the polarization P = ±1. Thus, the vortex core represents a binary digit that can be used for magnetic storage devices. Ferromagnetic resonance spectroscopy is used to investigate the high-frequency spectra of two stacked vortices depending on the relative polarizations. In this work, it is shown that magnetic field bursts with frequencies in the Gigahertz range can be used to control the polarization on a sub-nanosecond time scale. The demonstrated writing times are more than two magnitudes faster compared to settling times achieved with the gyrotropic mode. Furthermore, polarization dependent spin-wave modes that lead to a selective spin-wave mediated vortex core reversal are imaged by scanning transmission X-ray microscopy.