Although time-dependent random media with short-range correlations lead to (possibly biased) normal tracer diffusion, anomalous fluctuations occur away from the most probable direction. This was pointed out recently in one-dimensional (1D) lattice random walks, where statistics related to the 1D Kardar-Parisi-Zhang (KPZ) universality class, i.e., the Gaussian unitary ensemble Tracy-Widom distribution, were shown to arise. Here, we provide a simple picture for this correspondence, directly in the continuum, which allows one to study arbitrary space dimensions and to predict a variety of universal distributions. In d=1, we predict and verify numerically the emergence of the Gaussian orthogonal ensemble Tracy-Widom distribution for fluctuations of the transition probability. In d=3, we predict a phase transition from Gaussian fluctuations to three-dimensional KPZ-type fluctuations as the bias is increased. We predict KPZ universal distributions for the arrival time of a first particle from a cloud diffusing in such media.