Every so often I’m reminded that the effects of changing dimensionality on objects and processes can be surprisingly counterintuitive. Recently I ran across a great example of this, while I working on a model for the distribution of distances in spaces of varying dimension.

Suppose that I draw some values from a classic one-dimensional Gaussian, with zero mean and unit variance, but that I am actually interested in their corresponding distances from center. Knowing that my Gaussian is centered on the origin, I can rephrase that as: the distribution of magnitudes of values drawn from that Gaussian. I can simulate this process by actually samping Gaussian values and taking their absolute value. When I do, I get the following result:

It’s easy to see – and intuitive – that the resulting distribution is a half-Gaussian, as I confirmed by overlaying the histogrammed samples above with a half-Gaussian PDF (displayed in green).

I wanted to generalize this basic idea into some arbitrary dimensionality, (d), where I draw d-vectors from an d-dimensional Gaussian (again, centered on the origin with unit variances). When I take the magnitudes of these sampled d-vectors, what will the probability distribution of *their* magnitudes look like?

My intuitive assumption was that these magnitudes would *also* follow a half-Gaussian distribution. After all, every multivariate Gaussian is densest at its mean, just like the univariate case I examined above. In fact I was so confident in this assumption that I built my initial modeling around it. Great confusion ensued, when I saw how poorly my models were working on my higher-dimensional data!

Eventually it occurred to me to do the obvious thing and generate some visualizations from higher dimensional data. For example, here is the correponding plot generated from a bivariate Gaussian (d=2):

Surprise – the distribution at d=2 is *not even close to half-Gaussian!*. My intuitions couldn’t have been more misleading!

Where did I go wrong?

I’ll start by observing what happens when I take a multi-dimensional PDF of vectors in (d) dimensions and project it down to a one-dimensional PDF of the corresponding vector magnitudes. To keep things simple, I will be assuming a multi-dimensional PDF _{r}(**x**_{d})**0**_{d} mean and **I**_{d} for a covariance matrix satisfies these two assumptions. With this in mind, you can see that the process of projecting from vectors in **R**_{d} to their distance from **0**_{d} (their magnitude) is equivalent to summing all densities _{r}(**x**_{d})_{d}(r)f’(r)_{r}(**x**)**x** with magnitude of (r), and A_{d}(r) is the surface area of a d-sphere with radius (r):

The key observation is that this term is a *polynomial* function of radius (r), with degree (d-1). When d=1, it is simply a constant multiplier and so we get the half-Gaussian distribution we expect, but when

The above ideas can be expressed compactly as follows:

In my experiments, I am using multivariate Gaussians of mean **0**_{d} and unit covariance matrix **I**_{d}, and so the form for f(r;d) becomes:

This form is in fact the generalized gamma distribution, with scale parameter ^{1/2},

I can verify that this PDF is correct by plotting it against randomly sampled data at differing dimensions:

This plot demonstrates both that the PDF expression is correct for varying dimensionalities and also illustrates how the shape of the PDF evolves as dimensionality changes. For me, it was a great example of challenging my intuitions and learning something completely unexpected about the interplay of distances and dimension.