Dimensionality Reduction Notes

Principal Components Analysis (PCA)

How do you choose how many and which eigenvalues/eigenvectors to use?

Kaiser Criterion
This says to retain only factors with eigenvalues greater than 1. In other words, if a factor does not extract at least as much as the equivalent of one original variable then it is discarded. This criterion is named after Kaiser as he proposed it in 1960. It seems used quite frequently.

The Scree Test
This is a graphical test used to decide how many factors to keep. To perform this test, first, plot the eigenvalues in decreasing order. Next, Cattell suggests to find the place where the smooth decrease of eigenvalues appears to level off (to the right) similar geological scree (loose rock debris at the bottom of a rocky slope).

Here are some other useful terms and definitions from the dictionary:

Multicollinearity refers to linear inter-correlation among variables. Simply put, if nominally "different" measures actually quantify the same phenomenon to a significant degree -- i.e., wherein the variables are accorded different names and perhaps employ different numeric measurement scales but correlate highly with each other -- they are redundant.

K-Means Clustering

Basic Algorithm
1. Choose k cluster centers at random
2. Assign each point to nearest cluster center
3. Compute the new cluster centers based on the assigned points
4. Repeat until cluster centers converge

Shortcomings
Finds local minima
The random placement of cluster centers affects the outcome

Here is a nice K-Means Demo