—Although the high dimensionality of hyperspectral data increases the separability of land covers, it is difficult to distinguish certain classes using only the spectral information due to the widespread mixed pixels and small sample size problems. Three-dimensional Gabor wavelet transform takes the entire hyperspectral data cube as a tensor, captures the joint spectral-spatial structures very well and has shown great potential to improve classification accuracies. However, much redundancy exists in the extracted huge amount of Gabor features, which inevitably degrades the efficiency of the method. To make matters worse, according to the Hughes phenomenon, the less informative bands/features may sacrifice the classification accuracy. In this paper, a two-stage feature selection framework, Affinity Propagation-Gabor-Conditional Mutual Information (abbreviated as AP-Gabor-CMI), is proposed to deal with the problems, which chooses the most important features before and after the Gabor wavelet-based feature extraction procedure. Specifically, the first stage picks out the most distinctive bands from the original hyperspectral data through complex wavelet structural similarity (CW-SSIM) index based affinity propagation clustering algorithm. After applying the Gabor wavelet-based feature extraction on the chosen bands, the second stage selects the most discriminative features from them by means of conditional mutual information-based feature ranking and elimination. Experimental results on three real hyperspectral data sets demonstrate the advantages of the proposed two-stage feature selection framework and the superiority of AP-Gabor-CMI over state-of-the-art methods when only few labeled samples per class are available.