In order to monitor the accumulation of heavy metals effectively and avoid the damage to the health of agricultural soils, a promising approach is to predict low concentrations of heavy metals in soils using visible and near-infrared (VNIR) reflectance spectroscopy coupled with calibration techniques. This study aimed to (i) compare the performance of a combination of partial least squares regression with genetic algorithm (GA-PLSR) against a general PLSR for predicting low concentrations of four heavy metals (i.e., As, Pb, Zn and Cu) in agricultural soils; (ii) explore the transferability of GA-PLSR models defined on one subset of land-use types to the other types; and (iii) to investigate the predictive mechanism for the prediction of the metals. One hundred soil samples were collected in the field locating at Yixing in China, and VNIR reflectance (350–2500 nm) spectra were measured in a laboratory. With the entire soil samples, GA-PLSR and PLSR models were calibrated for the four heavy metals using a leave-one-out cross-validation procedure. The GA-PLSR models achieved better cross-validated accuracies than the PLSR models. For the transferability of GA-PLSR models, the soil samples were divided into three pairs of training sets and test sets from different land-use types. Three GA-PLSR models defined on the training sets had good transferability to the test sets, but nine GA-PLSR models were not successful. As for the predictive mechanism, besides the widely-used correlation analysis between OM and the metals, the relationship between the content of OM and the prediction accuracy of the metals was investigated and the similarity of the important wavelengths for OM and the metals was compared. The three methods verified that OM had a significant correlation with the predictions of the spectrally-featureless metals (Pb, Zn and Cu) from VNIR reflectance. We conclude that GA-PLSR modeling has a better capability for the prediction of the low heavy metal concentrations from VNIR reflectance, and it has a potential of transferability between different land-use types, and its accuracy is fundamentally influenced by OM.