Building orientation with respect to the radar look direction has a critical influence on the interpretation of multi-look PolSAR data in urban areas. In this paper, its impacts on polarimetric orientation angle (POA) estimation and model-based decomposition are discussed. The discussion begins with the analysis of the general double-bounce scattering model, of which the characteristics are dependent on the electromagnetic and geometric parameters of the related dihedral structure. Then, for multi-look PolSAR data, the polarimetric scattering mechanism in urban areas is modeled by two double-bounce scatterings from two orthogonal dihedral structures. From the model, the impacts of the building orientation on POA estimation can be revealed. With the increase of the building orientation, the POA difference between the two dihedral structures increases gradually and the feasibility to estimate the building orientation via the estimated POA is reduced dramatically. Upon further analysis, we illustrate the impacts on the model-based decomposition. With the increase of the building orientation, the dominant scattering mechanism labeling technique based on the model-based decompositions will gradually become invalid. Moreover, the processing of POA compensation, which is helpful to reduce the impacts of the building orientation, also becomes invalid when the building orientation increases to a certain value. At last, three L-band datasets of San Francisco acquired by AIRSAR are used to verify the inferences. The experimental results show that for L-band PolSAR data in urban areas, when the radar look angle is around 45, the threshold of building orientation for the validity of dominant scattering mechanism labeling is about 3, and for the POA compensation, the threshold is about 12.