Although people may recognize it when they see it or feel it, urban vibrancy is challenging to measure directly and comprehensively. In the context of the intense global competition, there is increased realization that urban vibrancy is vital to the social and economic sustainability of cities. Such vibrancy could be significantly shaped by the urban built environment (UBE), yet we know little about the intimate connections between vibrancy and UBEs. Empowered by the various newly available spatial big data sources that provide enormous information on both human dynamics and the built environment, this paper proposes a framework to evaluate and characterize urban vibrancy. Existing vibrancy measures mostly use single-source data that can hardly reflect the fact that urban vibrancy is largely a multi-facet affair. Therefore, we propose a more comprehensive measure of the multi-facet vibrancy, extracted as the common latent factor with its effects reflected in multiple surface attributes. Using the proposed framework, we evaluated and mapped the spatial dynamics of vibrancy in the city of Shanghai, a typical large city of post-reform China, and investigated the associations between vibrancy and various UBE indicators. Evidence shows that the horizontal built-up density, rather than vertical heights, is the leading generator of vibrancy in Shanghai, followed by the density and mixture of urban functions, accessibility, and walkability. In this vein, we contribute to current debates and future planning practices regarding vibrant spaces in large cities. Future urban studies can benefit from the proposed evaluation framework equipped with spatial big data.