China has a unique political system and administrative system, which makes the agglomeration and allocation process of urban resources present a spatial hierarchy, and its manifestation is the urban administrative hierarchy. Important factors or resources such as capital, energy, human capital, advanced technology, infrastructure, and preferential policies, all follow a gradual distribution pattern from the central to the local, from higher to lower hierarchies. This means that the hierarchy of a city determines the size of administrative power to a large extent, which in turn affects the flow and redistribution of factor resources. Compared with cities that tend to obtain high-quality resources, market segmentation between cities severely restricts the cross-regional flow of factor resources, which will cause problems such as blind investment by local governments, speed of comparison, isomorphism of industries, and resource competition, resulting in inter-regional problems. The lack of overall planning and coordination leads to a loss of economic efficiency, which is not conducive to the development of urbanization. The development of China’s urbanization has moved from traditional urbanization based on urbanization rate to new-type urbanization based on urbanization efficiency, and new-type urbanization pursues green development and high-quality development. This paper explains the implementation mechanism of China’s urbanization efficiency improvement from the two dimensions of urban administrative hierarchy and market segmentation, and uses empirical tests with China’s urban data (excluding Hong Kong, Macau and Taiwan) and dynamic spatial panel model from 2003 to 2017. For the evaluation of urbanization efficiency, the modified Super-SBM is used for calculation to consider the impact of environmental factors. This is not only a more scientific and comprehensive evaluation of the efficiency of China’s urbanization, but also a high-quality development concept of green, low-carbon, intensive and efficient urbanization in China. Since changes in urbanization efficiency involve the spatial transfer and allocation of various element resources, this paper intends to use a dynamic spatial panel model to empirically study the effects of urban administrative hierarchy and market segmentation on urbanization efficiency, and conduct a series of robustness and heterogeneity tests. The conclusion of the study shows that the hierarchy of urban administration and the degree of market segmentation are the two major institutional barriers that impede the improvement of China’s urbanization efficiency, and thus fall into the development dilemma of ‘high-hierarchy, high-segmentation and low-efficiency’. The mechanism inspection found that the reason is that high-hierarchy cities use administrative advantages to sacrifice ecological urbanization and excessively pursue population urbanization, economic urbanization, and land urbanization. This is due to differences in household registration systems, local protection and environmental governance. Further heterogeneity tests found that the establishment of high-hierarchy cities in the eastern region, as well as high-intensity environmental constraints, more advanced infrastructure, more efficient public services, and a more open investment environment can greatly alleviate the inhibition effect of urban administrative hierarchy to urbanization efficiency.