Using new data on roll-call voting of US state legislators and public opinion in their districts, we explain how ideological polarization of voters within districts can lead to legislative polarization. In so-called “moderate” districts that switch hands between parties, legislative behavior is shaped by the fact that voters are often quite heterogeneous: the ideological distance between Democrats and Republicans within these districts is often greater than the distance between liberal cities and conservative rural areas. We root this intuition in a formal model that associates intradistrict ideological heterogeneity with uncertainty about the ideological location of the median voter. We then demonstrate that among districts with similar median voter ideologies, the difference in legislative behavior between Democratic and Republican state legislators is greater in more ideologically heterogeneous districts. Our findings suggest that accounting for the subtleties of political geography can help explain the coexistence of polarized legislators and a mass public that appears to contain many moderates.