You know NIMBY stands for “not in my backyard,” and you know that when the attitude rears its ugly head, it can block or delay almost any proposed new housing development.

But you might have forgotten that, when the term was coined in the 1980s, it for the most part applied to local citizens groups expressing opposition to not-so-desirable projects like prisons and garbage dumps.

This summer, while vacationing on Nantucket, I got a close look at how NIMBYism works (or doesn’t work) when it comes to housing. Developers proposed building 60 homes and 96 condos on a 13-acre site, with 15 of those homes and 24 condos set aside for low- to moderate-income buyers.

Considering that Nantucket is one of the top U.S. counties with the highest median housing price and it has an acute shortage of affordable housing, you might think the development would garner lots of local support.

But when it comes to housing, almost no good deed goes unpunished. A few advocates applauded the Surfside Crossing project and continue to support it, but they are being shouted down by 285 island homeowners who formed Nantucket Tipping Point in opposition. The group hired a consultant and planted signs on the island saying traffic from Surfside Crossing would further clog the roads and wastewater from Surfside Crossing would overtax the fragile sewage system. (Never mind that many of the prospective owners already live on the island, drive cars, and use water.)

At the latest public hearing, the development’s opponents seemingly got a boost from a candidate for Nantucket’s state senate seat. He stood up and said, “This project will change the face of this island for the next 100 years. ... This island is losing its soul little by little.” The audience clapped. Surprise! The candidate is a Republican, and the island votes Democratic.

I’m pretty certain most of you think NIMBYism is a decidedly liberal Democratic movement. Turns out that’s not true. In fact, a recent New York Times article says it “knows no party limits.” To prove the point, the article referenced two recent surveys on homeowners’ attitudes toward new housing development.

The first, by a political scientist at Baruch College, found no evidence that conservatives particularly oppose regulation, but it showed they are less likely than liberals to support a 10% increase in their city’s housing supply and more likely to support a ban on new development in their neighborhood.

The second survey, by two Stanford professors, showed that even those liberals who support policies that redistribute wealth nonetheless oppose new developments for low- and middle-income families in their community.

Builders can expect to take a left jab and a right cross from groups who espouse the NIMBY mindset. Surfside Crossing likely will be able to take those punches because Massachusetts has passed a law that allows builders in many cases to override local zoning regulations if they’re building below market price housing in communities with less than 10% affordable housing stock. Only 4% of the housing on Nantucket is deemed affordable to lower-income buyers.

For most of the rest of you—at least until the U.S. political climate changes—know that new housing and, by implication, homeownership, has about the same status as a new prison or a garbage dump with many community activists on the left and the right. And that has to make you wonder how it’s come to this and what has to happen make affordable housing a national priority.