If you don't believe the competition for labor is fierce, you should talk to Jase Prewett, vice president of construction at Brookfield Residential's Southern California office.

In April, Prewett says a rival drywall company came into one of its job-sites in Southern California as his contractors hung drywall. The firm's intent? Stealing Brookfield's drywall subcontractor's workers. One of the site managers asked them to leave, but an hour later, the rival staked out another house in the community, trying to entice different drywallers to come aboard.

In the past, contractors would engage in fierce competition for labor in California. But Prewett, whose company subcontracts all of its labor, has never seen anything quite like this—trade contractors fighting over crews by offering them signing bonuses and raises to come aboard exclusively.

"In the west, we've kind of have this unspoken rule—a kind of gentleman's agreement—that none of that recruiting happens on site," Prewett says. "But that's out the window now." 

How did it come to this, one contractor entering a construction site to steal another contractor's labor? There's not one simple reason. Instead, a litany of factors—the real estate bust, the resulting migration of construction workers back to their home countries, the oil boom, and construction perking back up again—led to one simple place: The scarcity of labor potentially strangling the housing recovery before it even really starts.

A Growing Problem

The numbers back up what Prewett and other builders around the country see firsthand. Construction firms added 280,000 over 12 months, with 45,000 just in April. The sector's unemployment rate fell to a nine-year April low of 7.5%, according to an analysis by the Associated General Contractors of America.

But that's for the construction industry as a whole. Drill down to the single-family level and the stats really hit home.

Consider Metrostudy's Builder Labor Supply report, which was released earlier this year and sponsored by Acme Brick, Johns Manville, Lubrizol, and MiTek--Berkshire Hathaway Companies. In that survey, 70% of builders and general contractors across the country cited "labor" as their No. 1 challenge, blowing away "material pricing" in second place with 20%. More than two-thirds (67%) of respondents said they were experiencing labor shortages. And, in every region, at least 64% of respondents reported experiencing labor issues.

In a January National Association of Home Builders survey, builders listed labor as their greatest concern for 2015. In 2014, it was their second biggest concern.

"The one persistent problem that won't get better is people," said TRI Pointe's CEO Doug Bauer in an interview late last year. "Sub trade labor will continue to be a problem. You'll continue to have higher labor costs. A lot of the guys in the industry that were experienced have left."

In many ways, you would think deep-pocketed public builders like Bauer's TRI Pointe, with 3,100 closings last year, could lock in labor at will. But, all of the builders who constructed 250 homes or more per year reported labor issues.

Fayetteville, N.C.–based Caviness & Cates Communities closed 444 homes in 2014, and it felt the squeeze for contractors. "The trades are tough right now because there's a shortage of them and lot of them went out of business in the downturn," says Chris Cates, co-owner of Caviness & Cates Communities. "The good trades are in such demand that they can just name their number."

Darren Sutton, president of Bonterra Builders in Matthews, N.C., which completed 303 homes last year himself, thinks many bigger companies get hung up in bureaucratic barriers.