In April, the national participation rate fell to 62.8 percent, revisiting a 36-year low reached late last year, although a report on Friday is expected to show it ticked up in May. During 2007, before the financial crisis and the recession, it was as high as 66.4 percent.
Some of the decline has been due to an aging workforce and the retirement of baby boomers, a fact that may well keep participation from ever bouncing back to its pre-crisis level.
But some dropouts went to college and are bound to eventually restart job hunts. Others grew frustrated at the lack of available jobs, but may decide to try their luck again if the economy continues to improve.
Some prominent economists, including former White House adviser Alan Krueger, argue that many of the folks on the margins of the labor force are not coming back. If that is true, higher inflation, fueled by rising wages, could come sooner than the Fed expects.
Read MoreUS service sector growth accelerates in May: ISM
But Yellen, who took the helm at the Fed in February, has refused to write off Americans who have suffered long bouts of unemployment or given up the search for work entirely. She argues there is more slack in the economy than suggested by the nation's 6.3 percent unemployment rate, a key reason the Fed is expected to bide its time before hiking rates.
Along with the drop outs and record number of long-term unemployed, millions are working in part-time jobs even though they want full-time work—another fact Yellen has cited.
The state data, which can diverge from the national statistics because of adjustments the government makes to account for seasonal swings and other local economic factors, suggests she may be right to wait.
In places like Portland, local officials and entrepreneurs say a recovery in the job market is starting to gather pace.
Tech companies like business software manager Puppet Labs have been growing quickly in the city. Puppet Labs expects to double its work force to around 400 by the end of the year as it takes advantage of what CEO Luke Kanies said are wage levels as much as 20 percent lower than in hotter markets like San Francisco or Seattle.
While Oregon's labor participation rate has not gone up, officials say they feel the groundwork is in place.
Patrick Quinton, head of the Portland Development Commission, said the vacancy rate for commercial office space is now in the single digits because of the rapid local expansion in Portland of companies like accommodation booking service Airbnb and game maker Kixeye. That, he said, is expected to trigger a wave of office building and the creation of construction-related jobs in Portland, which accounts for around 15 percent of the state's population.
Job creation on its own is no guarantee that the country's labor pool will stabilize or expand. But recent research has held out some hope by focusing on the fate of the long-term unemployed—a group that, by historical standards, currently accounts for a disproportionate share of the unemployed.