Orszag, who plans to get married in September, had never intended to stay more than two years and wants to leave before the next budget season begins in July, the sources said.
Potential successors include Laura Tyson, an economist of the University of California at Berkeley, and Gene Sperling, counselor to Treasury Secretary Tim Geithner.
Both are former White House economic advisers who served in President Bill Clinton's administration.
Another candidate is Robert Greenstein, executive director of the Center on Budget and Policy Priorities.
Orszag, a former Congressional Budget Office director, was one of the major players helping to shape the Obama administration's $862 billion stimulus package, which passed last year.
He also helped to champion what the administration says will be long-term cost-savings from Obama's overhaul of the healthcare system.
The Obama administration has come under pressure to rein in soaring U.S. budget deficits, which hit $1.4 trillion last year.
A $1.6 trillion deficit is expected this year.
Orszag announced in a speech earlier this month that government agencies were being asked to plan for 5 percent cuts in an array of domestic programs.
Obama has also named a commission to recommend ways to tackle the budget deficit and asked it to report back a few weeks after the November congressional elections.