The House speaker will hold a crucial role in negotiations to fund the government later this year, as a recently passed stopgap spending measure expires in December. Some lawmakers have threatened a government shutdown over federal funding of Planned Parenthood.
At a party meeting, McCarthy told fellow Republicans: "I am not the one at this time," expressing doubt that he could garner the 218 votes needed to win the position. Some within the party have floated South Carolina Rep. Trey Gowdy, Utah Rep. Jason Chaffetz and Florida Rep. Daniel Webster as possible successors to Boehner. Chaffetz said Thursday that McCarthy's departure creates an opening for his candidacy.
Read MoreBoehner: House leadership turmoil would do harm
California Rep. Darrell Issa said he did not believe any current candidate for the position could have won a majority vote in the full House chamber.
In a statement, Wisconsin Rep. Paul Ryan stressed he would not pursue the position. The 2012 GOP vice presidential candidate said he was "disappointed" in McCarthy's decision, calling him "the best person to the lead the House."
Read MoreObama: 'Still a path' to budget compromise
McCarthy, the House majority leader from California, recently received criticism for implying the special panel investigating the 2012 attack on a diplomatic compound in Benghazi, Libya, helped to reduce support for Hillary Clinton's presidential bid. On Thursday, he said he "should not be a distraction from" the invesigation.
McCarthy acknowledged that his comments did not help his case for speaker.
Because his decision to leave the race could lead to Boehner holding the position longer than anticipated, the situation could bode well for a debt deal, said Edward Mills, a financial policy analyst at FBR Capital Markets. Boehner is seen as more willing to compromise with Democrats, especially on his way out of office.
Read MoreSo...what happens when the government shuts down?
Republican presidential frontrunner Donald Trump chimed in on the House leadership, saying on Twitter, "we need a really smart and really tough person to take over this very important job."
Read MoreHouse conservatives back Webster for speaker
Click here for the latest on the markets.
— Reuters contributed to this report
Correction: An earlier version misstated Webster's state. He is from Florida.