Some conservatives quietly campaign against Ryan for speaker

Getty Images

Some conservatives in the Republican party are sharpening their knives to keep Representative Paul Ryan from succeeding House Speaker John Boehner, even as they say publicly the election of the next speaker will center on reforms in the U.S. House of Representatives instead of the candidates.

"The focus is on, in my opinion, not who is speaker, but the process changes we can put in place," said Representative Barry Loudermilk, a member of the Freedom Caucus made up of the most conservative Republicans in the House.

At least one caucus member has outlined requirements a candidate must meet in order to win an endorsement in the speaker's race, and it is not clear if Ryan can pass the test. In recent days a quiet campaign has started against Wisconsin's Ryan, the former vice-presidential candidate considered the only House member able to gather enough votes to win the election.\

For his part, Ryan has repeatedly said he will not enter the race.

A video is circulating around Congress of the floor speech that Ryan gave supporting creation of the Troubled Asset Relief Program, the emergency scheme that used public funds to shore up banks during the 2008 financial crisis and that is a bugaboo for many conservatives.

The title of the video on Youtube is "Paul Ryan BEGS Congress to Pass TARP (PATHETIC)."

They have also sent out reminders that Ryan authored the 2010 book "Young Guns" with former Representative Eric Cantor and House Majority Leader Kevin McCarthy, two men considered part of the establishment wing of the party who have seen their political ambitions foiled by more radical conservatives.

McCarthy shocked Washington last week when he withdrew from the speaker election after facing some of the same pressure hard-liners had laid on Boehner, who folded and announced last month he would resign at the end of October.

Read More New speaker race: Webster, Chaffetz & Ryan?

As the party scrambled to restart its selection process, the rhetoric shifted to focus on the role of the speaker. Last week some conservatives even floated the idea that members could temporarily rotate into the most powerful job in the chamber.

Freedom Caucus member Dave Brat of Virginia has posted to his website "10 Commitments the Next Speaker and House Leaders Should Make.

Most of the commitments would decentralize the House's power structure and give rank-and-file members greater say in granting committee chairmanships, drafting budget legislation, and bringing bills up for a vote. Ryan, though, has a high profile in the current structure, overseeing spending legislation as chairman of the Ways and Means Committee.

After McCarthy drops out, where does the GOP turn?

The list also includes policy areas such as a requirement to oppose "amnesty for illegal immigrants." Right-wing groups frequently attack Ryan as "pro-amnesty."

Brat said the caucus would be happy to interview Ryan if he decides to run.

Still, when asked if he personally supported Ryan, Brat said: "It's not about personal anything, its about rules. If we have regular order, and they want to sign their name to that, I'll support that person."