UKIP favors an immediate British EU exit and sharply lower immigration. Mark Reckless, UKIP's candidate, was a Conservative lawmaker until he defected in September.
Read MoreEU tells Britain to pay extra €2.1bn
Reckless won 16,867 votes or just over 42 percent of the vote in the south-eastern English constituency of Rochester and Strood giving him a majority of 2,920. Cameron's Conservatives came second with 13,947 votes. Labor came third with 6,713 votes. The overall turnout was 50.67 percent.
UKIP's success highlights the threat it poses to Cameron in a national election in May 2015 and its ability to split the mainstream Conservative party's vote casting a cloud over its re-election prospects.
"Rochester and Strood was our 271st most winnable seat," Reckless said after the results were announced. "If we can win here we can win across the country."
Read MoreHopes that $375 billion can boost European growth
The loss is a serious blow to Cameron after he ordered his party to mobilize all its resources to hold Rochester and visited the area five times himself before Thursday's ballot.