To become certified, a farmer must start with land that hasn't grown anything conventionally for three years, a transition that can be costly. Farmers are not supposed to use synthetic pesticides or herbicides (though the USDA still allows a few), no genetically modified seeds, and they must meet other standards set out by the government's National Organic Program.
The farmer has to provide a plan and a paper trail to a third-party certification inspector who also does annual audits. These audits usually consist of going over receipts and other paperwork, and often a check of the farm itself.
Rarely does it mean gathering actual soil samples or testing products for things like pesticide residue.
Read MoreGirl Scout cookie prices jumping to $5 here...
"We, as an organic certification agency, are required to perform testing (for) chemical residue (in) at least 5 percent of our operations," said Jake Lewin, president of California Certified Organic Farmers. "We have broad discretion about whether or not we test finished products, whether we test soil or plant tissue." He said his organization does about 140 "surprise" inspections a year in addition to normal audits.
Lewin said most of the errors he finds are honest accidents by the farmer. "I think it's really a mistake to make the assumption that the system is full of cheaters, there's no evidence for that." However, he said about once every 18 months he recommends that someone lose their organic certification, at least temporarily. "There are always going to be cheaters in any system, whatever it is. We're out there working to address it."