The USDA conducts its survey every five years to track the equine population.
Though it doesn't account for every horse in the country—there may be as many as 9 million horses in the U.S.—the survey found that the recorded number of horses among ranchers and farmers by 2012 was 3,621,348. That's down from the previous census number of 4,028,827.
The number of farms and ranches with horses, which includes thoroughbred breeding operations, was 504,795 in 2012. That's down from the 2007 census, which counted 575,941 farms. The most common size of farms and ranches with horses was 10 to 49 acres, of which there are some 184,000 out of the total.
Read MoreWhy bet on the Derby when you can own it?
Texas had the greatest population of horses, with 395,818 on some 64,000 farms and ranches. It was followed in number of horses by Oklahoma, California, Kentucky and Florida.
Using figures from its own survey, which go beyond the census, the AHC says that the majority of horses, around 3,906,923, are used for recreation while about 2,718,954 are used for showing, and some 844,531 are used for racing.