Nearly all of the hospitals that scored at the top of U.S. News & World Report's oft-touted "Best Hospital" rankings failed to obtain a top five-star overall quality rating from Hospital Compare.
Among them was Massachusetts General Hospital in Boston, the top-ranked hospital in the nation by U.S. News — as prominently noted on the hospital's own website — which only received four stars overall from Hospital Compare.
Dr. Elizabeth Mort, senior vice president for quality and safety at Mass General, said, "I wouldn't hang my hat on this score, one way or the other."
"We're used to getting these different rankings," Mort said. "I'm not in the least surprised that in one we will get the highest-possible rating, and in one we will not ... these methods for the CMS star ratings are vastly different from the methods used by U.S. News."
Mort said some of CMS' star rating measurements "are significantly flawed," particularly the "readmission rate," which tracks how many patients are readmitted to a hospital within 30 days of discharge. "It's probably not a very good measure of hospital quality," she said.
Asked if she thought Mass General's business would be harmed by having four stars, instead of five, Mort said, "No."
"Smart consumers know that these reporting vehicles tell you something. But they don't tell you everything, and they might not tell you the most important thing," she said.
Another renowned Boston hospital, Brigham and Women's, received just three stars under the new ratings, despite being ranked as the sixth-best hospital in the nation by U.S. News, as its own website proudly says. Brigham and Women's is a major teaching hospital of Harvard University.
Brigham and Women's declined to comment when contacted by CNBC.
Three stars was the most common rating for the nation's hospitals, received by 1,770 facilities, or more than 38 percent of all hospitals.
Only the Mayo Clinic in Rochester, Minnesota, which is ranked second by U.S. News, received five stars.
Most others in U.S. News' top list received three or four stars from Hospital Compare.
But two hospitals on that list — the University of Pittsburgh Medical Center and Barnes-Jewish Hospital, which is affiliated with Washington University in St. Louis — received just a two-star rating from Hospital Compare. Barnes-Jewish is ranked 11th in the nation by U.S. News; UPMC is ranked 14th.
U.S. News says its rankings are based on data for nearly 5,000 medical centers and results from surveys of more than 140,000 physicians, as well as other factors including death rates, patient safety and hospital reputation.
In an emailed statement, a UPMC spokeswoman said, "UPMC is a 20-plus hospital integrated health system that typically cares for the sickest of the sick across western Pennsylvania. Ten of our 13 hospitals rated by CMS rank at three stars or higher."
"That said, UPMC and many leading academic medical centers across the country have raised concerns about the questionable methodology used in this process, which does not appropriately adjust for patient complexity and socio-economic status," said the spokeswoman, Wendy Zellner. "This has led to low ratings for many prominent hospitals that provide excellent care. We support the public reporting of quality data, but look forward to working with CMS and others to refine these measures and methodologies in the future."
Dr. Clay Dunangan, senior vice president and chief clinical officer at BJC HealthCare, the parent of Barnes-Jewish, echoed those criticisms of the methodology used in the star ratings, which he said are "significantly biased against large hospitals like Barnes-Jewish that provide care for patients with complex medical conditions" and "a higher proportions of individuals facing socioeconomic challenges."
"The star rating system is an oversimplification of performance data and isn't very helpful for consumers trying to identify the best provider for their specific health care needs," Dunangan said. "BJH is an outstanding hospital that delivers exceptional care and has been awarded numerous honors for its quality and reputation. We strongly believe using these star ratings as an indicator of total quality of care delivered is misleading and does not serve the public with all of the facts."
Only 102, or 2.2 percent of the nation's 4,600 or so hospitals, received a five-star rating, according to CMS.
One of those was the Hospital for Special Surgery in New York City, which specializes in orthopedic surgery and the treatment of rheumatologic conditions, and whose doctors are team physicians for a number of major league New York sports teams.
"We are very proud of our results and feel that it is a reflection of our high quality of care," said Dr. Catherine MacLean, chief value medical officer at HSS. "In addition to putting great effort into preventing complications, such as those reflected in the CMS scorecard, we are very focused on improving the health outcomes of the patients we treat."
More than 900 hospitals, or 20 percent, received a four-star rating. Another 723 hospitals, or nearly 16 percent, have two-star ratings. Almost 3 percent of hospitals, or 133, received the lowest rating of one star.
Another 937 hospitals, or more than 20 percent, were not assigned a rating. CMS said no star is assigned in cases of hospitals that do not report or do not have minimum amounts of data, which can occur if a facility is small or new, or has an insufficient number of cases.
Ratings for hospitals in Maryland are, for the moment, not publicly available. CMS said concerns were raised because Maryland hospitals are not currently required to report on a data element relating to health conditions acquired by a patient while hospitalized. CMS said it "will continue to work with the state hospital association to determine when Maryland data will continue to be publicly reported."