"One would expect an entire system to snap into place that would ensure that this patient receives all the treatments he or she needs," Dr. Neil Wenger of UCLA, one of the committee members, says in a video released with the report.
"That is not the way that this system works. We have the most highly trained oncologists but because we don't have coordination among all clinicians, this care doesn't serve patients well. Sometimes it even harms patients."
So someone with colon cancer who goes to his community hospital maybe treated by a surgeon who doesn't know to take out certain lymph glands for testing to see if the cancer has spread, says Ganz. "They may do too many tests," she told NBC News.
(Read more: Studies support fast track for Roche breast cancer drug: FDA)
Often doctors order too many CT scans or unnecessary MRIs, Ganz said. "These kinds of variations lead to potential risk or harm to the patient," she said. "Obviously if you don't have good access, you won't get good care." Plus it can be costly -- and patients often must pay a large chunk of this pricey and unnecessary care.
But patients shouldn't have to rely on being able to get to big, famous cancer centers like MD Anderson in Houston, Memorial Sloan-Kettering Cancer Center in New York or Fred Hutchison Cancer Center in Seattle, says Ganz.
"The truth is, not everybody can travel," says Dr. Clifford Hudis, president of the American Society of Clinical Oncology, who was not on the panel. "We have a golden opportunity now that we are in the age of bioinformatics." Electronic communications can help doctors connect to one another and share expertise, and it needs to happen more often, Hudis and the panel agree.
"Why shouldn't any doctor who is using a computer and electronic records ultimately be able to gain from the experience of everyone? Then it won't matter quite so much if you wander into a one-person office in a rural center," Hudis says.
(Read more: Rare disease at hospital raises concerns about VA health care)
Patients also need to get more informed, and they can't be left to the mercy of misinformation on the Internet, adds Ganz. "We do recommend there be good quality information available in both written and social media," she said. "What is out there and what is on blogs can be very distressing."
The report points out that Americans often don't understand a cancer diagnosis. Up to 80 percent given a diagnosis of incurable cancer don't fully get it — they think they can still be cured, the panel points out.
"Part of this has to do with human nature and the belief that you will be the exception," Hudis says. But many oncologists are also reluctant to deliver the bad news that a patient will never be cured.
Americans often emphasize fighting cancer, characterizing patients as courageous survivors who either beat the odds, or went down after a good fight. But evidence suggests that so-called palliative care — designed to ease pain and other symptoms -- not only makes patients more comfortable, but can help them live longer than intense chemotherapy.
And people shouldn't die of cancer in an intensive care unit, the report says — even though this is still happening far too often. Quality hospice care provides a better alternative.
On the other side, people often panic when they get a cancer diagnosis and may rush into treatment, says Ganz. "We don't want to frighten patients," she says. "I think the quality of care in many places around the country is really high and of a high standard. Patients need to understand that once they are told they have cancer, it is rarely an emergency."
With the exception of an unusual brain tumor or some forms of leukemia, there is usually not a hurry. "Most of the time you can make up your mind over several weeks," Ganz says.
"It will take training of professionals and it will take big changes at a policy level including how care is reimbursed," says Betty Ferrell, an oncology nurse researcher at the City of Hope cancer center in California who was on the IOM panel. "But everything that we are advocating for is ultimately extremely possible."
—By Maggie Fox, NBC News.