The PIRG report, "Fixing the Broken Textbook Market," suggests that students who are already struggling to afford college may do things that undermine their education to deal with the rising costs of their books.
(Read more: Paper or email? Pros and cons of digital receipts)
That conclusion is based on a survey of more than 2,000 students from more than 150 different campuses across the country conducted last fall.
- 65 percent said they had decided against buying a textbook because it was too expensive.
- Nearly half (48 percent) said the cost of books had an impact on how many or which classes they took.
- 94 percent of the students who had skipped buying a required book said they were concerned that doing this would hurt their grade in that course.
"Not only are students choosing not to purchase the materials they are assigned by their professor, but they are knowinglyaccepting the risk of a lower grade to avoid paying for the textbook," the study concluded.
(Read more: Colleges with the most bang for the buck: Survey)
PIRG says publishers use "a set of tactics to drive prices skyward," such as releasing new editions every three to four years regardless of changes in the subject material.
"They price these new editions quite high, which in turn dictates the price of used books and rental books," said Ethan Senak, higher education associate at U.S. PIRG. "Even as they move into e-textbooks, publishers incorporate paywalls, expiration dates and printing restrictions that further continue the practices they've used to control the traditional market."
(Read more: Consumers have had enough, 'rage survey' says)
The publishing industry says PIRG got it all wrong.
"We feel this report is highly distorted and biased," said David Anderson, executive director for higher education at the Association of American Publishers. "Comparing online open source content with textbooks is not a fair comparison."
The association said that students can use a website called CourseSmart to rent their textbooks for $200 a semester.
"Digital textbooks are the wave of the future and publishers are adjusting to that in a way that is very affordable for students," Anderson said.
Samantha Zwerling, student body president at the University of Maryland, College Park, sees the cost of textbooks having a negative impact on her classmates.
"Students might not be taking a course with the best professor or a course that they'd really like to take because the materials cost so much," she said. "Textbooks are not only hurting their wallets, but also their grades and their academic decisions."
(Read more: Americans need financial education, report says)
Consumer advocates believe the solution is to expand the market for "open textbooks." These books are written by faculty and peer-reviewed, just like traditional books, but they're free online and to download. They're typically available in print for between $20 and $40.
Irene Duranczyk, a University of Minnesota professor, uses an open textbook for her statistics course. She said the online material is high quality and can be customized to fit her teaching style. The traditional book is $180; the open source one is free.
"Students are very, very appreciative of being assigned a textbook that didn't break the bank," Duranczyk said.
Several major universities have invested in supporting and developing high-quality open textbooks for their students, but the percentage of the market is still minimal.
"Open textbooks have the potential to change education for the better and I plan to use them in my classroom in the semesters ahead," Duranczyk said.
—By CNBC contributor Herb Weisbaum. Follow him on Facebook and Twitter @TheConsumerman or visit The ConsumerMan website.