Some of the other survivors swam ashore, but others were rescued after search teams climbed aboard the upside-down hull and heard people yelling for help from within more than 12 hours after the ship overturned.
Footage from state broadcaster CCTV showed rescuers in orange life vests climbing on the upside-down hull, with one of them lying down tapping a hammer and listening for a response, then gesturing downward.
Divers pulled out a 65-year-old woman and, later, two men who had been trapped, CCTV said. It said additional people had been found and were being rescued, but did not say whether they were still inside the overturned hull.
Read MoreTrading travel: 4 plays on transportation names
"We will do everything we can to rescue everyone trapped in there, no matter they're still alive or not and we will treat them as our own families," Hubei military region commander Chen Shoumin said during a news conference broadcast live on CCTV.
The 65-year-old woman was rescued by divers who brought an extra breathing apparatus up into the bowels of the ship and spent about five minutes teaching her how to use it before bringing her out to safety, Chen said.
"That old woman had a very strong will and learned very fast, and after 20 minutes she surfaced to the water and was rescued," Chen said.
Thirteen navy divers were on the scene and would be bolstered by 170 more by Wednesday, he said. Chinese Premier Li Keqiang traveled to the accident site to help coordinate efforts.
The overturned ship had drifted about 3 kilometers (almost 2 miles) downstream before coming to rest close to the shore, where fast currents made the rescue difficult. The location is about 180 kilometers (110 miles) west of the Hubei provincial capital of Wuhan.
The fact that the capsized ship drifted downstream was a good sign for rescuers because it meant there was enough air inside to give it buoyancy, and could mean there are enough air pockets for survivors to breathe, said Chi-Mo Park, a professor of naval architecture and ocean engineering at South Korea's Ulsan University.
"It all depends how much space there is inside the vessel," Park said.
Xinhua cited the captain and the chief engineer as saying the ship sank quickly after being caught in what they described as a cyclone. The Communist Party-run People's Daily said the ship sank within two minutes.
CCTV said the four-level ship had been carrying 406 Chinese passengers, five travel agency employees and 47 crew members. The broadcaster said most of the passengers were 50 to 80 years of age.
Many of the ship's passengers took a bus from Shanghai to Nanjing for the departure to Chongqing. Relatives of passengers gathered in Shanghai at a travel agency that had booked many of the trips, and later headed to a government office to demand more information about the accident before police broke up the gatherings.
Huang Yan, 49, an accountant in Shanghai, wept as she told a reporter that she believes that her husband, 49, and his father, who is in his 70s, were aboard the boat. But she said she couldn't be sure because she hadn't seen an official passenger list yet.
"Why did the captain leave the ship while the passengers were still missing?" Huang shouted. "We want the government to release the name list to see who was on the boat."
A group of about a dozen retirees from a Shanghai bus company were on the trip, said a woman who identified herself only by her surname, Chen. Among them, she said, were her elder sister and her elder sister's husband, both 60, and their 6-year-old granddaughter.
"This group has traveled together a lot, but only on short trips. This is the first time they traveled for a long trip," Chen said.
The ship sank in the Damazhou waterway section, where the river is 15 meters (50 feet) deep. The Yangtze is the world's third-longest river and sometimes floods during the summer monsoon season.
More than 50 boats and 3,000 people were involved in search efforts.
The Eastern Star measured 251 feet long (76.5 meters) and 36 feet wide (11 meters) and was capable of carrying a maximum of 534 people, CCTV reported. It is owned by the Chongqing Eastern Shipping Corp., which focuses on tourism routes in the popular Three Gorges river canyon region. The company could not be reached for comment.
CCTV reported that 6 inches (150 millimeters) of rain had fallen in the region over the past 24 hours. Local media reported winds reached 80 mph (130 kph) during the accident.