"I feel ... regret and also anger," the poverty-stricken man, who has withheld his identity, told a press conference on Wednesday. "I want them to turn back the clock."
More from NBC News:
Joe Biden mum on airspace tensions after meeting with China's Xin Jinping
Mass grave found in Mali near military base
Rockefeller Center Christmas Tree set to sparkle Wednesday
While being filmed from the neck down to protect his requested anonymity, the 60-year-old said: "I heard that I was being sought after because of a mix-up. When I heard that, my initial feeling was, 'Is such a thing possible?' I didn't think it was possible that a hospital could make such a mistake."
On Tuesday, Tokyo's San-Ikukai Hospital was ordered by a court to pay the man 38 million yen ($371,233) in damages, significantly less than the 250 million yen ($2.5 million) he had been seeking.
Instead of the life of affluence for which he was destined, the man lived off welfare checks and grew up in a small apartment which had no electrical appliances. His given mother raised him and two siblings after their father died when he was two.
(Read more: 23andMe faces class action lawsuit in California)
The other baby grew up as the eldest of four siblings in a well-off family. He received private tutoring and went to university.
The error was uncovered in 2009 after the wealthy family realized one of the four brothers did not share their likeness and requested a DNA test. After they found out they were not related they searched hospital records and eventually found their true brother in 2011.
The man who grew up in poverty never knew his real parents.
"When I found out about my true parents, I wish I was brought up by them," he said. "That's the truth. When I was handed the photograph of my [real] parents, it made me want to see them. Every time I see their photograph, for several months tears would well up."
(Read more: Supreme Court hears frequent-flier case)
According to Britain's Telegraph newspaper, Judge Masatoshi Miyasaka told the court that it was "impossible to assess the scale of the pain and disappointment both the parents and the man had to suffer, as they were deprived of opportunities to enjoy their parent-child relationship forever."
The impoverished man now goes out to drinks with his "real" brothers once a month, while taking care of the elder brother with whom he grew up.