Carly Patterson: Don't give up if you want gold

Gold medalist: I almost walked away from it all

Carly Patterson might not have won her three medals in gymnastics at the 2004 Olympics in Athens if she had let herself give up.

"There was a time where I did want to quit. ... So I did ... for a week," the 28-year-old Patterson told CNBC. She said she was having some issues after her family moved to Houston. She watched her sister go to school and have a normal life.

"I was the exact opposite of that, but the thing is I would always look at that and say that looks really fun, but this is still what I want to do," she said.

Patterson returned to gymnastics and went on to become the first American woman since Mary Lou Retton in 1984 to take home a gold medal in the all-around competition. She also won two silver medals.

In this Aug. 3, 1984, file photo, Mary Lou Retton, of the United States, performs on the balance beam during the women's gymnastics individual all-around finals at the XXIII Summer Olympic Games in Los Angeles.
Mary Lou Retton: Ignore your critics, do what you love
Amy Purdy in action during training on Dec. 18, 2013, in Copper Mountain, Colorado.
Amy Purdy: Life is too short not to move forward

"I was so glad that I didn't give up and that I didn't stop just for that one bump-in-the-road period where things weren't going exactly right, because I never would've gotten to what I achieved today and the potential I still have left in me," Patterson said.

Despite her achievements, Patterson said she didn't set out with dreams of medals or national fame.

"At the very beginning, I didn't know what the Olympics were and things like that. I was just doing it because I loved it and it was fun for me," she said.

Carly Patterson of the US takes part in the balance beam event during the Women's Individual All-Around Gymnastics Final in the Olympic Indoor Hall at the Olympic Games in Athens, 19 August 2004.
Amy Sancetta | AP Photo

Patterson's love for the sport is what kept her going when she was training 40 hours a week. She said she was shuffling back and forth between home, training, school, to more training.

"When you're training that hard and that many hours all the time ... it can't come from your coaches that are pushing you or your parents pushing you. It definitely has to come from within and your love, your passion and drive for the sport to be able to work that hard and to be in the gym every single day," she said.

"No matter what, I would still always come back to gymnastics and always remember and have my goals in mind. Nothing was ever too big that made me want to stray away from what I wanted to achieve in life," Patterson said.

Paul Schlader (left) and Jeremy Lyman, cofounders of Birch Coffee
From bar mitzvah boy to coffee kings: How $10,000 turned into $10 million empire