Bezos took a few days to consider that advice and tried a mental exercise. He said he realized "the best way to think about it was to project my life forward to age 80."
That way, he could make a decision that "minimized [his] regrets." Bezos said it became obvious that he should leave his job to start Amazon.
"If it failed, I would be very proud when I was 80 that I tried," he says, according to TechCrunch.
When Amazon first launched in 1994, the company sold only books, and Bezos would drive packages to the post office in his 1987 Chevy Blazer. The company expanded to offer music and videos, and finally Bezos asked his customers what else they wanted to buy from the online retailer.
Today, Amazon sells everything from electronics to household goods and furniture. That wouldn't be possible if Bezos hadn't taken that leap more than 20 years ago.
"After much consideration, I took the less safe path to follow my passion, and I'm proud of that choice," he said in a 2010 baccalaureate address at Princeton.
You can either choose a life of "ease and comfort," Bezos told his brother during the Summit Series interview, or a life of "service and adventure."
"When you're 80," he said, "you'll be proud of the latter."
Like this story? Like CNBC Make It on Facebook.
How Jeff Bezos' 'two pizza rule' can help you hold more productive meetings
How Amazon founder Jeff Bezos went from the son of a teen mom to the world's richest person