The second finding is in a relatively newer area of the brain called the anterior insular cortex, which is "active during bodily discomfort and unpleasant emotional states, such as pain, anxiety and disgust," the researchers wrote in the paper.
It's associated with responding to gut feelings or signals from the body, said Alec Smith, a researcher at Caltech and the lead author of the paper.
Study participants who were more successful traders turned out to have higher activity in that area and more frequently sold before market peaks.
Read MoreWhat we really think about that wellness program
"It may be that high-earning traders are saying, 'Oh wow, I'm getting very excited, my heart's beating faster, maybe I should step back and figure out if what we're all doing here makes sense,'" Smith said by telephone. "They're selling before the peak."
The research builds on earlier findings about these areas of the brain, said Brian Knutson, a neuroscientist at Stanford University who was not involved in Monday's study.
"This research shows that neural signals not only correspond to but predict important financial events in an experimental market," Knutson wrote in an email. "If this experimental market operates according to the same principles as larger markets, the findings open up the possibility of a 'minority report' approach to predicting market events."
He noted that only some traders have the anterior insula activity that predicts a market crash. "Who are these individuals and do they do better in real life? (e.g. Warren Buffett). The findings provide neural evidence that emotion can move (experimental) markets."
Still left to study is whether different people do better in different markets, Knutson said. "For instance, bullish traders may do poorly in a rising market, but better in a falling market, and vice versa."
It also isn't clear whether the signals these more successful traders get are conscious, Montague said, but the effect is strong.
"It really is uncanny how tuned that group is," Montague said. They feel the same exuberance as other participants in the pricing bubble, he said, but they respond to signals that allow them to get out before it bursts. "It begs the question: Are these people made or are they born?"
The answer to that is beyond the scope of this research, he said. And, Smith notes, "I don't think we think the Fed is going to run out and buy a bunch of fMRI machines" to use brain signals to better regulate the market.
But the research does provide a glimpse into the biological differences between traders—and supports the investment thesis of one of the world's richest people, Warren Buffett.
"Our experimental results support the first part of his advice to a surprising degree," the researchers wrote. "Wiser traders who begin selling when their insula is active (indicating discomfort) sell a few periods before the peak to traders with the highest 'greed,' measured by increased nucleus accumbens-buying sensitivity."
—By CNBC's Meg Tirrell