In 2012, I decided to quit my six-figure job in investment banking and retire at 34. I had amassed a net worth of about $3 million that generated roughly $80,000 in investment income per year.
For seven years, I lived a charmed life in early retirement with my wife, who also retired from her finance job three years after I did. Together, we earned roughly $250,000 in passive income streams per year — mostly from dividend-paying stocks, interest from savings, municipal bonds, and rental income.
We traveled to more than 20 countries. I spent time doing things I enjoyed, like coaching a high school tennis team and writing on Financial Samurai, the personal finance website I started in 2009.
Fast forward to the end of 2019, and my wife and I are no longer comfortable living the early retirement lifestyle, especially in a big, expensive city like San Francisco.
According to a report from the California Association of Realtors, you now need a minimum household income of $309,400 to afford a median-priced home in San Francisco, which is around $1.6 million. So even with our annual passive income of $250,000, we're still short nearly $60,000 per year.
In order to own our existing home without a mortgage, we could try to sell our other investments. But that would create a much higher net worth allocation towards real estate than desired. (Our goal is to keep real estate expenses to no more than 35% of our income.) Furthermore, selling our investments would trigger a tax liability on capital gains.
Retiring from early retirement
When I left my job (and even got a nice severance package that paid out all my deferred stock and cash compensation), I thought I could retire and be set for life. I was wrong, and I'm not afraid to admit it.
If my wife and I don't take action soon, we'll no longer be able to live comfortably off of our passive income streams in San Francisco.
Here are the main reasons I now need to retire from early retirement:
1. We had a child.
My wife and never planned on becoming parents, but that changed in 2017, when we were blessed with a baby boy.
Retiring without children is like a walk in the park compared with retiring with children. In addition to costs for diapers, clothes, toys, baby food, and occasional babysitting, our biggest concern is paying for our son's education, especially now that he's in preschool.
In San Francisco, the public school system starting in kindergarten is based on a lottery system, so even if you pay property taxes, your child isn't guaranteed a spot in your neighborhood schools. Most parents are forced to pay big bucks to send their kids to private school, but even paying $30,000 or more doesn't guarantee admission to top-rated private schools.
Don't get me wrong: Fatherhood has been the most rewarding experience in my life — and we're going to do whatever it takes to support our son's needs and give him access to as many opportunities as possible.
2. I underestimated how low interest rates would go.
I'm a believer of "low interest rates for life," but I didn't think the 10-year bond yield would ever drop to below 1.5% in 2019. I thought we'd stay around 2.5%.
Now, instead of only needing $2 million in additional capital to generate $50,000 at 2.5%, I need $2.5 million in capital to generate the same $50,000 in passive income at 2%. At 1.5%, the required capital to generate $50,000 in passive income is over $3.3 million. Seeing such a large shift in the goalpost when you don't want to take more investment risk is disconcerting.
The only way I benefited from the recent interest rate decline was by refinancing my mortgage down to 2.6%. My cash flow is now about $13,000 greater per year. While the increase helps, it still doesn't offset the expected investment income decline.
3. Rising health insurance premiums.
In 2019, the monthly premium on our health insurance was $1,820. But last month, I received notice that our premium will rise to $1,940 in 2020. That's $23,280 per year in annual health insurance premiums, excluding any co-pays and co-insurance.
This will be an even bigger problem if we decide to have more children later on, because the 2020 amount would go up by another $440 (to $2,560) per month.
I've finally reached a point where paying so much in health insurance premiums feels like highway robbery, especially since we're a healthy family with no pre-existing conditions. While I understand that it's our duty to help subsidize others who are less healthy, the cost has become untenable.
4. The bliss of early retirement didn't last as long as I thought it would.
Retiring early improved my quality of life in ways I never would have imagined: I was able to fully commit to fatherhood, my relationship with my parents strengthened, my aging slowed, and I even became more self-sufficient.
After a few years, however, the bliss of taking a permanent vacation went away. I started getting the itch to do something more productive than playing tennis and sleeping in.
Also, when you have to start paying $2,000 per month for preschool, not using your six to eight hours of free time finding ways to cover that cost just doesn't feel right.
How I plan to retire again by 2022
In order to get to $309,400 a year — and retire again by 2022 — I've decided to spend some time focusing on entrepreneurship. For too long, I've treated Financial Samurai like a hobby.
The website has grown to amass more than one million readers per month. My immediate goal over the next sixth months is to grow traffic and boost advertisement revenue by partnering with relevant financial institutions and creating more products and services (like my severance negotiation book).
I also plan to apply for jobs at various tech, media, and financial firms. With the income from a full-time job, subsidized healthcare, and other benefits, I can accelerate my goal of accumulating extra capital to generate enough investment income.
Finally, I will continue investing in real estate in America's heartland, where valuations are cheaper, cap rates are higher, and job growth is stronger.
If all else fails, we will relocate to Honolulu in 2022, when our boy is eligible for kindergarten. The median price for a single-home family in Honolulu is $835,000 — or 47% lower than in San Francisco. Meanwhile, the price of a private school education (around $25,000) there is also 30% to 45% less.
Always plan for an unknown future
For those who want to retire early, my advice is to always plan several years ahead. Set a date for your exit plan and do whatever it takes to meet your goal.
My biggest mistake was failing to consider how my lifestyle and needs would change — and what to do when those changes take place. Downturns do happen. You can't plan for every variable, but the more prepared you are, the greater your chances for achieving financial freedom.
Now if you'll excuse me, it's time to turn in my early retirement membership card and get back to the salt mines!
Sam Dogen worked in investing banking for 13 years before starting Financial Samurai, a personal finance website. He received a B.A. in Economics from The College of William & Mary his MBA from the University of California in Berkeley. Sam has been featured in Forbes, The Wall Street Journal, The Chicago Tribune and The L.A.Times.
Don't miss:
Like this story? Subscribe to CNBC Make It on YouTube!