Everyone says cut the morning latte. Here's why I still buy one, without guilt.

Attapan Chan-In / EyeEm | Getty Images

"Skip the expensive coffee and you'd be rich."

We've all heard that advice. It's even the thesis of "The Latte Factor," by self-made millionaire and financial adviser David Bach.

The basic idea is that if you ditch your $4 latte every morning — or any small luxury you spend on each day — you'd have quite a bit of money to contribute towards savings instead. Over the course of a few decades, that money could grow substantially, thanks to compound interest.

I happen to have a weakness for almond milk lattes, particularly the ones sold at the coffee shop a few doors down from my apartment. And while I don't buy a latte every morning, I shell out $5 at Cafe Grumpy at least three times a week.

I'm fortunate enough to not have student loans and put about 30% of my income towards various investments each month. Coffee is an expense I've made room for in my budget and can afford.

Still, that's $60-$80 a month that I could be putting to work and letting accumulate over time. Trust me, I know.

But it's an expense I can completely justify, and here's why. I don't just swipe my credit card and head out the door, drink in hand. I order to stay. Then I pop a squat and spend at least an hour reading, writing in my journal, or setting goals.

I use the time to educate myself, to think of ways I can improve in various aspects of my life. And it's good for me: successful people emphasize the importance of investing in yourself, whether that means reading consistently, taking a course, or exercising.

Do I have to be at Cafe Grumpy to educate myself? Technically, no. And I work on my goals in other ways too, like while training for marathons. But the coffee shop is a productive, distraction-free place for me. It serves as a designated time when I can focus on what I want to focus on.

For that reason, I'm going to keep ordering the $5 latte to stay, no matter how many compound interest charts you show me. And it's going to feel worth it.