My daily Starbucks habit costs me $2,300 a year—here's why I refuse to stop

Like most people my age, I don't have $17,000 in a savings account. I'm still paying student loans, my credit-card balance isn't quite zero, and other mandatory expenses can eat away at my income.

Would I take the opportunity to make that kind of money if given a chance? Of course. As long as it didn't require something silly like giving up my daily cup of coffee at Starbucks.

I tracked my spending in 2018 to see how much I'd save if I cut back on my caffeine habit, and how much I could earn if I invested the money instead. Here's what I found.

My regular order, a grande chai tea latte with a shot of blonde espresso, costs $6.04, with tax. That's $6.04 Monday through Friday, or $30.20 for the week. And $30.20 times 52 weeks in a year is $1,570.40.

But that's not exactly everything. I'll get a warm butter croissant, for $2.75, at least three times a week. Plus, another "grande chai tea ..." once a weekend before errands. The croissants cost me a total of $429 a year. The extra coffee once per weekend comes to $314.08.

Grand total: $2,313.48 in 365 days.

Experts may think I have a problem.

While that number could vary slightly when you subtract the free item you occasionally get with the Starbucks mobile-app rewards, it's still more than what 57 percent of Americans had saved in 2017.

If I skipped a year of Starbucks and opted to invest the money, with compound interest, it could grow to $4,551 in 10 years assuming a 7 percent annual return, according to CNBC calculations. In 20 years, it could hit $8,952 and, in 30 years, it could reach a whopping $17,611.

As far back as college, I've needed caffeine to start my day. So, when I hop off the New York City subway on my way into work each morning, I go to my favorite Starbucks for that reason.

It's not just the taste or the many blends of coffee that make me so loyal, though, it's the experience overall: the cozy coffee-shop feel, the rapport with the staff, the ambiance. To me, Starbucks is a place where, even if you walk in alone, you'll feel like everyone knows your name.

But buying Starbucks every day can get pretty costly, especially in a city with one of the highest sales tax rates in the country: 8.88 percent. Compare that to somewhere like Milwaukee, Wisconsin, with only a 5.6 percent sales tax, or Portland, Oregon, with no sales tax at all.

Business guru and "Shark Tank" investor Kevin O'Leary never spends more than $2.50 on coffee. "That is such a waste of money for something that costs 20 cents," he tells CNBC Make It. "I never buy a frape-latte-blah-blah-blah-woof-woof-woof."

Instead, he makes his own. "One cup every morning. It costs 18 cents to make it and I invest the rest." Before making any purchase, in fact, he asks, "Do I really need this?" "Because if I don't buy it, the money is going to be invested and make money every year while I'm sleeping."

Wealth manager David Bach calls this "the latte factor." "We all throw away too much hard-earned money on unnecessary 'little' expenditures without realizing they add up," he writes in "The Automatic Millionaire."

If you ditch a $5 latte, or another small luxury, like fast food or soft drinks, you'd have a considerable amount to work with. Ultimately, he says, "you've got to find something you can give up to get up."

But I don't plan to give up Starbucks any time soon. And other money experts say I don't have to.

For starters, I can't make the same distinctive, high-quality drink at home, or get it for free from work, so I consider Starbucks a fixed cost. When I pay my rent, phone bill and other necessities, and apportion some money toward savings, I budget enough to stay caffeinated until at least the next payday.

That's a responsible first step. If you can cover your mandatory expenses and have a bit left over, experts say, you're free to spend your discretionary income on the things that make you happy.

Besides, "you can't cut your way to growth," says self-made millionaire Ramit Sethi.

He alludes to the much-maligned habit millennials supposedly have of overspending on avocado toast: "You may have seen an article telling you the only way to be a millionaire is to cut back on avocado toast. Now, suddenly, a million millennials in Manhattan started crying because they couldn't go out to brunch six days a week."

But the idea of getting rich by buying less, he says, is "horrible" advice.

After all, "do you know how many avocado toasts you'd have to forego to get just a 20 percent down payment" on a median-priced home? "Is it 100? Is it 1,000? No, it's over 2,500 avocado toasts."

Eliminating small, regular purchases is unlikely to help you save enough money to become wealthy, in other words. Save your energy and focus on something that matters more.

"There's a limit to how much you can cut," Sethi says, "but there is no limit to how much you can earn. You've got your salary. Negotiate it. Think about how much you can grow versus how much you can cut and protect, and that is how you start crafting your own rich life."

That's my philosophy, too.

Whatever your stance on the latte factor, experts agree it's useful to track your spending, so that you know where your money is going. It can be eye-opening: A colleague of mine discovered she spent $700 on Uber and other ride-sharing apps in just one month, and used that knowledge to adjust both her thinking and her spending.

So try recording your expenses in a planner or with a mobile app like Mint or Personal Capital. You may be surprised what you find out, and you may decide to make adjustments.

But it could also assure you that you don't need to feel too bad if you, like me, don't want give up coffee, as long as you're being financially responsible and making sure to save in other ways.

As it turns out, coffee could be the secret to getting along with your coworkers and it might even be good for you, too.

Americans are drinking a daily cup of coffee at the highest level in six years, according to a recent survey by the National Coffee Association.

For now, I'm happy to be one of them.

Like this story? Subscribe to CNBC Make It on YouTube!

Don't miss: Researchers think coffee is good for you but still can't be sure—here's why