But you can stop anxiously counting the hours and minutes you spend at work instead of home, says Dr. Tovah Klein.
The quality of the time you spend with your kids matters more than the quantity — and it's the single biggest catalyst for your child's development and future success, says Klein, a child psychologist and author of the book "How Toddlers Thrive," who is director of the Barnard College Center for Toddler Development. Barnard is an undergraduate women's college of Columbia University.
"It's always about the quality of the relationship [and] the quality of the interactions," Klein tells CNBC Make It.
Research bears this out: There's no correlation between the amount of time parents spend with children between the ages of 3 and 11 and those kids' success later in life, a 2015 study published in the Journal of Marriage and Family found.
With that in mind, focus on building meaningful connections with your children during the time you can spend with them, Klein says. That's the key to helping them grow into confident and motivated adults.
"We want our kids to be motivated because success in life, in the business world, in the arts world and anything, has to come from some internal motivation," Klein says.
Here are her top three tips:
Most likely, your kids are "not counting the minutes or the hours" you're spending at work, Klein says. Rather, they're judging the quality of your interactions when you're home.
"Is this somebody who listens to me, who responds to me, who delights in me, who supports me when I'm struggling with emotions or my friend wouldn't play today?" Klein says. "Do I have a parent who listens and doesn't judge me?"
There's no minimum amount of time — besides zero, of course — you need to spend with your children to build those connections, she says. You can still work on it during particularly busy periods at work or while you're traveling, albeit in small doses.
"If, during the week, you only have a half hour with your child, it doesn't have to be big fun, but it might be snuggling when they wake up and getting their breakfast on the table," Klein says, adding that you can always remind them "you'll see them at dinner, and you can't wait to hear about their day."
When you do have free time to spend with your kids, it's easy to try to do too much, Klein says. But you don't always need to take a big day trip or do a special activity.
"It almost always could be said that less is more for children," Klein says. "They just want to be with you. They want to be sitting in your lap reading a book, or your older child wants to throw a Frisbee with you."
Planning and executing a big day out might put more stress on yourself and your kids, ultimately taking the attention away from the sort of interactions that breed a stronger emotional connection.
You might obsess over whether or not your child is having a good time. "[But] what's their favorite part of it?" Klein says. "The car ride there, when they're singing songs with you or ... the five minutes that you walked and just held their hand and listened to them. Children want to be heard and valued by their parents. Valued, meaning 'I'm going to accept you for who you are.'"
You can still plan bigger activities or trips, especially when they help you bond with your kids in new environments and explore their interests. Still, sometimes, a grand gesture is less important than simply getting some quality time with your children in a low-key setting.
If your kids aren't getting enough quality time with you, they'll let you know with their actions, Klein says: "Their behavior changes if they're having a harder time."
In younger kids, this could take the form of more meltdowns or tantrums than usual, including throwing toys or hitting their parents or siblings. Other times, your children might simply withdraw if they don't feel you're paying enough attention to them.
Those signs can occur even when you're at home, which is probably an indicator that you're not being present enough during the time you're spending there, Klein says. One of the biggest culprits: your phone.
"All of us have to become aware that glancing down at our phones, and having all of those notifications when you're with a child, actually takes away that connection and that quality," Klein says.
If your kids' behavior starts suffering during a particularly busy stretch for you at work, try clearing your schedule for a day or weekend, if possible. At the very least, remember to let your children know you're there for them and thinking of them, even when you can't devote a lot of time.
"If you can't put your children to bed, you could come home and give them a little kiss while they're sleeping. And then tell them in the morning, 'When I came in last night, I kissed you and I put your blanket on you,'" Klein says. "That's the reunion that keeps the connection."
Get CNBC's free Warren Buffett Guide to Investing, which distills the billionaire's No. 1 best piece of advice for regular investors, do's and don'ts, and three key investing principles into a clear and simple guidebook.