Researchers have yet to release any estimate on U.S. productivity loss during this World Cup, but during the last tournament in 2010, InsideView projected that the U.S. economy lost $121.7 million, due to 21 million Americans watching for 10 work-minutes a day.
But such calculations don't quite jive with how most employees work, said Stan Veuger, an economist with the American Enterprise Institute—who took a break from watching the Spain-Netherlands match on Friday to speak with CNBC. "They assume people don't plan around this and just actively stop work to watch," he said. There's no accounting for fans who anticipate game-day distraction, and so schedule their days accordingly.
"The important part is, are you hitting your work goals and commitments?" said Ari Goldberg, chief executive of trend site Stylecaster. If that's happening, he said, he's happy to let employees tune in at one of the office TVs, or stream to their desktop.
Read MorePrivate jet charter to the World Cup: $250K, plus expenses
It helps U.S. workers that the games are in Brazil, which has a relatively small time difference compared to fans watching from Asia, who must tune in overnight for many matches. Even if more workers tune in after the U.S.'s surprise win Monday, World Cup viewers still represent a relatively small base. Just 3 percent of workers said their office is excited about the World Cup, vs. 5 percent for the Olympics, 7 percent for the World Series, 14 percent for March Madness and 53 percent for the Super Bowl, according to a survey from OfficeTeam.
Loss estimates also assume productivity is a constant throughout the workday, said Veuger. Ha! Never mind your lunch break, or worker surveys that tally time spent on social media and other personal pursuits (er, Buzzfeed quizzes?) while on the clock. A little more than 60 percent of workers say they spend more than an hour each week on such tasks, according to a 2012 Salary.com survey. World Cup watching might temporarily replace such distractions, rather than add to them.
"People throw a lot of blame at World Cup," said Jim Belosic, overlord (aka chief executive) at marketing firm ShortStackLab, where employees can watch favorite teams on their own or join company-hosted viewing parties. "But in reality, YouTube, email and texting costs billions of more in lost productivity than World Cup could ever hope."
It's not all about actual productivity loss, said John Challenger, chief executive of Challenger, Gray & Christmas. In some cases, it's perceived loss.
Read MoreBetting on the World Cup—Wall-Street style
"The problem is that a lot of managers still think about productivity, at the gut level, in terms of hours worked and level of focus while working," he said. "If they see an employee watching the game, there's some kind of visceral reaction that this is wrong somehow. This person isn't getting their work done."
With that in mind, fans may want to pre-emptively talk to their bosses about watching during work, Challenger said. Make sure it's OK to tune in. Set out a plan for how you'll meet goals and complete tasks on time. Take a vacation day if necessary.
Read MoreWorld Cup fever in Asia: sick notes and no sleep
Oh, and the boss? Look at the bright side. In-office watching could build camaraderie and foster teamwork, no easy feat in today's offices where many workers don't know each other, said Challenger.
"Let's be honest with each other: People are going to want to watch the World Cup one way or another," said Ware Sykes, chief executive of restaurant app NoWait, whose employees have watched several of the games together, as an office. They also received "referee packs" to card each other's work behavior during the tournament. (No fouls or bad calls so far, he said.)
"It would be naive of us to assume workers aren't interested," said Sykes. "We're using this as an opportunity for fun."
—By CNBC's Kelli B. Grant
Read MoreWorkaholic Americans don't take all their vacation