When is a vacation not a vacation?

Justin Horrocks | E+ | Getty Images

Summer means vacation for the vast majority of Americans. But more and more often, those days off aren't downtime.

Some 61 percent of Americans plan to work during vacation this year, according to a new survey by Harris Interactive. That's up from the 52 percent found in a similar survey last year and 46 percent in 2011.

That's a lot of leisure converted to labor.

(Read more: Most Americans plan to travel for summer vacation)

Are these working vacationers just happy, energetic warriors? Not exactly. A third of the survey respondents who plan to work are not pleased about it, and 4 percent intend to deal with their frustration by throwing something. (Six percent plan to use their vacation to update their résumé.)

People aren't just working while they're on vacation but on weekends, too. The latest AmericanTime Use Survey from the Bureau of Labor Statistics found that a third of the highest-paid workers (those earning $1,291 a week or more) worked an average of 3.6 hours on a Saturday, Sunday or holiday.

(Read more: No paid vacation? You must be an American)

The job market may be improving, but many people still feel nervous about uncoupling from the office completely.

People are also staying plugged in to avoid being overwhelmed when they return to work, according to Ellen Galinsky, president of the Families and Work Institute.

"It's less the big bad people making us do it [than ourselves worrying] what will happen when we don't," she said.

The Harris Interactive survey suggests things aren't likely to change anytime soon. The propensity to work during vacation is greatest for Generation Y: 73 percent of respondents ages 18 to 34 said they expect to work during their time off this year.

(Read more: Job growth posts large gain)

While staying plugged in may make things a bit easier your first day back in the office, it's not really a good thing, Galinsky said. People need relaxation and a change of scene to recharge.

"Why do you get your best ideas in the shower, or when you are walking the dog?" Galinsky asked. Stepping back "gives you a chance to pull together things that as we're rushing through we don't necessarily see together."

People become more productive, she said, "having a time where you don't feel like you're on a treadmill—or at least you're on a treadmill of your own choosing at a spa."

—By CNBC's Kelley Holland. Follow her on Twitter @KKelleyHolland.