To be clear, the threat is not seen necessarily as being the result of nefarious behavior, but rather sloppy habits like reusing passwords or writing them on paper and carrying them in your wallet. IT professionals are all too aware that common behaviors like this that can lead to hackers getting into a company's systems, Smith said.
Read More Microsoft fixes 19-year-old Windows bug
Of the IT professionals who responded in the survey, 77 percent said that employees are the weakest link in the security infrastructure. The number is even bigger in financial services firms: 81 percent.
Curiously, despite recent high-profile security breaches, a clear majority of IT professionals—64 percent—said they won't change any of their planned security-related infrastructure buying in 2015, though an even larger majority (89 percent) said they plan to provide more employee education next year.
Smith said educating employees on smart procedures is useful but is rarely enough by itself, so the fact that most IT pros don't plan to update next year could be a concern.
Read More Chinese hack US weather systems, satellite network: Wash Post
IT pros "won't get the desired effect they're hoping for," he said.
Other findings: 29 percent of IT pros at small- and medium-sized businesses have no plans to use the cloud for storage. Fully 75 percent of pros cite the cloud as their largest security concern.
The poll was conducted by Spiceworks Voice of IT on behalf of CloudEntr in September. The study collected 438 surveys from IT professionals at companies with 20 to 499 employees in industries related to financial services, manufacturing, professional services, government and non-profits.
CloudEntr is owned by Dutch security firm Gemalto.