Joe Lonsdale began his big data journey as an intern at PayPal, which was identifying Russian and Chinese crime networks committing fraud and reporting them to law enforcement. Problem was, no government agency knew what the data meant or how to use it to build a case.
These were the early days of the post- 9/11 era, and Lonsdale and colleague Alex Karp realized that government—law enforcement and intelligence agencies, in particular—had a problem. It was not attracting the best engineers and not doing its best with the available data, spending tens of billions of dollar on contractors who were below Silicon Valley standards.
Lonsdale, Karp and PayPal founder Peter Thiel set out to solve the problem by creating their own big data company, Palantir Technologies.
Today, the international diplomatic crisis and U.S. public discourse over the intelligence leaks of NSA contractor Edward Snowden have shined a light on leading names in Silicon Valley, including Facebook, Google and Palantir.
Lonsdale has seen the big data and privacy debate from the inside. He once held top-security clearance but has since given it up as it requires "too much paperwork." Though what Snowden did—revealing classified information and working against the United States—was criminal, he said, the conversation resulting from those revelations is one that needed to take place. The truth lies in both extremes of the debate.
(Read More: Investing in the Data Scientist Gap)
"There are really bad people in this world doing really scary things to try to hurt us. We need to stop it," Lonsdale said. "At the same time, we need to be doing more to watch how people are using the data. Both are valid."
To understand an action, analysts must grasp the network around it, and that means exposing data for analysts.
"You should only see what you are allowed to see, and we need to have strict rules," Lonsdale said. "But even with those strict rules, you can understand when bad things are going to happen. we have stopped a lot of attacks by using the data better."