Joe Lonsdale began his big data journey as an intern at PayPal, which was identifying Russian and Chinese crime networks committing fraud and reporting them to law enforcement. Problem was, no government agency knew what the data meant or how to use it to build a case.
These were the early days of the post- 9/11 era, and Lonsdale and colleague Alex Karp realized that government—law enforcement and intelligence agencies, in particular—had a problem. It was not attracting the best engineers and not doing its best with the available data, spending tens of billions of dollar on contractors who were below Silicon Valley standards.
Lonsdale, Karp and PayPal founder Peter Thiel set out to solve the problem by creating their own big data company, Palantir Technologies.
Today, the international diplomatic crisis and U.S. public discourse over the intelligence leaks of NSA contractor Edward Snowden have shined a light on leading names in Silicon Valley, including Facebook, Google and Palantir.
Lonsdale has seen the big data and privacy debate from the inside. He once held top-security clearance but has since given it up as it requires "too much paperwork." Though what Snowden did—revealing classified information and working against the United States—was criminal, he said, the conversation resulting from those revelations is one that needed to take place. The truth lies in both extremes of the debate.
(Read More: Investing in the Data Scientist Gap)
"There are really bad people in this world doing really scary things to try to hurt us. We need to stop it," Lonsdale said. "At the same time, we need to be doing more to watch how people are using the data. Both are valid."
To understand an action, analysts must grasp the network around it, and that means exposing data for analysts.
"You should only see what you are allowed to see, and we need to have strict rules," Lonsdale said. "But even with those strict rules, you can understand when bad things are going to happen. we have stopped a lot of attacks by using the data better."
"[Inside Silicon Valley] A lot of people know what's going on," he said. The engineers there "will not tolerate things that don't align with their values. These engineers are there because they want to be. They could work for anyone," he added.
"Google and Facebook should be able to talk more about what they are doing without compromising security. … It is possible to maintain civil liberties while strengthening intelligence and defense. It depends on how people are watching the data," Lonsdale said. "I am an ardent civil libertarian and think this is a really important conversation."
A major part of the solution to the big data-big brother dilemma will come with the government's moving away from its "service" tack, which does not work as a model in the technology sector, Lonsdale said, noting that 1.4 million people have top-security clearance now.
"The biggest thing the government needs to do better is get away from its service-based economic approach," he said. Governments build technologies but end up spending "an order of magnitude more" when they fit them into such a model.
"You end up having a massive number of people working on it," Lonsdale said, adding that products should be off-the-shelf rather than multibillion-dollar field services contracts.
(Read More: Sexiest 21st-Century Job: Data Analyst)
While Lonsdale has strong views about the proper role of big data in intelligence gathering, he has given up day-to-day duties at Palantir and is focused on two other companies he founded: Addepar, which is seeking to apply big data to what he called "broken" collection and analysis in the global investment sector; and Formation 8, a venture capital fund with more than $440 million in assets under management that is targeting big data and energy technology companies that solve societal problems.
"Finance is broken from a data perspective. There are too many closed systems. … It is a giant mess," Lonsdale said.
Addepar provides services to financial and investment entities, including big pension funds, registered investment advisory firms and family offices.
"CalPERS just spent half a billion on IT systems. That's money that should be going to teachers' pensions," he said.
—By CNBC's Eric Rosenbaum, with Richard Brenner.
