×

Big data dump: Using information 'like a car engine'

As lives become increasingly interconnected via the Internet, there is a vast pool of data generated with each interaction. This information, obtained via Facebook, tweets, music streaming and random Web searches, is mined by companies for new ways to reach the consumer – and tailor their products to suit the individual's needs.

Whether searching for the best travel deals, monitoring health, playing video games or calling a cab, technology is shaping the world around us. The perpetual stream of so-called "big data" is processed in real time, and lends insight to the mundane activities of every day life.

It's also worth an increasing amount of money that will be worth more than $41 billion in three years, according to estimates from the International Data Corporation. That amount is placing the onus on companies to make better judgments about what to do with the information they collect.

"We believe companies should have access to data and make smart decisions about their customers," says Neha Narkhede in an interview with CNBC.

Narkhede, along with Jay Kreps and Jun Rao, formed the Confluent software company. The trio met as former software engineers at LinkedIn, created an open-source solution called Apache Kafka in 2010, which enhances software that manages lives streams of data from websites, applications and sensors.

"Think of Kafka as an engine of a car," says Kreps. "Most people don't want an engine, they want a whole car, but for businesses to use the engine, they need training to drive the car."

Read MoreManager: Big data is the new bright spot in tech

Among the thousands of businesses who now use Kafka are Netflix, Airbnb, Hotels.com, Uber, Spotify, Cisco and Goldman Sachs. Brick and mortar business use the software to help solve their quality control issues and build better products.

For its part, Netflix uses Kafka to power its instant movie recommendations engine. Major retailer Wal-Mart uses Kafka for real-time analysis of sales information to maximize revenue. Kafka is now widely adopted by financial service firms to reduce credit card fraud.

"What's different about us from others is that we focus on data as it occurs, so that the data is more of a stream than a warehouse. That's a big key difference," says Neha.

As a former employee of the Confluent's co-founders, LinkedIn was not only an early investor in the Palo Alto based software start-up, but they also have the world's largest deployment of Kafka.

At LinkedIn, Kafka currently processes 1.1 trillion pieces of data per day for the professional social network, with potential for that number to grow in even bigger numbers. While there is no cost to use open source software, Confluent provides premium versions with added capabilities for a fee.

"We train companies and give them the software they need so that they have support internally required to drive their business to where they want to go," says Kreps who is also Confluent's CEO.

Yuri_Arcurs | iStock | Getty Images

Since Confluent's launch in 2014, the company has raised nearly $31 million in venture capital funding. It has been backed by Benchmark, LinkedIn, Data Collective and Index Ventures.

Modeled as a subscription based business, the tiny big-data start-up of less than 30 employees is generating a lot of buzz in Silicon Valley. Confluent is considered to have found technology's next big idea based on the free technology of open source code which is available to download by anyone to modify or enhance.

To improve their access to billions of bits of data, more companies prefer open-source software because the innovation cycle is faster and more flexible to use. More organizations, including local governments, are adopting open-source alternatives to commercial software.

Read MoreNetflix CEO: All TV will be Internet in 20 years

Source: NetflixOpenSource | YouTube

Kafka was built five years ago but since then Neha says there's been a huge shift in how companies want to use data.

Neha, who is Confluent's Head of Engineering, praises open sourcing. She credits the technology with enabling the world's best engineers to solve the biggest problems—and help formulate the best answers.

"There's really a ton of innovation happening in core technologies that is helping business become more effective and save money," she said. "When you have a complicated piece of software, you want to make sure things work easy, and it's hard to tackle problems if you don't know how things work."

On the Money airs on CNBC Sundays at 7:00 pm, or check listings for air times in local markets.