Big Data revolutionized the way American politicians win elections. In the process, it broke American politics.
Polarization is no longer just polluting the system — it's paralyzing it. The deepening divide between the right and the left has largely hollowed out the center of American politics, from the politicians who once occupied the large "middle" to the voters who once gravitated to them.
Here's our theory: The reason our lawmakers aren't responding to the center of the electorate is because they've concluded (with ample electoral evidence) that they don't need centrist or swing voters to win.
Why? Big Data — a combination of massive technological power and endlessly detailed voter information — now allows campaigns to pinpoint their most likely supporters. These tools make mobilizing supporters easier, faster and far less expensive than persuading their neighbors.
Of course, this isn't an argument that data itself — be it "good" data or "bad" data — broke the system. It's how the data was misused and manipulated that brought us to a breaking point.
And the result is today's crisis of governing, with the halls of Congress populated by lawmakers who feel beholden not to all their constituents, but only to their supporters.
Political strategists once entered campaigns with a single basic assumption about partisanship: Elections would almost always be decided by about 20 percent of voters who fell somewhere in the ideological middle. By the late 1990s and early 2000s, that sliver of persuadable voters shrunk to about 10 percent.
But, thanks to the advent of what was first known as "micro-targeting," campaign consultants realized that the easiest way to win wasn't to persuade the folks in the middle at all. Instead, data could be used to activate every possible base voter and build a partisan firewall.
Why try to change the mind of one skeptic, the logic goes, when in the same amount of time you could make sure five core supporters commit to go to the polls?
And there is an added benefit to avoiding persuasion: By courting only true believers, candidates don't have to promise the kind of "pragmatism" that avowed partisans label "squishiness."
"Data helps politicians to think of their constituents as not the people who vote and live in their areas, but as the people who vote for them," says Eitan Hersh, a professor of political science at Yale University. "Data helps you dismiss your opponents."
We should make one more important caveat before going on. Big Data is hurting American politics, but that isn't because the data is necessarily wrong, or that Big Data can't be used for good. In fact, a lot of it — despite the cries of "fake news!" and "unskew!" in the last few elections — is accurate.
Yes, many state polls in the Rust Belt woefully underestimated President Donald Trump's performance there, but most national polls weren't far off. The final NBC News/Wall Street Journal poll (and, for that matter, the final RealClearPolitics average) estimated a win for Democrat Hillary Clinton in the low single digits. When the dust settled after the election, she had won the popular vote by 2.1 percent.
Campaign modeling also predicted some of the most consequential results of the election. Both sides foresaw a surge of Latinos, which came to fruition in states like Arizona and Texas. And both campaigns knew that Trump's best shot at victory would come in the Rust Belt.
The story of the 2016 election was there in the data all along. It's just that almost everybody read it wrong.
Polarization isn't new, but it's definitely worse than it was 20 years ago. And thanks to technology and the manipulation of demographic data, those charged with the setting and re-setting of the rules of American politics — both official (i.e. redistricting) and unofficial (i.e. campaign tactics) — have set the stage and conditioned the country for a more permanent polarized atmosphere.
Some might try and make a chicken and egg argument at this point, blaming data and technological advances merely for accelerating an existing trend of polarization. But the fact of the matter is, when given a choice, the majority of Americans still prefer the option of less polarization. Because of the Big Data revolution, that option has simply been erased.
One of the best places to track just how bad things have gotten is the Pew Research Center, which has measured party identity for decades. One way they do it is by scoring the Republicans and Democrats they poll on a 10-item scale of political values.
In 1994, Pew found that 36 percent of Republicans were more liberal than the median Democrat. Conversely, 30 percent of Democrats were actually more conservative than the median Republican.
Twenty years later, each side has retreated much further into their corner, with less than 10 percent in each party overlapping ideologically with the other side. That's compared to about a third of each party during Bill Clinton's presidency.
Essentially, both parties have told moderates: "No need to apply."
Sure, the Republican Party will temporarily tolerate a GOP moderate from Massachusetts if that person can win a senate or governor's seat. Likewise, the Democratic Party may be happy to support a conservative Democrat who can win in North Dakota.
But those types of candidates are not just endangered political species, they're nearly extinct. And the more politicians demand ideological purity during campaigns, the more Americans are empowered to reject any leader for whom they personally didn't vote.
With the poles of each party hardening and moving further apart, are there really enough folks left in the middle for campaigns to reach?
Perhaps the most compelling evidence for these pure "persuadables" can be found in the county-by-county results from the last four elections.
Much has been made of the counties in the United States that flipped from backing President Barack Obama in 2012 to Trump in 2016, but let's take it even a step further. What about those that backed George Bush in 2004, Barack Obama in 2008 and 2012, and Donald Trump in 2016?
The number may surprise you. There are 81 counties with that pattern, including some pretty large ones. Ever wonder why you hear so much about Macomb County, Michigan, during political campaigns? Pinellas County, Florida? Much of New Hampshire?
All in all, 6.6 million people live in those 81 counties.
To put that number in perspective, it's about 5 percent of the overall electorate.
A shrinking middle, yes, but still a substantial one.
Perhaps unsurprisingly, what's left of the center of the electorate is deeply disillusioned with both parties.
According to our archive of NBC/WSJ polls, after the 1992, 2000 and 2008 elections, self-described independents have always had a net positive view of the president's party.
Now, no more.
Our latest survey showed that the Republican Party is 21 points underwater with independents. The Democratic Party is in even more dire straits, with a net negative 25 point deficit with independents.
So why are these indie voters and swing voters feeling so alienated by both parties? We're blaming Big Data.
Thanks to the assumption now among too many political strategists that the easiest and best path to victory is to search for like-minded voters, both parties have essentially stopped courting the vacillating middle entirely.
Political number-crunchers know that the world of Big Data has myriad inputs and outputs. But one of the oldest of the political dark arts has fundamentally been pretty much the same since the beginning of the Television Age: TV ads.
And yet, looking back at how political ads have changed in the last 15 or so years yields some pretty instructive contrasts about whom the campaigns of yore hoped to reach — and why. In fact, there's a pretty good dividing line between how the TV ad wars looked in the BBD era (Before Big Data) and after.
The year is 2000. Michigan Republican Spencer Abraham is in a tough re-election race against Democrat Debbie Stabenow.
Here's a snapshot of the kinds of ads Michiganders were seeing in that very close election.
One of Debbie Stabenow's ads from the same campaign featured this line: "As your senator, I am going to represent the seniors and families of Michigan. Frankly, the special interests can take care of themselves."
As Senate ads go, that's not a lot of fireworks.
But what makes these ads noteworthy are the assumptions that each campaign has made in deciding to air them at all. 1) That voters who are up for grabs care about the future of health care and 2) that their candidate can gain ground by making a better argument. BOTH campaigns concluded that in order to win, they had to debate each other over the same issue.
Same goes for these Missouri Senate ads, also from 2000, in the race between Mel Carnahan and John Ashcroft.
Both campaigns in both states were clearly targeting the same undecided voters.
Now, take a look at what warring Senate campaigns (and in some cases, the third party groups backing them) were talking about fourteen years later.
In Colorado, Obamacare and condoms.
In North Carolina, ISIS and education.
In New Hampshire, immigration and abortion.
Whether or not these were smart ads — or whether they worked — isn't the point here. What's clear is that both sides were fighting completely different wars. None of these ads were about winning an argument between the two parties or two candidates. These ads were about finding the people who agreed that their candidate's issue was the most important one, and getting those voters to show up.
If candidates are campaigning primarily by speaking to like-minded constituents, why would we expect them to govern any differently?
After all, it explains why unanimous Cabinet confirmation votes are almost a thing of the past.
It explains an Obamacare vote with no Republican support, and a likely effort to repeal Obamacare with no Democratic support.
It explains why more than three-quarters of House votes in 2015 and 69 percent of Senate votes were displays of party unity — close to a record.
In fact, according to National Journal's analysis of congressional voting records, ideological overlap between the two parties had almost ceased to exist by 2013.
In that year's rankings, not a single Senate Republican was more liberal than a Democrat in the same chamber, and not a single Senate Democrat was more conservative than one of their Republican colleagues.
On the House side, only two Democrats were more conservative than any Republican, and just two Republicans were more liberal than any Democrat.
Compare that to 2002. A total of seven senators and 137 members of the House fell in the ideological middle ground with voting records somewhere between the most conservative Democrat and the most liberal Republican.
Yet another manifestation of this phenomenon: the plummeting number of split-ticket districts.
Right now, there are just 23 House Republicans representing districts that were also carried by Hillary Clinton. And there are 12 House Democrats in seats carried by Trump. That's only about eight percent of the House Chamber.
Back in 2000? It was almost 20 percent, with a total of 86 split-ticket districts.
Data, fundamentally, is good when it's accurate and used in the aggregate. Big Data analysis and mining is helping to solve huge problems in many sectors, from health care to transportation to the environment and investing. And it's a key element of the journalism we showcase on "Meet the Press" and "MTP Daily" every day. It usually makes us more focused and more forward-thinking. Data makes us smarter.
But the misuse of this tool can make the people represented by all that data seem one-dimensional. It makes messages flat, and it robs voters of the chance to align with a winning argument rather than just picking the person who's picking a fight.
For all its complexity, the digital Big Data revolution has flattened our political campaigns into 1s and 0s, yeses and nos.
During and after elections, candidates and lawmakers have been increasingly incentivized — and enabled by Big Data — to cater to their bases to the exclusion of other voters. Political mapmakers and campaigners have discouraged the grey areas into nonexistence.
But guess what? What Big Data can take away, it can also revolutionize. Imagine if all of these great technological tools were used to create more (small d) democratic competition into the people's House? If we want politicians to practice the art of politics again, then let's create an election system that encourages it by incentivizing candidates to court all available voters, not just their own partisans.
A rethinking of how we use Big Data in our politics could go a long way in fixing what is clearly a disease of governing dysfunction that isn't just impacting the federal level, but is also quickly spreading to state capitols.
By all means, let's use all the digital power and data in the world to communicate, to analyze, to organize, to educate.
But when it comes to message, let's be less clinical.
Let's go analog again.
This story was adapted from a presentation NBC News' Chuck Todd delivered at SXSW in Austin, Texas.