Should you really blame Verizon if Netflix starts buffering?

Netflix-Verizon blame game

If you're among the many who want to stream Netflix's second season of "Orange is the New Black," just out Friday, and it starts buffering, who should you blame?

In the buffering blame game the answer is complicated—because the Internet isn't just a single pipe. It's more like a spider web with thousands of interconnecting networks.

"It's impossible to know who to blame. Verizon and Netflix know, but they don't want to share any data," says Internet streaming expert Dan Rayburn, executive vice president of and principal analyst at research firm Frost & Sullivan. "There are lots of different places there could be a breakdown."

Netflix and Verizon are locked in a battle over who's to blame for buffering. After Netflix blamed buffering on what it calls Verizon's "crowded network" in pop-up messages to viewers, Verizon shot back with a cease-and-desist letter, threatening legal action if Netflix didn't stop showing those messages and show evidence of its claims. Verizon says Netflix has made a trade-off, choosing lower costs and lower quality.

Read MoreVerizon, Netflix see red over network speeds

So where are the potential problems as a video gets from Netflix to your living room?

Here's how that stream makes its way to your home: Netflix sends data to servers, which are housed by a third-party data center. Then the data are sent to Verizon's hubs, where the data are sent to Verizon's roughly 9 million broadband subscribers.

Once at your home, if you hit buffering issues there could be issues with a Wi-Fi connection, or even your device.

Verizon, along with Comcast, has an "interconnect deal" with Netflix, in which Netflix pays both of them for a direct connection to improve streaming quality. With other Internet providers, like AT&T and Time Warner Cable, where there's no paid deal, there's an extra step: data also are sent through an outside company called a transit provider.

Read MoreNetflix CEO slams Comcast's bid for Time Warner Cable

According to Netflix's ISP Speed Index, its paid deal with Comcast is already paying off for its subscribers—Comcast's streaming speeds have improved by more than half since the two companies struck their deal in February.

Verizon's speeds have only improved slightly, but it only made its deal with Netflix at the end of April, and it can take some time to implement the changes to speed things up.

This weekend, with plenty of binge viewing of "Orange is the New Black" expected, streaming performance will be under the spotlight for Verizon and all the other Internet providers. And we'll see if Netflix decides to change its strategy of "testing" blaming Internet providers for buffering. If Netflix doesn't pull the plug on those messages they'll be hearing from Verizon's lawyers next week.

—By CNBC's Julia Boorstin.

Disclosure: Comcast is the owner of NBCUniversal, the parent company of CNBC and