3. They set prices too high. U.S. retailers often times price their products at a premium in Canada to compensate for their investment in the country, and to account for its steeper operating costs, including a higher minimum wage and cross-border duties. This is a particular concern given how cutthroat the market is in terms of prices, Sozzi said.
Antony Karabus, the Toronto-based CEO of Hilco Retail Consulting advisory firm, said because a many Canadian consumers shop in the U.S., they are aware of stores' prices. As a result, they often refuse to pay prices they think are unreasonably higher. Both Target and J. Crew suffered backlash over this misstep.
Read MoreFormer Sears exec calls for retailer to liquidate
"Prices often have to be higher. The question is how much higher," Karabus said.
4. They take on too much too quickly. Opening 124 stores in one year was too big of an undertaking for Target, which didn't yet understand the intricacies of the Canadian market, Karabus said. Aside from a separate consumer base, these include a different set of government regulations and a different supply chain.
Karabus said the supply chain is particularly difficult to nail down in Canada because the country is so large, and the population so spread out. On top of that, the people differ significantly by city. For instance, in Quebec, the official language is French.
"It was a superhuman effort," he said of Target.
A.T. Kearney's Hillier said a further challenge lies in the constrained real estate market, which makes it difficult to secure top locations in the primary cities. What's more, Canadian provinces are not states, and "they have distinct rules and regulations and ways of engaging consumers."
5. They do not fill a hole in the market. One of Target's biggest issues was that Wal-Mart, Costco and domestic chain Giant Tiger have a huge hold over Canada's discount shoppers, Karabus said. The large penetration of discount retailers also played a role in Sears Canada's potential demise, as consumers looking for cheap products at a one-stop destination would rather shop at Wal-Mart or Costco than Sears, Sozzi said.
Karabus said retailers who will perform well in the country are those who offer Canadian shoppers something different than what is already being offered by a domestic retailer.
"If you have a unique differentiation you will do well," Karabus said.
6. They overlook domestic competitors. There are a number of popular retailers in Canada that already have a stronghold on shoppers' loyalty. Some of the country's most formidable names include Canadian Tire, which sells products from sporting goods to housewares, as well as Giant Tiger, Karabus said. Retailers also cannot disregard the country's major shoe chains—Aldo and Town Shoes, or men's retailer Harry Rosen, he said.
—By CNBC's Krystina Gustafson.