Along with the car seat, Wal-Mart over the past two years has added "hundreds" of products from about 20 national brands, including Baby Bjorn and Tommee Tippee.
"We are not just focusing on quantity but [on] quality products customers want," said Diana Marshall, vice president of baby for Walmart U.S.
Read MoreAmazon Prime Day: It's not all about Wal-Mart
It's no wonder why the two big-box stores are gunning for a larger share of the baby category. A recent report by Goldman Sachs said the estimated $1 trillion parents spend on children each year is set to grow.
That's because 2014 experienced the first increase in U.S. births in seven years, and it came against the backdrop of a strengthening economy. The fact that the highly sought-after millennial group made up nearly 90 percent of the 1.5 million new mothers last year only sweetens the pot.
"We see the $1 trillion that parents spend on children each year growing, but also shifting as millennials' unique set of values and influencers—aided by technology—leads to new choices," Goldman Sachs wrote.
Along those lines, Wal-Mart has also updated its baby registry to be accessible on its mobile app.
Read MoreCaution lights flashing for back-to-school sales
Under the leadership of CEO Brian Cornell, Target has listed baby as one of its four key categories for growth. The bull's-eye retailer recently rolled out in 1,000 of its stores a one-stop shop dedicated to wellness for new and expectant moms.
Target said that in the first quarter, comparable sales in its so-called "signature" categories—which also include style, kids and wellness—grew more than double the company average of 2.3 percent. They accounted for more than a quarter of the company's sales in 2014.
Wal-Mart does not break out sales for the baby category.