VMA organizers had announced only three hours before Sunday's show began that Beyonce would perform. The singer arrived on the red carpet accompanied by the mothers of three African-American men who have died at the hands of U.S. police in the past two years.
Her politically charged "Formation" beat Adele's 2015 comeback single "Hello," Drake's "Hotline Bling," Justin Bieber's "Sorry" and Kanye West's sleeping nude celebrity look-alikes in "Famous" for video of the year.
She also won best female video and prizes for choreography, editing, and long form video in the awards that are voted on by fans.
Where Beyonce was angry and bold, Spears, 34, dressed in a sparkly gold leotard and matching over the knee boots was slinky retro pop. It was the "Toxic" singer's first time on the VMA stage since her humiliating 2007 performance when her personal and professional life was falling apart, and follows the release of a new album last week.
Rihanna, 28, sang 13 of the biggest hits in her 13-year career, including "Rude Girl," "Diamonds" and "Only Girl in the World." She was presented with the Michael Jackson Video Vanguard award by Canadian rapper Drake - with whom she has often been romantically linked.
"She's someone I've been in love with since I was 22 years old. She's one of my best friends in the world. All my adult life, I've looked up to her even though she's younger than me," said Drake.
Fifth Harmony, the five-girl group that won TV talent show "The X Factor," took home 2 "Moonman" awards.
Kanye West, who has often hijacked the live VMA show, presented a new video for his single "Fade." West preceded the video with a rambling, improvised speech that touched on poverty, his wife Kim Kardashian, his long-running feud with Taylor Swift, and violence in black neighborhoods.
"We are undeniably the influencers, the thought leaders," he said, before screening what he called a "piece of my art."
Many of pop music's biggest names, including Justin Bieber, Britain's Adele and Taylor Swift were absent on Sunday.
Follow CNBC International on Twitter and Facebook.