The firing came amid heightened tensions surrounding the North after the U.N. Security Council condemned Pyongyang for its mid-range missile launches last week, just as the leaders of South Korea, Japan and the United States met to discuss the North's arms program.
The area where the artillery exchange occurred is near the so-called Northern Limit Line, drawn up at the end of the 1950-53 Korean War, which the North has refused to recognise. Past clashes between the two navies in the area killed scores of sailors on both sides.
"It's up to the two militaries either to recognize or reject their own claimed line, and challenge the other's - this goes back and forth, so this is probably another episode of that," said Daniel Pinkston of the International Crisis Group.
The two sides are still technically at war as the conflict ended in a mere truce, not a treaty.
Residents on Baengnyeong island were evacuated to bomb shelters, a local government official said by telephone.
(Read more: Why China wants North Korea's rare earth minerals)
South Korea had warned of a strong response if the North attacked its territory.
Monday's drills are the latest incidents in a long line of clashes along the sea border.
A South Korean navy ship was sunk four years ago near the area of the latest infringement. An international team of investigators said it was torpedoed by the North, but Pyongyang denied the charge. Months later the North bombarded a village on South Korean island in the same area, killing the four people.