North Korea fired more than 500 rounds of artillery in a drill on Monday and about 100 of them landed in the water south of a disputed maritime border off the western coast of the peninsula, South Korea's defense ministry said.
The South returned fire with more than 300 rounds aimed at an area north of the so-called Northern Limit Line, ministry spokesman Kim Min-seok said, calling the North's action "a test of the South's determination to defend the naval border".
(Read more: North Korea fires 2 more missiles amid US drills)
The firing came amid heightened tensions surrounding the North after the U.N. Security Council condemned Pyongyang for its mid-range missile launches last week, just as the leaders of South Korea, Japan and the United States met to discuss the North's arms program.
The area where the artillery exchange occurred is near the so-called Northern Limit Line, drawn up at the end of the 1950-53 Korean War, which the North has refused to recognise. Past clashes between the two navies in the area killed scores of sailors on both sides.
"It's up to the two militaries either to recognize or reject their own claimed line, and challenge the other's - this goes back and forth, so this is probably another episode of that," said Daniel Pinkston of the International Crisis Group.
The two sides are still technically at war as the conflict ended in a mere truce, not a treaty.
Residents on Baengnyeong island were evacuated to bomb shelters, a local government official said by telephone.
(Read more: Why China wants North Korea's rare earth minerals)
South Korea had warned of a strong response if the North attacked its territory.
Monday's drills are the latest incidents in a long line of clashes along the sea border.
A South Korean navy ship was sunk four years ago near the area of the latest infringement. An international team of investigators said it was torpedoed by the North, but Pyongyang denied the charge. Months later the North bombarded a village on South Korean island in the same area, killing the four people.