Whether you remember it or not, you may have once granted Facebook permission to store the contacts in your phone's address book. If you did, Facebook is continuously uploading these contacts to its servers.
You can find out if your phone is doing this, and stop it pretty easily.
Here's how to do that on Facebook for Android:
How to do it on Facebook for iPhone and iPad:
How to remove contacts that have already been uploaded: