It's not easy being a gold bug and it may get even harder.