The doctor's withdrawal follows a bombshell New York Times report Wednesday that said Jackson allegedly provided a "large supply" of the opioid Percocet to a White House staffer, and wrecked a government car while intoxicated.
After that story was published online, Jackson told reporters that he "did not wreck a car." Jackson, who is Trump's personal physician in the White House, also said his nomination is "still moving ahead as planned" before walking away from reporters.
Jackson also allegedly wrote himself prescriptions, and after he was caught doing so he asked a physician assistant to provide the medication, the Times reported, citing a Democratic Senate staff summary of alleged conduct by Jackson.
The summary is based on testimony from 23 current and former colleagues of Jackson.
Previously, there were allegations that Jackson oversaw to a hostile work environment and drank alcohol on the job. There were already concerns about his experience and skill set for the job of leading an agency with 377,000 employees before the allegations emerged.
The Veterans Health Administration is the nation's largest integrated health-care system, providing services for 9 million vets at 1,240 health-care facilities.
"The White House should take its time to carefully select and vet a new nominee for VA Secretary," said Dan Caldwell, executive director of advocacy group Concerned Veterans for America. "Considering the tremendous challenges that the last three VA Secretaries have faced, it is important that a capable individual with a high level of integrity is selected for this important cabinet position."
Trump picked Jackson last month after he fired Secretary David Shulkin, who was under scrutiny for alleged ethics violations and for his resistance to privatize veterans' medical care.
Since Shulkin's firing, the interim head of the VA has been Robert Wilkie, undersecretary of Defense for personnel and readiness.
Jackson's appearance before the Senate committee reviewing his nomination, which was supposed to happen Wednesday, was postponed because of the claims.
But the White House aggressively defended Jackson on Wednesday, saying that at least four background investigations had found no areas of concern. Later that night, however, key White House officials were seen huddling with Jackson in the West Wing.
Here is Jackson's statement announcing his withdrawal:
One of the greatest honors in my life has been to serve this country as a physician both on the battlefield with United States Marines and as proud member of the United States Navy.
It has been my distinct honor and privilege to work at the White House and serve three Presidents.
Going into this process, I expected tough questions about how to best care for our veterans, but I did not expect to have to dignify baseless and anonymous attacks on my character and integrity.
The allegations against me are completely false and fabricated. If they had any merit, I would not have been selected, promoted and entrusted to serve in such a sensitive and important role as physician to three presidents over the past 12 years.
In my role as a doctor, I have tirelessly worked to provide excellent care for all my patients. In doing so, I have always adhered to the highest ethical standards.
Unfortunately, because of how Washington works, these false allegations have become a distraction for this President and the important issue we must be addressing – how we give the best care to our nation's heroes.
While I will forever be grateful for the trust and confidence President Trump has placed in me by giving me this opportunity, I am regretfully withdrawing my nomination to be Secretary for the Department of Veterans Affairs.
I am proud of my service to the country and will always be committed to the brave veterans who volunteer to defend our freedoms.
CNBC's Eamon Javers contributed to this report.