Adams retired from public service after his bitter loss to Jefferson. He held a grudge against the Federalists in Hamilton's faction of the part they played in undermining him ahead of the election of 1800 and refused to help. With Hamilton dead and Adams refusing to get off the bench, the Federalist Party was leaderless and disorganized and was unable to mount an effective organized campaign in 1804. This was made worse by Pinckney's insistence on not openly campaigning for office, which undermined the Federalist strategy of trying to win South Carolina, the entire basis for his candidacy.
That said, a major scandalous accusation was presented in the Federalist papers before Hamilton's death alleging that Jefferson held one of his slaves, Sally Hemings, as a concubine at Monticello and that her Son, Tom, was a dead ringer for the President. These accusations were alluded to in 1800 but presented with more evidence and force in the 1802 article. (you can read the article for yourself here: https://www.encyclopediavirginia.org/_The_President_Again_by_James_Thomson_Callender_September_1_1802). There is in general an attempt to make Jefferson appear decadent and amoral portraying this relationship as tawdry and indicative of Jefferson's overall lack of ethics.
Pinckney's service as Ambassador of France is somewhat tainted by his involvement in the XYZ affair in which French diplomats demanded bribes in order for negotiations regarding France's collection of their debts. While they made him popular in the run-up to the Quasi-War the mood of the country has softened on France following the Louisiana Purchase.