The Jefferson-Adams rematch was the last gasp of the Federalists as a real political force in the United States; oh, they kept some seats in Congress, and ran for the Presidency, but they never regained their majorities in the former or took the latter. The only interesting presidential election from 1804 to 1820 was 1812, when James Madison ran for reelection against DeWitt Clinton, nephew of Vice President George Clinton, who had died in April. Clinton at least made the popular vote interesting, as Madison barely got over 50%, but 128 to 89 is not that close a race.
Thing is, DeWitt wasn't actually a Federalist. Oh, he ran with Federalist support, but that only accounted for about a third of the vote; the other 15% came from Republicans unhappy with Madison, particularly those who wanted a Northern president. Jefferson and Madison were both from Virginia, as was Washington, and Secretary of State James Monroe, a post that J&M both held before their elections, was a Virginian as well. The Republican Party was the only game in town, and the Federalists only had any success in New England.
The other major reason DeWitt made it a real contest was the War of 1812, which Federalists really didn't like, partly because Madison supported it and partly because they wanted to resolve our problems with Great Britain peacefully. There were a few problems, and they weren't trivial. England didn't believe that people could stop being British and start being American, and their fleet was understaffed, so they came onboard American ships and took anyone who was British, or looked British, and told them "You're in His Majesty's Navy now".
Great Britain* also didn't like us trading with France, since they were at war with Napoleon at the time, and started dicking us over by blockading France, which meant no sweet sweet francs pouring into American pockets. Lastly, a bunch of people wanted to annex Canada and take more Indian land, and the UK really wasn't in the mood to lose more of their North American territory, not to mention trading with the people who caught and skinned all those beavers and such*, so that made things kind of awkward.
Anyway, Federalists didn't like the nascent war, and DeWitt was the only one who was against it (or at least he was against it when he was campaigning in the northeast; back then, you could get away with telling some people one thing and others the complete opposite if they were far enough apart), so he had their vote. What he didn't have was their formal endorsement, mostly because they realized that would do more harm than good since most people didn't like them that much.
DeWitt's campaign in 1812 confirmed the decline of the Federalist Party, and would be echoed in 1824, as all four candidates then were Republicans, reflecting tensions within the party, and the different interests of the South, Northeast, and West. (It was just the West back then, since the country wasn't wide enough to have a distinct left-right middle region.)
*I use the terms England and Great Britain interchangeably in this post, for the sake of variety, but they're not actually the same thing. See CCP Grey's video for more information, which I highly recommend.
*There was a spirited arms trade between the British and the various tribes, most of which ended up being used on frontier settlers in the Northwest Territory, what's now the Midwest.