If anything, the people most likely to still be using cash are also the ones most likely to be annoyed by the change.
The difference is, you can always choose which stamp you want.
If you go to the Post Office and ask to buy stamps, they'll usually give you a selection of at least 3-4 different options... the US flag is generally default, they'll often have holiday themed stamps at certain times of year (so maybe a stamp with holly on it around Christmas), sometimes they have "icon" stamps for honoring people (whether feminist icons or political figures or famous musicians or whatever), and sometimes they have other commemorative stamps (like, say, having a stamp with a lunar lander on it to represent the 50th anniversary of the Apollo 11 moon landing).
So if, say, I'm filled with rage about anything involving Christmas, I can ask for something other than Christmas-themed stamps if I buy them. I'm given a choice.
If they officially change who is on the $20, no matter who or what they put on it, people really don't have much choice (other than to not use cash at all) as to which bills they're getting or using. They're stuck taking what they're given.
Personally, I think they should drop the "honor" aspect entirely (because in our current outrage society, no matter WHO they choose to put on a bill, it's going to annoy someone, because literally no politician, public figure, or celebrity we've ever had in this country is completely pure and free from controversy or "problematic" actions). We should just go the Euro route and have the money be mostly symbolism and buildings or eagles or whatever, with no people at all.
I don't disagree with any of this either.
Money is numbers. Make the physical representation of money also have numbers. We don't need people on money.