Is how adjectives work. White trash is trash that is white, carrying no implication that any other things that are white are also trash. Specifying that the trash is white implies that it is somehow noteworthy that they are, while the absence of such specification for trash of other races implies that it is not noteworthy in those cases.
The argument can certainly be made that "white trash" would not be racist against non-whites if "black trash," "brown trash," or any similar terms saw a similar presence in colloquial speech, but they don't, so it is.