i mean given that his power was that he cant be hurt by and has complete control over anything he understands, he can understand everything he can see, and he can see all points of history, Im forgiving of kryptonite because I dont see what else you can do.
Maybe don't have that be his deal in the first place since it's writing yourself into a corner.