Basically, in the Kingdom Hearts universe, your heart determines how you look. So when Riku began to let out Ansem's darkness, it was like he was surrendering his heart to Ansem. So with Ansem's darkness having such a huge grasp on his heart, Riku's appearance changed to look more like Ansem.
Granted, he still looks a bit different from the real Ansem, due to it being a different character model. Not sure if that was intentional, though.
