It drops some points, but the only CE you really NEED equipped is Poster Girl, if you can't do it with Gudaos equipped then yeah switch CEs. Of course the Berserker one is probably easier and I don't think there's a big enough disparity in mat drops between these two.
The later 40 AP ones do have better drop rates though and get harder depending on class advantages ---