Is weight loss really just calories in vs calories out??

supposed to be yes, but with some caveats:

fat is worth 9 calories per gram, whereas carbs and protein are worth only 4.

there is a "window" that you have to aim for but can't see. above a certain amount you gain weight. below a certain amount you go into starvation mode and your body starts holding on to more fat and consumes organs and muscle instead. you need to aim between those two points.

this window shrinks and slides down as you lose weight.

exercise makes the window larger and moves it up.

exercise generally helps you lose more fat and less muscle. without exercise a lose/gain weight cycle can tend to shift you towards a higher fat% over time. with exercise, it should tend to shift you towards a lesser fat% over time.