You can't miss anything other than some heart to hearts if you rush IIRC. It doesn't take that long to do those if you've been using various characters regularly, at least from my own experience. God, remember in the original it took a gazillion years to raise affection?
"Iwata was awesome" - Mr. Nintendo
https://i.imgur.com/krtFHol.jpg