Have you tried playing around with your spirits? Some of the tougher fights have environmental hazards that can be countered with the right spirit, and the higher your team’s level compared to theirs the easier the fight becomes.
I’ve just been going into my party and pressing y to autofill the primary and the secondary spirits so not put too much thought into it, but that seems to have done the job so far.