Yeah I remember that. Wonderfully, when I did it there was some bug that made backtracking and coming back later impossible (I forget the details), so that was good!!
It didn't happen again though. There was a world boss or something that I had to nope the fuck away from, but it wasn't at the end of a big castle or anything.
I can't remember at what stage things get unlocked, so I'll be vague to avoid spoilers, but if you're in a thing that has lots of vertical levels and find it difficult, come back later.
Also, Mages and Cyphers have absurd amounts of crowd control, which can make those big group fights trivial if you spec for them. Priests are also very, very strong.