Oh that so rings a bell! This is a massive bugbear of mine and you’re right to highlight it.
A few years back I joined an organisation and had quite a full on first year handling a fairly major project. As a reward me and Mrs F decided to take a three week holiday interrailing around Europe the following summer, which was fab. When I returned I discovered that nobody had done literally anything at all to move my project forward in my absence. That triggered a descent into depression that eventually ended up with me losing my job a while later.
There’s no easy solutions to your current position, but I’d take it as seriously as you can manage. I’ve been there and it doesn’t get better unless you can change something. Sorry I can’t come up with more than the obvious solutions, but if going back to work is causing you stress then it’s not exaggeration to say you should consider talking to your GP about it.
Best of luck anyway. Hope this isn’t too alarming.