If he owns the whole freehold and leasehold apart from your first floor flat, then there’s probably no legal document that partitions the outside space.
If this is the case it would make things a lot easier.
If I was going to divvy up the curtilage (technical term, there), backed up by legal documents, I would split it left/right, with you having the side access and the left hand side of the garden, and the lower flat having the right, accessible direct from the rear of their flat.
To access the garden you would have to exit your flat at the front and walk around, but that’s better than creating an exit staircase from the rear of your flat.
I think the first thing to do would be to seek clarification on how the outside space is divided up legally at the moment. I suspect that it won’t be at all, and it will all be under his single freehold.