The biggest problem with sea swimming is that the sea temperature lags behind the air temperature, so you get a hot day in May, and think it’s a good idea, and you get halfway in before you realise the sea is still really chilly.
I guess the flip side is that it’s still fairly warm in September / October