Phelps is definitely an outlier, but not because he peaked early. He’s an outlier because he was consistently winning multiple gold medals at Worlds and Olympics for 15 years, which is virtually unprecedented in any sport, let alone swimming.
It’s definitely a noticeable trend that elite swimmers now have longer careers and peak a little later than they used to.
It wasn’t unusual for 15 year old girls to set world records at one Olympic games and then never be heard from again. If they were lucky, they may have been able to stretch their career to five years and coincide with two Olympic games. Overtraining, pressure, systematic doping etc, all led to those early peaks. Thankfully the situation is better now.