I just made some changes to my code, to separate the loading and decoding of the audio, and I found out that it is actually the decoding of the audio files that takes so long, the loading is actually fast.
I was not expecting this because the decoding part used to take no time.
I will deal with chrome and firefox dev now, they must have made a change recently.

Uploading is still quite slow sometimes, but it is not a big deal, I was mostly concerned by this slow loading issue.

Thanks for your help anyway, I am sorry for having you losing time on this.