Investigate resampling without pitch change
If we have a speed rate that's a rational number, we can factorize it and do resampling as a series of interpolate and decimate operations as opposed to the normal "interpolate; change pitch" or "decimate; change pitch".
Of course, this requires that all of the numbers after factorization are reasonably small and that there aren't too many of them, otherwise we'd consume a ridiculous amount of RAM or it would simply take too long, so we have to limit what speed rates we can use.
The simplest case are speed rates like 2 or 0.5. In that case it's a single interpolation or single decimation respectively. We should probably still handle cases like rates 1.25, 1.5 or 1.75 though, as these are pretty common, so the main question becomes: how do we limit the speed rates or what speed rates to allow.