Max-MSP and (free) PureData are softwares of graphic programming conceived for the treatment of real-time musical / video streams.
They allow you to act on the size of the audio buffer and on the size of the computed vectors.
The minima which you could reach always depend on the specification of your soundcard.
Max-MSP: http://www.cycling74.com/
PureData: http://puredata.org/