It's a bit more complex than that. The response is damped by the speaker motor working against the 2nd order loudspeaker (not 1st, 2nd) impedance.
The speaker impedance's resistive parts is in series with the source impedance AND the crossover inductor resistance. This total sets the Qts of the speaker. The box design is a very critical function of Qts for any good design (those of you using sealed enclosures can argue with me in another thread) and must combine with the box's acoustic impedance to provide a good low frequency response that does not either just be missing OR alternatively peak terribly. Vented boxes and passive radiator boxes are especially sensitive to this problem.
Source impedance does matter. It also can matter in headphones, and most expect a voltage driven system.