(I did this before in a 1.080 OG Double IPA that I brewed, only took one pak, no starter, and a good oxygen infusion after the pitch and I turned a 75% attenuation w/ 8.0% ABV) I have a feeling the oxygen infusion is the key to the rapid fermentation.
FYI, a fast start/rapid fermentation is not necessarily indicative of a healthy/optimal fermentation. Under pitching and compensating by over-oxygenating can lead to a fast fermentation, but can increase the production of off-flavored byproducts.
Depending on the OG, 3 days may be typical. Also, the beer has not really been "sitting idle" for a week. As BDawg mentioned, the yeast will use this time to clean up byproducts produced during fermentation like diacetyl. Sorry if I'm just feeding you information you already know, but I've tasted too many beers that suffered from being prematurely racked off the yeast that it's become something of a pet peeve of mine