Sounds like a particular situation with that particular torrent to me. Keep in mind that among the many factors affecting your speed in any torrent is the relative geo-location of yourself and the other peers - it could be, for instance, that those few peers happen to be located close to each other geographically, while you are at a longer distance from them (we could be talking countries, or entire continents). This is just one possible explanation out of many others, and most or all of them wouldn't necessarily imply an anomaly on your side.
Just in case, you could try Tixati's optimizing guide (in the above kind of scenario, it obviously wouldn't change anything):
http://support.tixati.com/optimizing%20tixati