I see many cases where "Connection Refused" when T attempts to connect to a Peer - in Trackers.

Is it wise, even necessary, to remove the trackers that are consistently refused?

I'm thinking in terms in overall reducing T overhead by no longer attempting such a connection.

Connection refused likely is the torrent not being registered with a tracker, but it could be other things also. It is a connection to a tracker not a peer. One tracker that I know of goes to refused when it gets bogged down, most time out though. The data sent to trackers is small unless they connect with you, then info from you and them is exchanged. There is no reason to worry about any failed tracker and "removing the load". It is not significant.
You can go deeper and try and determine what the refusal is for, often it is a port error or other error in your client that has likely been configured by you. I do know most trackers have restrictions on what ports they may communicate with. On my old site we had a range and would not connect to users if they were communicating from a port outside that range. To do this highlight a torrent then on the lower group of tabs(depending on view) chose event log. Make sure it is set to 3 or higher and that will give you tracker events.