Strange...one WOULD think it would help speed things up if only a bit...
Maybe it has something to do with a sort of Coriolis effect? You know how water will corkscrew down a pipe? Maybe same thing with the vacuum and air and it's fighting the fan somehow... Maybe try holding the vacuum off the sander's dust port just a smidge and at different angles. You might find a position that will work, then fab something up to hold it that way.
Or maybe make up an adapter that goes from sander port to vacuum hose but has some holes drilled in it to allow "fresh" air to enter into stream. Might keep things humming along.
Keep us posted - I'm interested to see what happens.