Serena and Venus Williams are out of the US Open women's doubles after losing to Hao-Ching Chan and Maya Joint in a third-set tiebreaker.