Previously this would always error if ipv6 errored after ipv4 succeeded which was incorrect.
This explicitly tests the order of erroring and also handles the case where ipv6 succeeds with no data by waiting on ipv4.
Fixes #1644 (closed)
@pwithnall I also removed this from flaky optimistically. I'm sure you will yell if proven otherwise.