512e9b3b added a call to schedule_pending_close() in the read
callback after the reference to the worker is already gone. In case this was
the last reference to the worker this resulted in a use-after-free.

6f3d57d2 made this more likely to happen because on connection close
the worker cancel action is now async while the reference to the worker
gets dropped right away.

Move the call to schedule_pending_close() before the unref.

Fixes #1686