In commit 7ed81918 things were changed to copy the wrapped boxed after the Python function returned. In case the wrapper is only used during the call there is no need to copy the boxed as the wrapper will be freed anyway.
Add a special case and don't copy the boxed in case the refcount at cleanup is only 1. This should speed up the common case.
thanks to @mathieudu for suggesting this