Add marshalling tests to cover cases where C functions return suspicious values

Adds some test cases for out parameters that are left unfilled by the C function, and noncanonical NaN bit patterns returned by C functions.

See: gjs#591 (closed)