testmatrix.c: use G_SQRT2 insead of M_SQRT2

Math implementations are platform dependent, and for example msvc requires a define before enabling the constants [1]

Use G_SQRT2 implementation which is independent instead.

[1] https://docs.microsoft.com/en-us/cpp/c-runtime-library/math-constants?view=msvc-160