Don't assume CPU_ISSET will return 0 or 1

At least on musl libc with x86_64/aarch64 this is not the case, which results in g_get_num_processors returning very large values such as 65535.

This should be backported to 2.80. Found via gstreamer test suite (failing test).

Ref !3784 (merged)