CI: Don't build everything with warnings=fatal

Build only pangomm with warnings=fatal.
Select latest released version of libsigc++3.
5 jobs for master in 13 minutes and 41 seconds (queued for 4 seconds)