App tests are not fatal, ensuring we test all apps even if one or more tests don't pass. Part of #431