Odd 400 response with https GET 302 redirecting to http
I ran into this issue when developing a test for a Pulp PR (https://github.com/pulp/pulp_container/pull/1301 "Implement flatpak index view") that uses Flatpak to download data via libsoup 2. I was only able to reproduce the issue in that Pulp PR's automated GitHub CI pipeline, but not locally, which made it significantly harder for me to come up with a hypothesis of what exactly could be broken. However, I'll share here all the information I could gather, in the hope that we can find a working solution going from that:
In the test code, a flatpak install
invocation tries to download a data file from a location that typically involves a 302 redirect (which libsoup apparently handles transparently for the caller). The automated CI runs that test code in various environments. For one environment ("test (s3)"), it consistently fails with flatpak install
reporting Error: Failed to install net.fishsoup.BusyBoxPlatform: Server returned status 400: Bad Request
. For other environments, it consistently succeeds. One thing that appears to set the failing environment apart is that the original request goes to a https https://pulp/...
URL while the 302 redirect goes to a http http://minio:9000/...
URL (and host pulp
resolves to 172.18.0.3
while host minio
resolves to 172.18.0.2
).
The relevant Python test code was
# See <https://pagure.io/fedora-lorax-templates/c/cc1155372046baa58f9d2cc27a9e5473bf05a3fb>
# "lorax-embed-flatpaks.tmpl: Run the flatpak-install under dbus-run-session" for the need for
# dbus-run-session to avoid "error: Cannot autolaunch D-Bus without X11 $DISPLAY":
subprocess.check_call(
"OSTREE_DEBUG_HTTP=1 dbus-run-session flatpak -vv --ostree-verbose --user install"
+ " --noninteractive pulptest net.fishsoup.Hello 1>&2",
shell=True,
)
It was temporarily modified that way to get maximally verbose output. (You can just ignore that dbus-run-session
wrapper, it should not be relevant here.) The relevant tail of the combined stdout/-err output (see the 1>&2
above) of the failing test, as reported by the CI, was
Installing runtime/net.fishsoup.BusyBoxPlatform/x86_64/2023
F: Loading https://pulp/v2/pulptest/oci-net.fishsoup.busyboxplatform/manifests/sha256:92581c1b8ac813ccc52d68c0d53ffc39d0d587dca80ef647d94a52c9ccfc7f8d using libsoup
Error: Failed to install net.fishsoup.BusyBoxPlatform: Server returned status 400: Bad Request
> GET /v2/pulptest/oci-net.fishsoup.busyboxplatform/manifests/sha256:92581c1b8ac813ccc52d68c0d53ffc39d0d587dca80ef647d94a52c9ccfc7f8d HTTP/1.1
> Soup-Debug-Timestamp: 1689838411
> Soup-Debug: SoupSession 1 (0x564d2d703220), SoupMessage 1 (0x564d2dbe9370), SoupSocket 1 (0x564d2dc39400)
> Host: pulp
> Accept: application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.index.v1+json
> Authorization: ***
> User-Agent: Flatpak 1.10.7
> Connection: Keep-Alive
< HTTP/1.1 302 Found
< Soup-Debug-Timestamp: 1689838411
< Soup-Debug: SoupMessage 1 (0x564d2dbe9370)
< Server: nginx/1.14.1
< Date: Thu, 20 Jul 2023 07:33:31 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Location: http://minio:9000/pulp3/artifact/92/581c1b8ac813ccc52d68c0d53ffc39d0d587dca80ef647d94a52c9ccfc7f8d?response-content-type=application%2Fvnd.oci.image.manifest.v1%2Bjson&response-content-disposition=attachment%3Bfilename%3Dsha256%3A92581c1b8ac813ccc52d68c0d53ffc39d0d587dca80ef647d94a52c9ccfc7f8d&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIT2Z5TDYPX3ARJBA%2F20230720%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20230720T073331Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=952e096bde2c59a94fc6881e8632d3022aa3d7a0e0db0a2e9df52a8d5f60912b
< X-PULP-CACHE: MISS
< Allow: GET, PUT, HEAD, OPTIONS
< Docker-Distribution-Api-Version: registry/2.0
< X-Registry-Supports-Signatures: 1
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Cross-Origin-Opener-Policy: same-origin
< Correlation-ID: d9e45678dc5e46cfa93dd7c7195f01d4
< Access-Control-Expose-Headers: Correlation-ID
< Strict-Transport-Security: max-age=15768000
<
> GET /pulp3/artifact/92/581c1b8ac813ccc52d68c0d53ffc39d0d587dca80ef647d94a52c9ccfc7f8d?response-content-type=application%2Fvnd.oci.image.manifest.v1%2Bjson&response-content-disposition=attachment%3Bfilename%3Dsha256%3A92581c1b8ac813ccc52d68c0d53ffc39d0d587dca80ef647d94a52c9ccfc7f8d&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIT2Z5TDYPX3ARJBA%2F20230720%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20230720T073331Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=952e096bde2c59a94fc6881e8632d3022aa3d7a0e0db0a2e9df52a8d5f60912b HTTP/1.1
> Soup-Debug-Timestamp: 1689838411
> Soup-Debug: SoupSession 1 (0x564d2d703220), SoupMessage 1 (0x564d2dbe9370), SoupSocket 2 (0x564d2dc2c0b0), restarted
> Host: minio:9000
> Accept: application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.index.v1+json
> Authorization: ***
> Connection: Keep-Alive
> User-Agent: Flatpak 1.10.7
What is odd is that we see Server returned status 400: Bad Request
reported first, before seeing details about the GET
requests, and that we don't see the response for the second (redirected) GET request at all (but which is presumably the 400 response reported above). What might explain that is that, for one, Flatpak apparently calls into libsoup in some asynchronous fashion, and, for another, the libsoup logging goes to stdout while the other output goes to stderr (though that shouldn't be relevant with the 1>&2
?).
The issue went away when I replaced the system's flatpak-1.10.7-1.el8.x86_64
with a locally built installation of recent Flatpak trunk, but which uses curl instead of libsoup. The issue reappeared when I configured that locally built Flatpak to use the system's libsoup-2.62.3-4.el8.x86_64
instead of curl. The issue also still appeared when I configured that locally build Flatpak to use a locally built installation of the latest libsoup 2.74.3 (unless I made a mistake with the LD_LIBRARY_PATH
fiddling required to set that up). (All the "locally built" in this paragraph refer to installations built as part of the test scripting run on the automated CI.)
From the above, my hypothesis is that the libsoup 2 code somehow garbles the redirected URL (http://minio:9000/pulp3/artifact/92/581c1b8ac813ccc52d68c0d53ffc39d0d587dca80ef647d94a52c9ccfc7f8d?response-content-type=application%2Fvnd.oci.image.manifest.v1%2Bjson&response-content-disposition=attachment%3Bfilename%3Dsha256%3A92581c1b8ac813ccc52d68c0d53ffc39d0d587dca80ef647d94a52c9ccfc7f8d&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIT2Z5TDYPX3ARJBA%2F20230720%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20230720T073331Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=952e096bde2c59a94fc6881e8632d3022aa3d7a0e0db0a2e9df52a8d5f60912b
) when sending it out, in a way that makes the server consider it invalid and send a 400 response.
I have not yet looked into how to set up some libsoup test code to verify my hypothesis. Is there maybe a relatively easy way to do that (within the existing libsoup tests/
infrastructure, perhaps)?