GitLab

I think it would be quite useful to add high-level support for network profiling, especially of HTTP/HTTPS requests. A lot of applications do them, and don’t do adequate caching, or don’t adequately deal with latency or being offline. If sysprof could expose information about the number of requests, how much is downloaded, how long it takes, cache headers, and TLS status, I think that would go a long way to allowing people to improve the networking behaviour of their applications.

It would be good to support other protocols, but I think HTTP/HTTPS should be the focus, as they are the most frequently used protocols in general applications.

I don’t have a concrete plan for how to implement it, but here are my thoughts so far:

• Don’t reinvent the wheel: Wireshark is really good for packet/protocol analysis, and reimplementing all their packet filters would be a complete waste of effort (and really hard)
• So sysprof needs to capture in pcap format
• The advantage that sysprof has, by being integrated into the process (probably via libsysprof-capture code in libsoup) is that it has access to TLS session keys, so can dump them into the capture so that Wireshark can decrypt HTTPS traffic
• So potentially what we want is a new SysprofCapture frame format to signal the start of a connection (including TLS session keys and information about the peer and the request headers), and then to
• tee each connection into a separate mmapped ring buffer so that ring buffer overflow (and hence dropped packets) on the network connection doesn’t cause ring buffer overflow (and hence dropped packets) on the main sysprof mmapped ring buffer.
• I think the question marks arise around the boundary of what’s implemented in libsoup vs what’s implemented as a wrapper around the POSIX connect() etc. functions. Something needs to be implemented in libsoup to grab the TLS keys; but if we want generic support for other protocols, it needs to be implemented as a tee wrapper around connect()/recvmsg()/sendmsg()/read()/write()/etc.
• Common HTTP problems should be highlighted by the capture:
  • Missing ETag/If-Modified-Since cache headers in requests
  • Use of HTTP rather than HTTPS (almost always a bad sign)

Inspiration: Firefox’s web developer panel.