Skip to content

Speed up CI pipeline proposal

Dunja Lalic requested to merge dlalic/librsvg:ci_cleanup into master

Motivation:

  1. Complaints about the slow pipeline (cc @svenfoo )
  2. Less and less C, more an more Rust

Approach:

  1. Fail fast, save electricity. For example no need to run anything if cargo check fails, no need to do the full make check if unit tests fail. Use case is newcomers and project contributors probably won't fiddle with C, but Rust.
  2. Rust development woes: not having to worry about making both tests green and fmt and clippy happy at the same time, the pipeline will run them in parallel and complain separately.
  3. Caching that doesn't slow the builds. Some trial and error showed caching ./target produces 400 MB artifacts, for very little speed gain as compiling is reasonably fast. Cache is currently only consisted of cargo crates cache and shared across branches depending on Cargo.lock or Makefile checksum.
  4. Tarball and cross distro can be run manually on request and perhaps scheduled (nightly/weekly) builds. One can still trigger them by clicking play on the separate jobs or an entire step (or per docs in the PR), screenshot below.

The above is all up for discussion.

Notes:

  • didn't mean to rewrite everything nor invalidate any previous work, just personally found the YAML anchors unparsable hence utilized other options where possible
  • GitLab does support build matrix but it's under a feature flag which requires an admin to fire up Rails console, hence it was easier to DIY the "Cross Distro"

Tried and gave up on:

  1. Test reports
  • They require converting Tap to JUnit with questionable added value. Demo here.
  • The Perl Tap library chokes on custom output, but a Python lib doesn't and it probably doesn't make sense to bloat the docker images with any due to questionable added value.
  • Junit
  1. Code Quality
  • Created a small Rust project to consume clippy output (cargo clippy --message-format=json &> foo.json) and convert it to desired output
  • Neither setting artifacts:reports:codequality nor copying the config from the example project didn't work
  • Screenshot reports are under a feature flag

TLDR:

Screen_Shot_2020-10-05_at_22.07.00

Screen_Shot_2020-10-05_at_22.09.09

Screen_Shot_2020-10-05_at_22.09.35

The above run for 5-6 minutes.

Screen_Shot_2020-10-05_at_22.05.08

The whole pipeline runs around 20 minutes (with or without cross distro in parallel).

TODOs:

  • gather feedback
  • put back all the features
  • remove debug output
  • figure out if C cache makes a difference > it doesn't that much
  • check if pages job works > it' does's complicated
Edited by Federico Mena Quintero

Merge request reports