From fb09d7b80c5dd341e0e43c03260347342297c458 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Fri, 18 Aug 2023 13:03:46 +0200 Subject: [PATCH 1/3] introduction/languages: List C as a supported language The overview mentions that C is the primary language, but if the reader just glances at the table then it looks like only C++ is supported. --- source/introduction/languages.rst | 4 ++++ source/introduction/overview/libraries.rst | 2 ++ 2 files changed, 6 insertions(+) diff --git a/source/introduction/languages.rst b/source/introduction/languages.rst index 99fa014..ecde232 100644 --- a/source/introduction/languages.rst +++ b/source/introduction/languages.rst @@ -25,6 +25,10 @@ The following languages can be used to write GNOME applications. - Project - Documentation - Notes + * - C + - Core libraries + - :ref:`Core libraries ` + - Language extensions supported by GCC and Clang compilers can be used. * - C++ - `gtkmm `__ - `Documentation overview `__ diff --git a/source/introduction/overview/libraries.rst b/source/introduction/overview/libraries.rst index c7f27e5..26c0ccc 100644 --- a/source/introduction/overview/libraries.rst +++ b/source/introduction/overview/libraries.rst @@ -1,6 +1,8 @@ Libraries ========= +.. _core-libraries: + An overview of GNOME platform libraries. User Interfaces -- GitLab From 3e94e3a7b20799696343051c96c7832f05f3ac83 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Fri, 18 Aug 2023 13:11:31 +0200 Subject: [PATCH 2/3] introduction/languages: Document recommended build tool --- source/introduction/languages.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/source/introduction/languages.rst b/source/introduction/languages.rst index ecde232..9cf0c37 100644 --- a/source/introduction/languages.rst +++ b/source/introduction/languages.rst @@ -49,3 +49,13 @@ The following languages can be used to write GNOME applications. - `Vala `__ - `API References `__ - Vala is a programming language which wraps GNOME libraries and outputs C code. Applications which use Vala include `Games `_, `Boxes `_, `Clocks `_ and `Gitg `_. + +Build systems +------------- + +Each GNOME module must have build instructions that describe how to build and install +the source code. + +We recommended writing build instructions using `Meson `_, a +stable tool which supports all major programming languages, and has a built-in +module of `GNOME-specific helpers `_. -- GitLab From eed6b760336b043907c57301148c1ab656a6ac50 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Fri, 18 Aug 2023 20:28:24 +0200 Subject: [PATCH 3/3] Add initial documentation on testing --- source/guidelines.rst | 3 + source/guidelines/testing.rst | 17 ++++ source/guidelines/testing/component-level.rst | 80 +++++++++++++++++ source/guidelines/testing/end-to-end.rst | 87 +++++++++++++++++++ source/introduction/languages.rst | 2 + 5 files changed, 189 insertions(+) create mode 100644 source/guidelines/testing.rst create mode 100644 source/guidelines/testing/component-level.rst create mode 100644 source/guidelines/testing/end-to-end.rst diff --git a/source/guidelines.rst b/source/guidelines.rst index 5250503..6d20235 100644 --- a/source/guidelines.rst +++ b/source/guidelines.rst @@ -11,6 +11,8 @@ and contributing to the GNOME platform. The following guidelines are available: accessible apps and software. * :doc:`Localization `: standard practices for making software that can be used across languages and locales. +* :doc:`Testing `: standard approaches for testing GNOME + software. * :doc:`Maintainer `: recommended practice for GNOME module maintainers. * :doc:`Developer documentation style `: the @@ -32,5 +34,6 @@ those who are using the GNOME platform to create software of their own. guidelines/programming.rst guidelines/accessibility.rst guidelines/localization.rst + guidelines/testing.rst guidelines/maintainer.rst guidelines/devel-docs.rst diff --git a/source/guidelines/testing.rst b/source/guidelines/testing.rst new file mode 100644 index 0000000..a450a9f --- /dev/null +++ b/source/guidelines/testing.rst @@ -0,0 +1,17 @@ +Testing Guidelines +================== + +Contents +-------- + +The testing guidelines include the following sections: + +* :doc:`Component-level testing ` +* :doc:`End-to-end testing ` + +.. toctree:: + :maxdepth: 1 + :hidden: + + testing/component-level + testing/end-to-end diff --git a/source/guidelines/testing/component-level.rst b/source/guidelines/testing/component-level.rst new file mode 100644 index 0000000..0fd0370 --- /dev/null +++ b/source/guidelines/testing/component-level.rst @@ -0,0 +1,80 @@ +Component-level testing +======================= + +GNOME is developed as a set of modules. Each module corresponds +to a specific `Gitlab project `_, +and each module has a :ref:`build system `. + +The first step of testing a change is testing it at the module +level. Modules should provide tools to help with this: + + * A **testsuite**, which can be run locally by the developer. + * A **Gitlab CI pipeline**, which ensures the testsuite passes + for every proposed change before it is merged. + +All modules should provide instructions on how to run the testsuite. +For modules that use Meson, you can run ``meson test``; see +`Meson's documentation on testing `_ +for more information. + +Read on for detailed documentation on component-level testing. + +Writing unit tests +------------------ + +For C, C++ and Vala projects, you can use the +`GLib testing framework `_ +to help you write unit tests. + +It's possible to use other testing tools, and to integrate those with +Meson so that `meson test` runs the entire test suite. Look at existing GNOME +modules for examples of good practice. + +Remember that unit tests should be quick to execute, and should run be as +isolated as possible. Avoid accessing the internet, D-Bus services, and +any files outside of the module's own source tree. + +For more guidelines, see: +``_. + +Writing integration tests +------------------------- + +There is an intermediate stage between unit testing and :doc:`end-to-end`, +which can usefully be done at the component level. + +Some tools you might find useful for writing integration tests are listed below: + + * `Appium `_ and the `Selenium webdriver for + AT-SPI `_. Together these + allow you to write integration tests that verify your app's behaviour using + *AT-SPI*, which is a control interface used by accessibility tools such as + screen readers. + * `python-dbusmock `_, a tool + for creating mock objects on D-Bus, with ready-made mocks of some common + system services. + +Setting up CI pipelines +----------------------- + +Gitlab CI is a powerful automated testing tool. Each GNOME module defines a +**pipeline** which contains one or more **jobs**, and runs against a specific +Git commit of the project. + +Each job executes on a **runner**, which will fetch a +container image and execute commands inside it. The job is considered to have +passed if the commands succeed. + +Full documentation of Gitlab CI is available `here +`. For inspiration, you can look at +pipelines of core GNOME modules, such as +`gnome-shell `_, +and `gtk `_. + +Here are some resource that can help you build CI pipelines: + + * `Freedesktop CI templates `_. + These help with building persistant container images for your CI jobs. + * `DevOps with Flatpak `_. + A guide for app developers on building a Flatpak app bundle as + part of your CI pipeline. diff --git a/source/guidelines/testing/end-to-end.rst b/source/guidelines/testing/end-to-end.rst new file mode 100644 index 0000000..b0e704a --- /dev/null +++ b/source/guidelines/testing/end-to-end.rst @@ -0,0 +1,87 @@ +End-to-end testing +================== + +The GNOME desktop consists of over one hundred modules. These +interact with an operating system built of many more components. +After :doc:`component-level` is done, you are ready to integrate the new +version of the module into an operating system and perform end-to-end testing. + +GNOME does not distribute an operating system ready for end-users - we +leave that to dedicated OS distributors - but we do practice continuous +integration, and as part of that we produce testing-only operating system +called `GNOME OS `_. + +GNOME OS has first-class support for installing applications using Flatpak. +This means most apps can be easily tested in context against the latest version +of GNOME. That is documented in the first section of this guide. + +Libraries and system services are more complicated. GNOME OS is built as a +single, atomic base image - it's not a general-purpose OS and does not have a +package manager. The second part of the guide discusses how to test these +types of module. + +End-to-end testing for apps +--------------------------- + +Manual testing +~~~~~~~~~~~~~~ + +You can test an app on any Linux distribution that supports Flatpak. This is +useful, to ensure your app works against the GNOME release used in that distro. +When you test on GNOME OS, you're testing against the latest version of GNOME +that has not yet been released. This mean you can catch bugs early, and you're +also helping to test unreleased changes that target the next version of GNOME. + +The first step is to install GNOME OS into a virtual machine. Follow the +instructions on the `GNOME OS website `. to set up a +virtual machine. If you already have a VM, update it to the latest snapshot. + +Now, use ``flatpak install`` in the VM to make your app available, and +ensure that it works as expected. + +Automated testing with openQA +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Manually testing your app gets boring fast. GNOME provides an instance +of `openQA `_ that can automate tests inside of GNOME OS. + +Currently openQA is only available for core apps that ship inside GNOME +OS. The tests are maintained in the +`GNOME/openqa-tests project `_ +where you can also find links to documentation. + +End-to-end testing for system components +---------------------------------------- + +Building a custom GNOME OS image +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you want to test major changes in GNOME, you can create a custom GNOME OS +image. + +For instructions, see the `Building System Components guide +`_. + +Rebuilding GNOME OS images on your local machine can take a long time. You +can also push your changes to a branch of +`gnome-build-meta `_ and +use the CI to build a new image. + +Once you have your new image, deploy it to a VM as before. + +Building an overlay with systemd-sysext +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can avoid building a new OS image, and instead build an overlay which you +apply using +`systemd's 'sysext' feature `_. + +For details on how to do this, see the blog post +`Developing in GNOME OS: systemd-sysext `_. + +Automated testing with openQA +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tests for the overall GNOME OS system are developed in the +`GNOME/openqa-tests project `_ project. +Check there for documentation. diff --git a/source/introduction/languages.rst b/source/introduction/languages.rst index 9cf0c37..00c3df6 100644 --- a/source/introduction/languages.rst +++ b/source/introduction/languages.rst @@ -50,6 +50,8 @@ The following languages can be used to write GNOME applications. - `API References `__ - Vala is a programming language which wraps GNOME libraries and outputs C code. Applications which use Vala include `Games `_, `Boxes `_, `Clocks `_ and `Gitg `_. +.. _build-systems: + Build systems ------------- -- GitLab