|
|
## Pre-requisites
|
|
|
|
|
|
**Damned Lies** is a Python Web application based on [Django](https://www.djangoproject.com/) to manage data, templates and the user interaction. It is used to coordinate the translations done by contributors distributed in language teams. It is connected to our software forge, [gitlab.gnome.org](https://gitlab.gnome.org) to send the translated [`.po`](https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html) files to the associated projects.
|
|
|
|
|
|
For the CSS, it uses a custom theme based on [Boostrap](https://getbootstrap.com/) (Currently version 5.X [30/11/2022]). This theme is called [`gnome-bootstrap-theme`](https://gitlab.gnome.org/Infrastructure/gnome-bootstrap-theme) and also hosted on [gitlab.gnome.org](https://gitlab.gnome.org).
|
|
|
|
|
|
It benefits from multiple system tools provided by the [`gettext`](https://www.gnu.org/software/gettext/) project and [`git`](https://git-scm.com/), in order to send translations to the forge.
|
|
|
|
|
|
## Environment
|
|
|
|
|
|
**Damned Lies** is deployed in a Kubernetes environment hosted by GNOME/Red Hat that relies on the [OpenShift technology](https://www.redhat.com/en/technologies/cloud-computing/openshift).
|
|
|
|
|
|
As a consequence, the application is deployed in containers that follow the OCI principles and are compatible with `podman` and `buildah` (`docker` as well). The files to build the container image are located in the `containers` directory. Images are built with `buildah`.
|
|
|
|
|
|
It’s a two-step building. First, a runtime is built with all the necessary dependencies to run the application. The runtime is based on [Fedora OCI images](https://hub.docker.com/_/fedora), the target being the old stable version. This runtime is used both in the Gitlab CI progress and to deploy the application. Then, to create the production images (*testing*, *staging* and *production*), the build process is based on these runtimes.
|
|
|
|
|
|
**Note**: the user ID in the `buildah` files has a specific UID/GID (`1001010000`) which is managed by OpenShift in order to run in an unprivileged environment. See the [documentation](https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids) for more information. This user (the one that runs the process) is called `l10n` inside the container.
|
|
|
|
|
|
## Image configuration
|
|
|
|
|
|
The **Damned Lies** configuration is based on a few files. They are Jinja2 template you can render with the `containers/production/render_configuration_templates.sh` shell script. This script expects an argument, the basename of the `.json` file to render the Jinja template (here, `production` or `test`).
|
|
|
|
|
|
1. `entrypoint.sh.jinja2`: the image entrypoint, a shell script that runs the Django commands to prepare the application and that replaces (see the `sed` commands) temporary variables in the `local_settings.py` files with ones found in the deployment environment. All those variable are set in the OpenShift environment in order to pass each pod sensitive data or on the fly configuration.
|
|
|
1. `damnedlies/local_settings.py` in the **Damned Lies** repository. This file is used to configure the application itself. The template of this file is located in `containers/production/configuration/local_settings.py.jinja2`. In this file, all the configurations that **must not** be visible from outside are prefixed with the `$` sign.
|
|
|
1. `httpd.conf.jinja2`: The Apache HTTPd configuration file. A web server runs in the container to serve the Web application with [`WSGI`](https://wsgi.readthedocs.io/en/latest/what.html), which is the [recommended method to deploy a Django application](https://docs.djangoproject.com/en/4.1/howto/deployment/).
|
|
|
|
|
|
## Container repository
|
|
|
|
|
|
Once built (after Gitlab CI tests and check pass), the image is pushed to [quay.io/organization/gnome_infrastructure](https://quay.io/organization/gnome_infrastructure/damned-lies) where all OCI images used by the GNOME project land.
|
|
|
|
|
|
There is a trigger in the OpenShift configuration that waits for changes in the registry in order to deploy brand-new images.
|
|
|
|
|
|
## OpenShift
|
|
|
|
|
|
There are, in OpenShift, three environments, all synchronised through the Gitlab CI process:
|
|
|
|
|
|
1. [**testing**](https://damned-lies-testing.apps.openshift4.gnome.org/) (image: `damned-lies:testing`), synchronised with the `develop` branch. It includes all the latest developments and is used to show end-users the new requested features and to test visual changes, for instance.
|
|
|
2. [**staging**](https://damned-lies-staging.apps.openshift4.gnome.org/) (image: `damned-lies:staging`), when a change occurs on `master` (a merge or a new commit for instance), the last version is pushed to this environment, once the change has been validated, it goes to production.
|
|
|
3. [**production**](https://l10n.gnome.org/) (image: `damned-lies:latest`), synchronised with `master` but triggered manually after the changes on `staging` has been validated.
|
|
|
|
|
|
OCI images for **staging** and **production** are strictly the same ones and based on the same `master` branch.
|
|
|
![Deployment configs (DC) used by Damned Lies and Cron Jobs (CJ) that run maintenance tasks.](openshift_dl_deployment.png)
|
|
|
|
|
|
### Filesystem access
|
|
|
|
|
|
* Files distributed by the **Damned Lies** application are located in a folder set in the Django `settings.py` file (`DATADIR`). This directory mush be shared through OpenShift to offer persistent storage (PVC - Persistent Volume Claim in OpenShift).
|
|
|
* A `/home/l10n/.ssh` file is necessary. If contains the SSH configuration to connect to the [gitlab.gnome.org](gitlab.gnome.org) application and send translations. (Secret Volume in OpenShift). It contains two files: `config` (`man 5 ssh_config`) and a private key (referenced in the `config` file). For instance:
|
|
|
```text
|
|
|
Host gitlab.gnome.org
|
|
|
Hostname ssh.gitlab.gnome.org
|
|
|
User git
|
|
|
IdentityFile ~/.ssh/ssh-privatekey
|
|
|
StrictHostKeyChecking no
|
|
|
UserKnownHostsFile=/dev/null
|
|
|
```
|
|
|
So the private key will be called `ssh-privatekey`.
|
|
|
|
|
|
* A `home/l10n/.gitconfig` file is expected, in order to declare the identity of the commiter (the application itself). It could be, for instance:
|
|
|
```text
|
|
|
[user]
|
|
|
email = gnome-sysadmin@gnome.org
|
|
|
name = GNOME Translation Robot
|
|
|
```
|
|
|
|
|
|
### Cron jobs for maintenance tasks
|
|
|
|
|
|
There are two jobs that are done regularly in order to maintain consistency in the Damned Lies data.
|
|
|
|
|
|
1. `manage.py run-maintenance` (every day). As described in `stats/management/run-maintenance.py`, it cleans not activated accounts and obsolete ones, cleans old actions and removes tar files.
|
|
|
|
|
|
1. `manage.py update-stats --non-gnome` (twice a day). Regenerates the statistics for non GNOME projects (externally hosted).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|