HTTP 418 when batch-pushing git LFS files
(from Infrastructure/Mirrors/mirroring-config#10 (closed))
Some repositories are failing to be mirrored. The specific ones found were:
- Infrastructure/Mirrors/lorry-mirrors/cairographics_org/tar-files
- Infrastructure/Mirrors/lorry-mirrors/files_libburnia-project_org/tar-files
- Infrastructure/Mirrors/lorry-mirrors/ftp_gnu_org/tar-files
- Infrastructure/Mirrors/lorry-mirrors/releases_pagure_org/tar-files
- Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files
- Infrastructure/Mirrors/lorry-mirrors/wpewebkit_org/tar-files
In the logs for the lorry service, the offending response seems to be:
batch response: Client error: https://gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git/info/lfs/objects/batch from HTTP 418
There are git LFS files that are pushed correctly, but they appear to be for
I suspect this is related to #762 (closed)
For completeness, the full log of the mirroring failure is:
DEBUG sys.argv: ['/usr/local/bin/lorry', '--config=/config/lorry.conf', '/tmp/tmpkkwsknd8']
DEBUG current working directory: /
DEBUG uid: 0
DEBUG effective uid: 0
DEBUG gid: 0
DEBUG effective gid: 0
DEBUG environment variables:
DEBUG environment: HOSTNAME=5d3ef3646753
DEBUG environment: PYTHON_VERSION=3.10.2
DEBUG environment: SSH_AUTH_SOCK=/tmp/ssh-Kavs0OoLrRtY/agent.9
DEBUG environment: SSH_AGENT_PID=10
DEBUG environment: PWD=/
DEBUG environment: PYTHON_SETUPTOOLS_VERSION=58.1.0
DEBUG environment: HOME=/root
DEBUG environment: LANG=C.UTF-8
DEBUG environment: GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D
DEBUG environment: SHLVL=0
DEBUG environment: PYTHON_PIP_VERSION=21.2.4
DEBUG environment: PYTHON_GET_PIP_SHA256=7c5239cea323cadae36083079a5ee6b2b3d56f25762a0c060d2867b89e5e06c5
DEBUG environment: PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/2caf84b14febcda8077e59e9b8a6ef9a680aa392/public/get-pip.py
DEBUG environment: PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG Config:
[config]
output =
log = stderr
log-level = debug
log-max = 0
log-keep = 10
log-mode = 0600
dump-memory-profile = simple
memory-dump-interval = 300
working-area = /data/working-area
mirror-base-url-push = ssh://git@gitlab.gnome.org
mirror-base-url-fetch = ssh://git@gitlab.gnome.org
pull-only = False
verbose = True
repack = True
command-stdout = None
command-stderr = None
bundle = never
bundle-dest =
tarball = never
tarball-dest = git-tars
keep-multiple-backups = False
push-option =
bazaar-command = /usr/bin/bzr
check-certificates = True
DEBUG Python version: 3.10.2 (main, Jan 29 2022, 03:24:05) [GCC 8.3.0]
DEBUG Processing spec file /tmp/tmpkkwsknd8
DEBUG Getting: Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files
DEBUG Getting Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files
DEBUG .. updating existing clone
DEBUG .. checking if we need to fetch libglib-testing-0.1.0.tar.xz
DEBUG nothing to do for libglib-testing-0.1.0.tar.xz
DEBUG .. checking if we need to fetch malcontent-0.10.1.tar.xz
DEBUG nothing to do for malcontent-0.10.1.tar.xz
DEBUG .. checking if we need to fetch uhttpmock-0.5.3.tar.xz
DEBUG nothing to do for uhttpmock-0.5.3.tar.xz
DEBUG .. no need to run, nothing to do
DEBUG .. repacking Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files git repository
DEBUG Running: argv=['git', 'config', 'pack.windowMemory', '128M'] kwargs={'cwd': '/data/working-area/Infrastructure_Mirrors_lorry-mirrors_tecnocode_co_uk_tar-files/git-a', 'stdin': <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>}
DEBUG run external command: [['git', 'config', 'pack.windowMemory', '128M']]
DEBUG Command: ['git', 'config', 'pack.windowMemory', '128M']
Exit: 0
Stdout:
Stderr:
DEBUG Running: argv=['git', 'gc'] kwargs={'cwd': '/data/working-area/Infrastructure_Mirrors_lorry-mirrors_tecnocode_co_uk_tar-files/git-a', 'stdin': <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>}
DEBUG run external command: [['git', 'gc']]
DEBUG Command: ['git', 'gc']
Exit: 0
Stdout:
Stderr:
DEBUG .. pushing Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files to mirror server ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git
DEBUG Running: argv=['git', 'push', 'ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git', '+refs/heads/*', '+refs/tags/*'] kwargs={'cwd': '/data/working-area/Infrastructure_Mirrors_lorry-mirrors_tecnocode_co_uk_tar-files/git-a', 'stdin': <_io.TextIOWrapper name='/dev/n
DEBUG run external command: [['git', 'push', 'ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git', '+refs/heads/*', '+refs/tags/*']]
DEBUG Command: ['git', 'push', 'ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git', '+refs/heads/*', '+refs/tags/*']
Exit: 1
Stdout:
Remote \"ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git\" does not support the LFS locking API. Consider disabling it with:
$ git config lfs.https://gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git/info/lfs.locksverify false
Stderr:
batch response: Client error: https://gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git/info/lfs/objects/batch from HTTP 418
error: failed to push some refs to 'ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git'
Error mirroring:
Traceback (most recent call last):
File \"/usr/local/bin/lorry\", line 303, in process_args
self.gitify(name, specs[name])
File \"/usr/local/bin/lorry\", line 475, in gitify
self.push_to_mirror_server(name, active_repo, spec[\"refspecs\"])
File \"/usr/local/bin/lorry\", line 1013, in push_to_mirror_server
self.run_program(
File \"/usr/local/bin/lorry\", line 1039, in run_program
raise Exception(
Exception: git push ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git +refs/heads/* +refs/tags/* failed (exit code 1):
batch response: Client error: https://gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git/info/lfs/objects/batch from HTTP 418
error: failed to push some refs to 'ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git'
ERROR Traceback (most recent call last):
File \"/usr/local/bin/lorry\", line 303, in process_args
self.gitify(name, specs[name])
File \"/usr/local/bin/lorry\", line 475, in gitify
self.push_to_mirror_server(name, active_repo, spec[\"refspecs\"])
File \"/usr/local/bin/lorry\", line 1013, in push_to_mirror_server
self.run_program(
File \"/usr/local/bin/lorry\", line 1039, in run_program
raise Exception(
Exception: git push ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git +refs/heads/* +refs/tags/* failed (exit code 1):
batch response: Client error: https://gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git/info/lfs/objects/batch from HTTP 418
error: failed to push some refs to 'ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git'
DEBUG Total Mirrors failed: 1
DEBUG Done
Processing spec file /tmp/tmpkkwsknd8
Getting: Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files
Getting Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files
.. updating existing clone
.. checking if we need to fetch libglib-testing-0.1.0.tar.xz
nothing to do for libglib-testing-0.1.0.tar.xz
.. checking if we need to fetch malcontent-0.10.1.tar.xz
nothing to do for malcontent-0.10.1.tar.xz
.. checking if we need to fetch uhttpmock-0.5.3.tar.xz
nothing to do for uhttpmock-0.5.3.tar.xz
.. no need to run, nothing to do
.. repacking Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files git repository
.. pushing Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files to mirror server ssh://git@gitlab.gnome.org/Infrastructure/Mirrors/lorry-mirrors/tecnocode_co_uk/tar-files.git
Done