Project

General

Profile

Bug #15906

Feature #10034: Translation web platform

Feature #11759: Install & configure a fresh weblate on the VM

Frequent memory issues on translate.lizard

Added by intrigeri about 1 year ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
09/03/2018
Due date:
% Done:

40%

Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

Since months icinga2 tells us that translate.lizard regularly goes to "memory is CRITICAL" or "memory is WARNING" state. Please check if that's OK and if yes, tune the memory limits; if not, give the VM more RAM. Goal: avoid false positives in our monitoring so we don't get used to ignoring alerts. Thanks in advance!


Related issues

Duplicated by Tails - Bug #15928: translate.wizard needs more memory Duplicate 09/10/2018

History

#1 Updated by intrigeri about 1 year ago

  • Tracker changed from Feature to Bug

#2 Updated by intrigeri about 1 year ago

  • Duplicated by Bug #15928: translate.wizard needs more memory added

#3 Updated by intrigeri about 1 year ago

See #15928 for info about practical consequences.

#4 Updated by bertagaz about 1 year ago

  • Assignee changed from groente to bertagaz

Being on sysadmin duty I had a look (also because I was annoyed too).

Two cronjobs for weblate were not correctly set:

# Puppet Name: weblate updatechecks
* 6 * * * cd /usr/local/share/weblate; ./manage.py updatechecks --all
# Puppet Name: weblate cleanup
* 4 * * * cd /usr/local/share/weblate; ./manage.py cleanuptrans

Indeed, with such lines, this scripts were run every minutes at that hours. That could explain why every morning the system was lacking memory and OOM killer came into play. I've fixed the related lines in the puppet manifest and deployed it. We'll see in the next days if it fixes the issue.

#5 Updated by intrigeri about 1 year ago

Indeed, with such lines, this scripts were run every minutes at that hours.

Wow, good catch!

#6 Updated by bertagaz about 1 year ago

  • % Done changed from 0 to 40

Two days later, no other notifications about memory issues on translate.li. I'll lower its memory later to get it at a reasonable amount.

#7 Updated by bertagaz about 1 year ago

  • Assignee changed from bertagaz to groente
  • QA Check set to Info Needed

bertagaz wrote:

Two days later, no other notifications about memory issues on translate.li. I'll lower its memory later to get it at a reasonable amount.

So I've lowered it to 1248576KiB, but then we had notifications about memory issues. I've raised it to 1748576KB, and then no new notifications poped up.

One thing I noticed though, is that each time puppet is run in this VM, it runs pip install for every configured python packages. When a weblate cronjob (which can be quite expensive memory wise) is ran at the same time, the memory thus gets sparse.

I wonder if we shouldn't rather use puppet's vcsrepo module for this python packages, and have a puppet exec 'pip install' snippet subscribed to them. This way, puppet would run the pip install command only when the python packages' git are upgraded. What do you think?

#8 Updated by groente about 1 year ago

  • Status changed from Confirmed to Resolved

hey, first of all, thanks a lot for fixing the cronjob fuckup! :)

So I've lowered it to 1248576KiB, but then we had notifications about memory issues. I've raised it to 1748576KB, and then no new notifications poped up.

great, i think with that we can close this ticket.

One thing I noticed though, is that each time puppet is run in this VM, it runs pip install for every configured python packages. When a weblate cronjob (which can be quite expensive memory wise) is ran at the same time, the memory thus gets sparse.

I wonder if we shouldn't rather use puppet's vcsrepo module for this python packages, and have a puppet exec 'pip install' snippet subscribed to them. This way, puppet would run the pip install command only when the python packages' git are upgraded. What do you think?

This whole pip dance will change again with the next weblate upgrade. I'll pass your suggestion on, but for now I don't think the extra overhead really hurts, so let's just leave it as is.

#9 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.10.1 to Tails_3.9.1

#10 Updated by u 10 months ago

  • Parent task changed from #10034 to #11759

#11 Updated by intrigeri 5 months ago

  • Assignee deleted (groente)

Also available in: Atom PDF