There are now five other similar alerts that are over a month old:

Backup steps that would likely be needed:

• Once Archiva content is static, make a snapshot
• Disable regular backup jobs
• Once mirrored content is purged, make a snapshot
• Ensure the backups are no longer rotated / are retained forever

So just to make sure it's 100% clear, would the preference be to keep a backup of the final state of Archiva (minus mirrored artifacts) and also separately keep backups of what's in GitLab?

One other clarifying question, does "50G of space used over the next 5 years" mean:

• Start from 0 and get to 50 over 5 years
• Start from 50 and maintain this over 50

There is T369341: Some VRTS emails to Gmail accounts fail the SPF check but the switch to new outgoing MX servers only happened on July 3rd so may be unrelated.

Since we're unable to reproduce the problem, if anyone is aware of another ticket with a failed delivery it would be helpful to have a link to it here.

To make things event more interesting I successfully responded to a test ticket from a queue with an @wikipedia.org address.

The return email is defined in the "System address" field of the queue settings and is set to different values for different queues.

In T369341#9955747, @LSobanski wrote:

The comment in T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines said:

Since VRTS is sending under <SUPPORT LIST>@wikimedia.org domain, most of what applies to the wikimedia.org domain applies here. Google sees the domain as in compliance with dkim, spf & dmarc.

but the email is actually sent from <SUPPORT LIST>@wikipedia.org

SPF record for wikipedia.org does not include

_cidrs.wikimedia.org

As a side note, https://wikitech.wikimedia.org/wiki/Sender_Policy_Framework should be updated or removed.

Timing-wise this could also be related to moving outbound email to mx-out{1001,2001}.wikimedia.org (https://gerrit.wikimedia.org/r/1051803)

The comment in T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines said:

SPF for services sending mail from their own domains (gerrit, gitlab, phab) was updated in T366113: Update SPF records as needed

A few more questions around backups, to make sure we don't surprise Data Persistence. What are the expectations for backing up the artifacts, would they be easier to restore or regenerate? Is the current ~120GB backed up and if yes, can it be purged after the migration or should it be retained?

Related to T365675: Upgrade GitLab to major version 17

Setting to low priority until we get additional reports.

Changing the partition layout in the correct way would mean reimaging the hosts and if we're doing that we should also upgrade them to Bookworm.

Prior related discussions in T278495: Figure out plan for mailman IP situation which can likely be closed once this task is resolved.