Anomalie #5094
Opération de mantenance sur base de donnée
Début:
06/01/2021
Echéance:
% réalisé:
0%
Temps estimé:
Description
Lors de la mise à jour vers 3.3.0 et la migration de la base, il y avait un warning concernant la consistance des données, mais un message disait que c'était pas forcément grave.
Après relecture des notes de release je vois qu'une commande "fix-duplicates" a été intégrée à tootctl dans la version 3.2.2 qui était passée inaperçue entre deux release candidates.
Tout le détail est ici : https://docs.joinmastodon.org/admin/troubleshooting/index-corruption/
Historique
Mis à jour par Quentin Gibeaux il y a plus de 3 ans
J'ai procédé à un backup de la db et ait lancé la commande à chaud, j'ai eu des erreurs de status dupliqués, puis j'ai vu qu'il fallait la lancer offline (le helper le précisait pas, seule la page https://docs.joinmastodon.org/admin/troubleshooting/index-corruption/ le précise).
mastodon@pouet:~/live$ RAILS_ENV=production bin/tootctl maintenance fix-duplicates
(…)
Restoring statuses indexes…
Traceback (most recent call last):
24: from bin/tootctl:8:in `<main>'
23: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
22: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
21: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
20: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
19: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:243:in `block in subcommand'
18: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:116:in `invoke'
17: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
16: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
15: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
14: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:159:in `fix_duplicates'
13: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:445:in `deduplicate_statuses!'
12: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/schema_statements.rb:466:in `add_index'
11: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:73:in `execute'
10: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:571:in `log'
9: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
8: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `block in log'
7: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `synchronize'
6: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:581:in `block (2 levels) in log'
5: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:74:in `block in execute'
4: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
3: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
2: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
1: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `block (2 levels) in execute'
/home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `exec': ERREUR: n'a pas pu créer l'index unique « index_statuses_on_uri » (PG::UniqueViolation)
DETAIL: La clé (uri)=(https://framapiaf.org/users/Yahiko/statuses/105508116838601114) est dupliquée.
24: from bin/tootctl:8:in `<main>'
23: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
22: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
21: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
20: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
19: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:243:in `block in subcommand'
18: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:116:in `invoke'
17: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
16: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
15: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
14: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:159:in `fix_duplicates'
13: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:445:in `deduplicate_statuses!'
12: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/schema_statements.rb:466:in `add_index'
11: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:73:in `execute'
10: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:571:in `log'
9: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
8: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `block in log'
7: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `synchronize'
6: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:581:in `block (2 levels) in log'
5: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:74:in `block in execute'
4: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
3: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
2: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
1: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `block (2 levels) in execute'
/home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `exec': ERREUR: n'a pas pu créer l'index unique « index_statuses_on_uri » (PG::UniqueViolation)
DETAIL: La clé (uri)=(https://framapiaf.org/users/Yahiko/statuses/105508116838601114) est dupliquée.
24: from bin/tootctl:8:in `<main>'
23: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
22: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
21: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
20: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
19: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:243:in `block in subcommand'
18: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:116:in `invoke'
17: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
16: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
15: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
14: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:159:in `fix_duplicates'
13: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:445:in `deduplicate_statuses!'
12: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/schema_statements.rb:466:in `add_index'
11: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:73:in `execute'
10: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:571:in `log'
9: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
8: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `block in log'
7: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `synchronize'
6: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:581:in `block (2 levels) in log'
5: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:74:in `block in execute'
4: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
3: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
2: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
1: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `block (2 levels) in execute'
/home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `exec': PG::UniqueViolation: ERREUR: n'a pas pu créer l'index unique « index_statuses_on_uri » (ActiveRecord::RecordNotUnique)
DETAIL: La clé (uri)=(https://framapiaf.org/users/Yahiko/statuses/105508116838601114) est dupliquée.
: CREATE UNIQUE INDEX "index_statuses_on_uri" ON "statuses" ("uri")
mastodon@pouet:~/live$
J'ai donc prévenu les utilisateurs, coupé mastodon, relancé la commande de maintenance, et cette fois-ci elle est arrivée à son terme.
mastodon@pouet:~/live$ RAILS_ENV=production bin/tootctl maintenance fix-duplicates This task will take a long time to run and is potentially destructive. Please make sure to stop Mastodon and have a backup. Continue? Yes Deduplicating accounts… for local accounts, you will be asked to chose which account to keep unchanged. Restoring index_accounts_on_username_and_domain_lower… Deduplicating user records… Restoring users indexes… Removing duplicate account domain blocks… Restoring account domain blocks indexes… Removing duplicate account identity proofs… Restoring account identity proofs indexes… Removing duplicate account identity proofs… Restoring announcement_reactions indexes… Deduplicating conversations… Restoring conversations indexes… Deduplicating custom_emojis… Restoring custom_emojis indexes… Deduplicating custom_emoji_categories… Restoring custom_emoji_categories indexes… Deduplicating domain_allows… Restoring domain_allows indexes… Deduplicating domain_allows… Restoring domain_blocks indexes… Deduplicating unavailable_domains… Restoring domain_allows indexes… Deduplicating email_domain_blocks… Restoring email_domain_blocks indexes… Deduplicating media_attachments… Restoring media_attachments indexes… Deduplicating preview_cards… Restoring preview_cards indexes… Deduplicating statuses… Restoring statuses indexes… Deduplicating tags… Restoring tags indexes… Deduplicating webauthn_credentials… Restoring webauthn_credentials indexes… Finished!
J'ai aussi lancé les deux commandes postgresql de la page ci dessus, qui analysent la db pour vérifier qu'elle n'est pas concernée par le problème, et je n'ai pas vu d'erreur.
=(^-^)=root@pouet:~# su - postgres
postgres@pouet:~$ psql mastodon-production
mastodon-production=# CREATE EXTENSION IF NOT EXISTS amcheck;
SELECT bt_index_check(c.oid)
FROM pg_index i
JOIN pg_class c ON i.indexrelid = c.oid
WHERE c.relname IN ('index_account_domain_blocks_on_account_id_and_domain',
'index_account_proofs_on_account_and_provider_and_username',
'index_accounts_on_username_and_domain_lower', 'index_accounts_on_uri',
'index_accounts_on_url', 'index_conversations_on_uri',
'index_custom_emoji_categories_on_name',
'index_custom_emojis_on_shortcode_and_domain',
'index_devices_on_access_token_id', 'index_domain_allows_on_domain',
'index_domain_blocks_on_domain', 'index_email_domain_blocks_on_domain',
'index_invites_on_code', 'index_markers_on_user_id_and_timeline',
'index_media_attachments_on_shortcode', 'index_preview_cards_on_url',
'index_statuses_on_uri', 'index_tags_on_name_lower',
'index_tombstones_on_uri', 'index_unavailable_domains_on_domain',
'index_users_on_email', 'index_webauthn_credentials_on_external_id'
);
CREATE EXTENSION
bt_index_check
----------------
(22 lignes)
mastodon-production=#
mastodon-production=# CREATE EXTENSION IF NOT EXISTS amcheck;
SELECT bt_index_parent_check(c.oid)
FROM pg_index i
JOIN pg_class c ON i.indexrelid = c.oid
WHERE c.relname IN ('index_account_domain_blocks_on_account_id_and_domain',
'index_account_proofs_on_account_and_provider_and_username',
'index_accounts_on_username_and_domain_lower', 'index_accounts_on_uri',
'index_accounts_on_url', 'index_conversations_on_uri',
'index_custom_emoji_categories_on_name',
'index_custom_emojis_on_shortcode_and_domain',
'index_devices_on_access_token_id', 'index_domain_allows_on_domain',
'index_domain_blocks_on_domain', 'index_email_domain_blocks_on_domain',
'index_invites_on_code', 'index_markers_on_user_id_and_timeline',
'index_media_attachments_on_shortcode', 'index_preview_cards_on_url',
'index_statuses_on_uri', 'index_tags_on_name_lower',
'index_tombstones_on_uri', 'index_unavailable_domains_on_domain',
'index_users_on_email', 'index_webauthn_credentials_on_external_id'
);
NOTICE: l'extension « amcheck » existe déjà, poursuite du traitement
CREATE EXTENSION
bt_index_parent_check
-----------------------
(22 lignes)
mastodon-production=#
J'ai relancé le service mastodon.
NB: ils préviennent qu'il est possible qu'il faille relancer ce fix-duplicates à chaque màj de libc
Mis à jour par Christian P. Momon il y a environ 3 ans
- Version cible changé de Backlog à Sprint 2021 janvier