[ENG-9044] Add manage command to resync preprint dois v1 by Vlad0n20 · Pull Request #11617 · CenterForOpenScience/osf.io

Vlad0n20 · 2026-03-02T19:39:37Z

Ticket

ENG-9044

Purpose

Changes

Side Effects

QE Notes

CE Notes

Documentation

cslzchen

Looks good overall. In addition to my questions/comments:

Can we add logs of the output of the logs for you local run?
We should also work with CE to test this command with a copy of production DB.

cslzchen

Looks good 👍

Make sure we test this against a copy of prod DB before running this on prod.
- If there are errors, need look further on why they fail, and probably another fix.
- Test if the default batch size is good, how long does it take, and also see how many times we need to repeat this.
Make sure whoever uses this command knows that we need to repeatedly run this command.

cslzchen · 2026-03-13T16:44:51Z

+            help=(
+                'Maximum number of preprints to process per run (default: 500). '
+                'The command processes the first N eligible preprints and exits; '
+                're-run the command to continue with the next batch.'
+            ),


Oh, I see. This is an OK alternative to the loop I suggested. Just need to make sure whoever runs this command is aware of the fact that they need to keep running this command until none exists. Cc @adlius for your input on this.

antkryt · 2026-05-04T09:23:44Z

LGTM

mkovalua

LGTM, @Vlad0n20, please change target branch to current one feature/pbs-26-9, I have no permission to do it

adlius · 2026-08-06T14:09:33Z


        user.date_last_logged_in = timezone.now()
        user.external_identity[provider][provider_id] = 'VERIFIED'
+        if provider.lower() in OSFUser.SOCIAL_FIELDS:


Is this part of this ticket?

adlius

So I have a couple of suggestions. Since Crossref has a limit of 10000 submissions from a single user, might as well set the default batch size to 1000 and no rate limit. Instead let's run the task every five minutes so that we can send about 12000 requests per hour.

Also there are some commits from other branches that is in this PR.

adlius · 2026-08-06T14:10:26Z

        assert len(user.system_tags) == 1

-        tag = Tag.all_tags.get(name=tag_name, system=True)
+        tag = Tag.all_tags.get(name=tag_name.lower(), system=True)


Same here. Is this part of this ticket?

adlius · 2026-08-06T14:19:16Z

+    return qs
+
+
+def resync_preprint_dois_v1(dry_run=True, batch_size=500, rate_limit=100, provider_id=None):


The rate limit of 100 is perhaps to restrictive. According to conversations with Crossref, they have a limit of 10000 pending items for each user.

A 500 batch size is also perhaps too small.

adlius · 2026-08-13T08:12:16Z

+
+        if rate_limit and not record_number % rate_limit:
+            logger.info(f'Rate limit reached at {record_number} preprints, sleeping {RATE_LIMIT_SLEEP}s')
+            time.sleep(RATE_LIMIT_SLEEP)


I am afraid this would clog the celery worker when this is run as a celery beat task.

brianjgeiger

Drive by:

What queue are the tasks going onto? The default queue? We should probably ensure that they go on the Low Priority queue to keep from clogging the pipes too much.
If we have a cron job that happens every five minutes, how are you going to run this in a dry-run scenario to make sure it's going to work before the cron job runs? We might want to rethink our strategy here.

adlius

Some issues found during local testing.

adlius

Some more issues to fix.

adlius · 2026-09-10T16:26:51Z

    def create_identifier(self, preprint, category, include_relation=True):
        if category == 'doi':
-            metadata = self.build_metadata(preprint, include_relation)
+            metadata = self.build_metadata(preprint, include_relation, include_unversioned_doi=True)


One more issue found during local testing: on an unmoderated preprint provider, when a user creates a new version but never publish that version, build_unversioned_posted_content would read preprint.get_guid().referent, which hands back that unpublished draft instead of the preprint actually being deposited. This is problematic because the unpublished draft will have date_published to be None and that would fail when building the payload.

adlius · 2026-09-11T17:51:26Z

@@ -48,6 +49,23 @@ def post(self, request):
                if record.get('status').lower() == 'success' and doi:
                    msg = record.find('msg').text
                    created = bool(msg == 'Successfully added')


So a DOI in the doi_unversioned category is only written when Crossref replies Successfully added. This could be a potential issue because if the DOI
is already registered, the email probably comes back with something like Successfully updated, so for those preprints
this branch never runs.

This would affect legacy preprints that exists before versioning was a thing. Their base guid is their original guid, so the DOI deposit here is the one registered years ago.

I checked against a local copy of prod db. One of preprint 22hq2 has 10.31219/osf.io/22hq2 already registered with crossref. So it's possible that the crossref response email wouldn't contain Successfully added but Successfully updated or some other language. Will need to confirm if that is indeed the case.

If that is the case, one possible fix is that for an unversioned DOI, store doi_unversioned on any successful record rather than only on Successfully added.

There is another problem: if we run the resync task every five minutes, and if the preprint that was resynced in the last run still haven't received an email back confirming the update of the DOI, then in the next run it will be included in the resync task again. So we'd have to rethink how we should do this. One idea suggested by @cslzchen is that we follow the example in this PR and monitory the number of active tasks and spawn new tasks if the number of active task falls below a certain number (say 1000). In this case, we don't have to worry about running the task every five minutes.

adlius · 2026-09-11T17:59:15Z

+    ).exclude(
+        tags__name='qatest',
+        tags__system=True,
+    ).select_related('provider').distinct()


we probably need an order_by here otherwise the ordering of the queryset would be undeterministic.

…DOI resync

bodintsov · 2026-09-29T15:10:31Z

                            ).first()
                            if v1_preprint:
                                v1_preprint.set_identifier_value(category='doi_unversioned', value=doi)
+                                Preprint.objects.filter(id=v1_preprint.id).update(doi_resync_queued_at=None)


PR comment: Added doi_resync_queued_at, so not include them to eligible for resync

bodintsov · 2026-09-29T15:11:32Z

                    _, version = Guid.split_guid(guid) if guid else (None, None)
                    if not version:
                        logger.info(f'Unversioned DOI confirmed by CrossRef: {doi}')
-                        if created and guid:


PR comment: removed created so preprints which have updated status have also been processed

bodintsov · 2026-09-29T15:13:24Z

        return posted_content

    def build_unversioned_posted_content(self, preprint, element):
-        latest = preprint.get_guid().referent


PR comment: removed referent which hands back that unpublished draft instead of the preprint actually being deposite

bodintsov · 2026-09-29T15:15:09Z


+
+@dataclass
+class ResyncPassResult:


PR comments: added helper methods.

bodintsov · 2026-09-29T15:16:10Z


        try:
            async_request_identifier_update.apply_async(kwargs={'preprint_id': preprint._id})
+            Preprint.objects.filter(id=preprint.id).update(doi_resync_queued_at=timezone.now())


PR comment: set doi_resync_queued_at so the preprint be marked and we know it is processing

bodintsov · 2026-09-29T15:16:29Z


        try:
            async_request_identifier_update.apply_async(kwargs={'preprint_id': preprint._id})
+            Preprint.objects.filter(id=preprint.id).update(doi_resync_queued_at=timezone.now())


PR comment: set doi_resync_queued_at so the preprint be marked and we know it is processing

bodintsov · 2026-09-29T15:19:17Z

+        return max(0, self.total - self.queued)
+
+
+def get_in_flight_doi_resync_count():


PR comment: The count of the preprints is being processed right now

bodintsov · 2026-09-30T13:23:24Z

                    elif 'possible preprint/vor pair' not in msg.lower():
                        # Directly updates the identifier
                        preprint.set_identifier_value(category='doi', value=doi)
+                    Preprint.objects.filter(id=preprint.id).update(doi_resync_queued_at=None)


PR comment: set doi_resync_queued_at to None when preprint is being processed

adlius · 2026-10-08T14:39:46Z

+            continue
+
+        try:
+            async_request_identifier_update.apply_async(kwargs={'preprint_id': preprint._id})


How is this line different from line 123? If line 123 submits a doi record for the v1 version, then this would also submit a doi record for the v1 version, right? Doesn't that mean the unversioned DOI would now associate with the earliest version instead of the latest version? Am I missing something here?

adlius · 2026-10-08T14:42:41Z

            body.append(self.build_posted_content(preprint, element, include_relation))

+        if include_unversioned_doi:
+            body.append(self.build_unversioned_posted_content(preprints[0], element))


This is related to the previous comment: if we always use preprints[0] for the include_unversioned_doi, then that means we would use whatever preprint that gets passed to this function for the unversioned doi, instead of the latest version.

adlius · 2026-10-08T14:56:23Z

+                f'Skipping preprint {preprint._id}: '
+                f'provider {preprint.provider._id} has no DOI prefix'
+            )
+            skipped += 1


Would it actually get skipped? Cuz it seems that the queryset never excludes preprints whose provider doesn't have a doi_prefix, resulting in the same preprint getting queued up and skipped again and again without actually skipping.

adlius · 2026-10-08T16:08:11Z

            versioned_guids__version__lt=preprint.version,
            include_rejected=False,
-        )
+        ) if include_relation else []


This is where I don't quite understand: what does include_relation means and why we only care about preprint_versions when include_relation is true? We seem to be adding new meaning to the include_relation flag here, which seems to be only meant for whether there is an article_doi to the preprint.

cslzchen requested changes Mar 5, 2026

View reviewed changes

Vlad0n20 changed the title ~~Add manage command to resync preprint dois v1~~ [ENG-9044] Add manage command to resync preprint dois v1 Mar 9, 2026

cslzchen approved these changes Mar 13, 2026

View reviewed changes

adlius changed the base branch from feature/pbs-26-2 to feature/pbs-26-6 April 7, 2026 13:13

Vlad0n20 force-pushed the fix/ENG-9044 branch 2 times, most recently from e803a0c to e1d4931 Compare April 14, 2026 11:46

mkovalua suggested changes May 12, 2026

View reviewed changes

Vlad0n20 changed the base branch from feature/pbs-26-6 to feature/pbs-26-9 May 13, 2026 09:02

Vlad0n20 force-pushed the fix/ENG-9044 branch from 7e0d863 to 9095a49 Compare May 13, 2026 09:02

mkovalua approved these changes May 13, 2026

View reviewed changes

Vlad0n20 force-pushed the fix/ENG-9044 branch from 9095a49 to 4855f92 Compare May 27, 2026 14:08

Vlad0n20 changed the base branch from feature/pbs-26-9 to feature/pbs-26-15 July 28, 2026 14:13

bodintsov force-pushed the fix/ENG-9044 branch from 4855f92 to 1973527 Compare July 28, 2026 14:49

adlius requested changes Aug 6, 2026

View reviewed changes

adlius requested changes Aug 13, 2026

View reviewed changes

brianjgeiger reviewed Aug 27, 2026

View reviewed changes

adlius requested changes Aug 27, 2026

View reviewed changes

Comment thread website/identifiers/clients/crossref.py

adlius requested changes Sep 11, 2026

View reviewed changes

Vlad0n20 and others added 11 commits September 22, 2026 18:46

Add manage command to resync preprint dois v1

abfc3c9

Update command

af27fef

Added a class-level docstring

8e93ef4

Register unversioned DOI alongside versioned on Crossref submissions

6efa09a

Add daily Celery beat task and unversioned DOI tracking for preprint …

78ba103

…DOI resync

fix test

03b77a5

fix comments

ab09a25

fix tests

88f467a

add to low priority queue, fix short circuit

47798c1

monitor active tasks and spawn new tasks below threshold

9c4a270

update migration

71a3038

adlius changed the base branch from feature/pbs-26-15 to feature/pbs-26-18 September 22, 2026 16:08

bodintsov force-pushed the fix/ENG-9044 branch from ecac94f to 71a3038 Compare September 22, 2026 16:13

cslzchen self-requested a review September 29, 2026 14:56

cslzchen changed the base branch from feature/pbs-26-18 to feature/pbs-26-19 September 29, 2026 14:56

bodintsov reviewed Sep 29, 2026

View reviewed changes

bodintsov reviewed Sep 30, 2026

View reviewed changes

adlius requested changes Oct 8, 2026

View reviewed changes

bodintsov added 2 commits October 9, 2026 14:59

remove skipped, fix select logic, fix to latest preprint version

4895983

flake8

3a6ad34

		return qs


		def resync_preprint_dois_v1(dry_run=True, batch_size=500, rate_limit=100, provider_id=None):

		return max(0, self.total - self.queued)


		def get_in_flight_doi_resync_count():



		@dataclass
		class ResyncPassResult:

Conversation

Vlad0n20 commented Mar 2, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ticket

Purpose

Changes

Side Effects

QE Notes

CE Notes

Documentation

Uh oh!

cslzchen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cslzchen left a comment • edited by adlius Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antkryt commented May 4, 2026

Uh oh!

mkovalua left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adlius left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brianjgeiger left a comment

Choose a reason for hiding this comment

Uh oh!

adlius left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adlius left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vlad0n20 commented Mar 2, 2026 •

edited by atlassian Bot

Loading

cslzchen left a comment •

edited by adlius

Loading

mkovalua left a comment •

edited

Loading