Indexing tasks that fail ten times are abandoned without any recovery

Documents are indexed through the platform's persistent task queue (org.xwiki.index.TaskManager, DefaultTasksManager). When a task fails, the queue puts it back at the front and retries it immediately; after ten failures it logs abandoned because it has failed too many times at ERROR level and deletes the task from the database.

The usual failure of an indexing task is the embedding call: the model is down, the provider rejects the request (quota, key), or the local model is broken until a restart. Against a model that is down, ten immediate retries take a few seconds, so an outage of a minute is enough to abandon every page saved during it. An abandoned page keeps the chunks of its previous version, or has none if it is new, until it is saved again or its collection is reindexed by hand. The startup requeue of the XWiki document store only runs when a Solr core version bump has marked it. There is no place where an administrator can see which pages failed to index.

Steps to reproduce

1. Create a collection on the XWiki document store over a space, with a working embedding model.
2. Make the embedding model unreachable. Edit a page of the space.
3. Watch the log: ten warnings Error during the execution of task ... in quick succession, then the ERROR line above.
4. Make the model reachable again. Wait; search for the new text.

Expected: the page is indexed once the model is reachable, or at least the failure is visible and recoverable from the administration.

Actual: the page is never indexed again until it is saved again; the only trace is the ERROR line in the log.

Proposal, in three independent parts

1. Make the retries meaningful. The retry loop belongs to the platform queue and the consumer can only throw; the consumer can wait with an increasing delay before rethrowing an embedding failure, or re-queue the task itself with a delay if the platform queue allows it, so that ten attempts span minutes to hours instead of seconds.
2. Record abandoned documents (a marker in Solr per collection and document, or a small table) and re-queue them at startup and when the collection's reindex action runs.
3. Show them: a count and a list of documents whose indexing failed, on the collection page or in the index administration section, with an action to retry them.