elastic/elasticsearch

ETA for Reindex API

Open

#76,904 opened on Aug 25, 2021

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Java (25,882 forks)batch import
:Distributed/Reindex>enhancementTeam:Distributedhelp wanted

Repository metrics

Stars
 (76,700 stars)
PR merge metrics
 (Avg merge 2d 5h) (1,000 merged PRs in 30d)

Description

The get task API (e.g. GET _tasks/lOHn4ATPST6LfKCW6m539w:517820) should include an estimated time of completion where applicable.

Specifically when using the reindex API, I had to write myself a basic tool some time ago to estimate when the task would be completed using the total, created, updated, start_time_in_millis, and running_time_in_nanos properties. (Although I believe in cases where the reindex operation omits some documents, this may not even be sufficient!)

It would be great if this functionality was built into this API.

Contributor guide

Research direction
Investigate the Elasticsearch task management system, specifically the get task API implementation at `GET tasks/{taskId}`. Understand how the reindex operation tracks progress using fields like `total`, `created`, `updated`, `start time in millis`, and `running time in nanos`. Research potential algorithms like exponential moving average to estimate ETA, considering that documents may be omitted. Review existing discussions in the issue and any linked resources. Look at the `TasksClient.java` and `TransportTasksAction.java` source files for implementation details. Proposing adding an `estimated completion time` field to the task response is a starting point.
Tech stack
javarest api
Domain
apisearchobservability
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Elasticsearch task frameworkREST API endpointsJava concurrencyTime estimation logic
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20