flyteorg/flyte

[Core feature] Make execution_id and execution_domain available for plugins

Open

#6,308 opened on Mar 5, 2025

View on GitHub
 (3 comments) (0 reactions) (2 assignees)Python (378 forks)batch import
enhancementgood first issuehelp wanted

Repository metrics

Stars
 (3,705 stars)
PR merge metrics
 (Avg merge 3d 8h) (116 merged PRs in 30d)

Description

Motivation: Why do you think this is important?

BACKGROUND

https://flyte-org.slack.com/archives/CP2HDHKE1/p1741181647724639

ISSUE

We are trying to fetch Flyte execution domain in a Spark task running in Databricks using the Fyte Databricks plugin.

We tried to use the same method that works well in Python tasks running on K8s (flytekit.current_context().execution_id.domain)

Unfortunately, this approach does not work in case of Databricks plugin. It would be nice to pass this info in the API so that we can access it in tasks.

Goal: What should the final outcome look like, ideally?

We are able to fetch execution domain in tasks.

Describe alternatives you've considered

Pass the domain as workflow input param.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes

Contributor guide

Research direction
The issue requires making execution id and execution domain accessible in Flyte plugins, specifically the Databricks plugin. Currently, Python tasks on K8s use flytekit.current context().execution id.domain, but this doesn't work in Databricks. The research should investigate how context is propagated in the plugin SDK and how to extend it to support execution metadata. Look at the plugin interface in flytekit and the Databricks plugin code to understand the current limitations and propose a solution to pass execution information to task executions.
Tech stack
python
Domain
backenddata
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Understanding of Flyte plugin systemPython
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25