Shelley Kim
10/23/2024, 12:42 PMMarvin
10/23/2024, 12:42 PMpython
from prefect import flow, task
from prefect.artifacts import create_link_artifact
@task
def my_first_task():
create_link_artifact(
key="irregular-data",
link="<https://example.com/data.csv>",
description="## Highly variable data",
)
@task
def my_second_task():
create_link_artifact(
key="irregular-data",
link="<https://example.com/low_accuracy.csv>",
description="# Low prediction accuracy",
)
@flow
def my_flow():
my_first_task()
my_second_task()
if __name__ == "__main__":
my_flow()
2. Retrieving Artifacts: You can retrieve existing artifacts using the Artifact.get
class method in the Prefect Python SDK:
python
from prefect.artifacts import Artifact
my_retrieved_artifact = Artifact.get("my_artifact_key")
3. Using CLI Commands: Alternatively, retrieve artifacts by using Prefect CLI commands for artifact inspection:
- Inspect artifacts by key:
prefect artifact inspect <my_key>
- List all artifacts:
prefect artifact ls
4. REST API: Access artifacts programmatically using the Prefect REST API, which allows for creating, reading, or deleting artifacts. This can be useful when integrating artifact management directly in your workflows.
For more details, you can visit the Prefect 3.x artifacts documentation here: Prefect 3.x Artifacts Documentation
Let me know if there's anything else you need help with!Shelley Kim
10/23/2024, 1:01 PMMarvin
10/23/2024, 1:02 PMcreate_table_artifact
function in Prefect allows you to create table artifacts that are associated with a particular flow or task run. When it comes to uploading an artifact from a subflow to a parent flow, Prefect doesn't directly support associating artifacts created in a subflow to the parent flow's artifact list through native functionality.
However, here are a few points to consider:
1. Artifact Association: Artifacts created within a flow (including a subflow) are typically associated with that specific flow run or task run. They are not automatically linked to a parent flow if created within a subflow.
2. Manual Linking: If you need artifacts from a subflow to appear in the context of the parent flow, you might consider manually configuring how and where these artifacts are handled. This might involve designing your flows such that you handle the data or metadata at the parent flow level, or potentially triggering the artifact creation at the parent flow after receiving data from the subflow.
3. Integration with REST API: For more advanced integration, you might need to use the Prefect REST API to programmatically manage and link these artifacts. This would involve manually fetching artifacts from subflows and processing or storing them in a manner that's contextually relevant to the parent flow.
Here is a simple example to illustrate how you might create a table artifact within a task:
python
from prefect import flow, task
from prefect.artifacts import create_table_artifact
@task
def create_table():
table_data = [
{'customer_id': '12345', 'name': 'John Smith', 'churn_probability': 0.85},
{'customer_id': '56789', 'name': 'Jane Jones', 'churn_probability': 0.90},
]
create_table_artifact(table=table_data, description="Customer Churn Probabilities")
@flow
def my_subflow():
create_table()
@flow
def my_parent_flow():
my_subflow()
if __name__ == "__main__":
my_parent_flow()
For more information on handling artifacts in Prefect, you can visit the Prefect 3.x artifacts documentation: Prefect 3.x Artifacts DocumentationMarvin
10/23/2024, 1:03 PMpython
from prefect import flow, task
from prefect.artifacts import create_table_artifact
@task
def process_data():
# Process data and return it
return [
{'key': 'value1', 'data': 'data1'},
{'key': 'value2', 'data': 'data2'},
]
@flow
def my_subflow():
return process_data()
@flow
def my_parent_flow():
data_from_subflow = my_subflow()
# Handle or process the data and create artifacts at the parent flow level
create_table_artifact(table=data_from_subflow, description="Data from subflow processed in parent flow")
if __name__ == "__main__":
my_parent_flow()
For detailed use cases and potential updates on this feature, you can follow Prefect's GitHub issues such as issue #12222, which mentions support for hyperlinks in table artifacts and is related to linking subflow runs.