@Marvin, I'm having troubles with getting results out of my tasks. Is there a way to visualize it from the webapp at all? Using the self-hosted & latest version, unless I log the task output, it's nowhere to be found in the UI. It's OK if it's only available in the configured storage, however it is not very well communicated or intuitive

In Prefect 3.x, task results are stored in memory by default, which is not ideal for large data or when you want to visualize them later. For self-hosted setups, you can configure task outputs to be stored in external storage like AWS S3. This allows you to access them even if they're not visible in the UI by default. You can use storage blocks from Prefect's integration libraries to set this up. More details are available in the Prefect documentation. As for visualizing task results directly in the UI, this isn't a default feature. However, you can log outputs, as you've mentioned, or use the

visualize()

method to view the structure of your flows, though this won't show task outputs. More about using

visualize()

can be found here. For configuring your self-hosted Prefect server, you can find more information here. If you have specific storage needs or further questions, feel free to ask!

to avoid any risk of marvin taking you on a wild goose chase, we don't store task run results / parameters in the API at all the API only knows the path of the result storage

That's fair, I've been sent on a wild goose chase asking chatgpt on this subject, and this is the final station lol. Thanks for clarifying!

yep, Task Inputs in the UI refers to upstream result references ie if task A passes its result to task B then it knows about that relationship but the API doesn't have the actual inputs

I'm quite comfused, for flows it says the results are random UUIDs, and it doesn't accept a

result_storage_key

, how are we supposed to figure out which result file is corresponding to a given flow

can you clarify what you’re trying to do?

for now, just trying to get a grip on the platform. Later, I want to build a recon pipeline with traceability of discovered assets. I.e. we found this IP, how did we do it? That would mean we'd probably have to crawl the tasks-flows hierarchy, doesn't matter if it's done manually, but right now I'm trying to find the pieces of the puzzles. I could get the task id from the results by setting result_storage_key, I don't know where to look for for the parent other than in the prefect DB, or I could store it in the results itself...

To retrieve task data from the database using a task ID in Prefect 3.x, you can use the

get_client()

method from the

prefect.client

module. Here's a general approach to achieve this: 1. Use the

get_client()

function to create a client session. 2. Use the

read_task_run()

method to fetch the task run details by providing the task ID. Here's an example of how you might implement this:

python
from prefect.client import get_client
from uuid import UUID

async def get_task_data(task_id: str):
    async with get_client() as client:
        task_run = await client.read_task_run(UUID(task_id))
        if task_run:
            return task_run
        else:
            print("Task not found.")
            return None

# Example usage
import asyncio
task_id = "your-task-id-here"
asyncio.run(get_task_data(task_id))

This will fetch the task details from the database. Note that you need to replace

"your-task-id-here"

with your actual task ID. The task data will be returned if found, otherwise it will indicate that the task is not found. For more information, you can refer to the Prefect GitHub source code where similar functionality is implemented.