Does anyone else not have the ability to run custom flow runs from the UI? Yesterday I was able to, but that option seems to have disappeared today on the UI for me. Anyone else experiencing this problem?

Hi, is Prefect Cloud 2.0 HIPAA complaint ?

In Prefect 1.0... How do I run multiple instances of the same flow, based on a list of data? An example would be I've created a generic flow to go grab 1 table from a source system. I want to be able to neatly specify dozens of tables in a convenient location (config file?) and have an instance of the flow run for each of the tables. Do I need to create a 'master' flow that loops through and runs a bunch of child flows? I've already created a working flow with the .map() function to iterate over the list of tables, but that doesn't seem like a good approach since the Prefect UI doesn't seem to neatly separate things out for debugging and retrying.

Hi, the KubernetesJob docs say that you need to have a remote file store configured to use it. And the Storage docs say that none of the remote file storage libraries are installed by default. Now my flow stored in S3 is failing to deploy via the k8s agent because of the missing file store libraries. How do you recommend including something like s3fs in the KubernetesJob environment, since it isn’t included in the default package set? The Storage docs say that you can use

EXTRA_PIP_PACKAGES

for the Docker executor. Is there something similar for the k8s job executor?

Hi, in Prefect 1.0 (Cloud) I want to register a new version of flow A from another flow B. If I am authenticated to Prefect inside flow B I should be able to do this, right?

This message was deleted.

This message was deleted.

I always find it interesting how data / data science products don’t consider prod/stage separation an “ante up” aspect of a SaaS service.

Is there any way to achieve something like

prefect register --dry-run

? I would like to add a CI step that ensures that we won’t fail after a merge when we try to register flows in CD

in prefect 2.0 what is the best way to chain results between map tasks? Specifically, how should errors on specific iterations be handled if the remaining results are still desired? This seemed seemless in prefect 1.x but I am running into issues while migrating to 2.0

Hi, I am using Prefect 1.0 and I tried to create two tasks that share a lot of common code, and at a certain moment they invoke a function which uses the task library to create an instance on of a task and runs it. However, this function is not annotated via @task, while the two invoking functions they are. I observed a weird behavior where only one of the two tasks was run. Should mytask.run() only be invoked within fuction annotated with @task ?

What strategies are folks using for flow storage blocks in 2.0? It seems like there are at least two distinct possibilities: 1. Save all flows using one storage block and running the

deployment

build

and

apply

commands from a parent directory containing multiple flows. This way, we use only one storage block but if multiple developers are working in the same shared bucket like in a staging environment, the blast radius is bigger. For example, if a developer is working on only a single flow, there is a risk of overwriting someone else’s in-development code if you commit code from another flow/folder accidentally.

deployment build flow_a/flow.py:entry_point -n flow_a_deployment --storage-block s3/universal-storage-block

flows/ <working directory>
   flow_a/
   flow_b/

2. Every flow gets its own storage block and running

deployment

build

and

apply

commands from that flow’s root directory. This will obviously require us to use more storage blocks, but seems to decrease blast radius.

deployment build ./flow.py:entry_point -n flow_a_deployment --storage-block s3/individual-storage-block

flow_/a <working directory>

It seems to me like option 2 is more optimal, but are there any disadvantages or limitations we should be aware of in using multiple storage blocks?

Hi there... I need to push a geojson file to Postgres (PostGIS). What is the most efficient way to do this? One option is to write a shell command (using ogr2ogr) within a task. Is there a recommended way to do this?

Hi, is there any possibility of setting the write path and read path of your deployment storage to a different location, e.g. the read path to a Github Directory and the write path to a Local File System?

HI all, Is it safe to add a table into orion.db? or a column into flow_run ? We would need to "log" additional info about flow results, but using "normal" logging may be not optimal, as log table may be big and we would need to do searching in text column...

Hello! In Prefect 1.0, I have my flow run_config set to call a function set_run_config. From that function, is it possible to access the labels that have been added to the flowrun?

What's the recommended / expected way to update a deployment in Prefect 2? I have an existing deployment, and used

prefect deployment apply …

after updating Python files, but I can see in the S3 bucket that the Python files aren't being updated to match what's on my local computer.

Hello! Quick question: is there an equivalent for task_run_name in Prefect 2.0?

Is there a way to “un-register” a custom block type? (2.0)

Hi! Posted it also on the data tricks and tips channel but I see here there is more activity. ---- I am pretty new to Prefect, Metaflow and other MLOPS platforms (have a strong theoretical and MLE background though). I am now in part of a team of MLEs to perform hyper-param optimization experiments and general ML experiments. As you all know, these experiments are really hard to quantify and I have been looking the past 2 weeks at MLOPS frameworks to help us out with quantifying everything. Hence, I landed with the semi-decision to use Prefect on a AWS backend. How hard is the configuration part ? Can I expect it to be set up by myself over the weekend ? What are the typical costs incurred by my AWS Backend (VMs/ Buckets etc). And is it useful for my use case : I want to perform hyper-parameter optimization experiments and new model experiments - be able to see which configuration of parameters has the best results? And quickly deploy it ?

Hi! Decided to try to migrate to Prefect 2 but have some trouble getting a Docker agent going. I get the error:

Flow could not be retrieved from deployment.

On Prefect 1 I would package all my python dependencies in an Docker image, as well as the flow code (using storage=Module()). What would be the equivalent for Prefect 2? I saw the option of putting the flow definitions on AWS but I don't really see the point

Hey! Random question - with v1, is it still possible to use the Cloud UI, and connect it up to my apollo/graphql endpoint? With server, you can change the endpoint when clicking on this button

If I have a flow, with a task

task_name

, which returns "some_value", and I do:

state = flow.run()

How to I get the return of

task_name

? (ie "some_value")

I was mapping through 1000 tasks, Orion logger can get mad and throw an error, but I just realized that the flow will still continue. I was cancelling the flow and trying to debug what I did wrong but I just had to be a little more patient 🤦‍♂️

Hi, I'm quite new to Prefect, very much still learning. I'm using Prefect 1.0. I have a flow that launches another flow using create_flow_run. I'd like to be able to access a Parameter's value and use it to set the run_name of the subsequent flow. I read this doc and saw the use case of using the state_handler to rename the current flow. Is there a way to get the Parameter value in the flow and use it in my create_flow_run for run_name? Can state_handler be used for that? Here's an example of what I'm trying to achieve:

with Flow(name="my_parent_flow") as flow:
    name = Parameter('name')
    ...
    create_flow_run.map(
        flow_name=unmapped('child_flow')
        ...
        run_name=unmapped(f'<use Parameter 'name''s value> child run')
    )

I'm creating a deployment via prefect deployment build & prefect deployment apply. When I check prefect cloud, I can see the deployment but it says 'This deployment is deprecated.' Any tips? I've tried deleting via UI and trying again.

In 2.0, is there a way to see a list of running/not-running agents similar to that in Cloud 1.0?

we have installed required packages that need to connect to the redshift cluster but still when we try to register the flows we are seeing the following error

Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'redshift_connector\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')

Hello folks! I got a bit confused about hosting Prefect on a cloud VM explained here: https://discourse.prefect.io/t/hosting-a-remote-orion-instance-on-a-cloud-vm/967 Is my understanding true? We have our code locally in vscode, and we create a VM. Then we do all the settings mentioned in the link. Now, if we run the flow locally, it will be executed on the VM and not the local machine, right?

Hey everyone! I’m currently evaluating different orchestrators like Airflow, Prefect and Metaflow for an MLOps pipeline in a large enterprise. I’ve already done a little bit of research and came across even with combinations of Prefect as a general orchestrator and Metaflow for ML DAGs. I’m currently leaning more towards Prefect. Has anyone worked with any of the aforementioned tools to offer some insights? (edited)