Hey all Perhaps a strange Q - Does any one have any experience of deploying/running prefect in Heroku? I’m trying to reuse our current Heroku to host our Prefect agent/server, which would point at our cloud backend but even though it appears to be running, the agent doesn’t appear to be querying

I ran my flow from CLI

prefect run --name

and it set the flow to be scheduled on the cloud UI. I clicked

start now

and the time has passed the 'Schedule to start' but my tasks are still in the scheduled state. Any thoughts on what I could do to start the flow?

After upgrading Prefect to version 2.0.0, I get the following message

UserWarning: Attempted import of 'prefect.Parameter', which is part of Prefect 1.x, while Prefect 2.0.0 is installed. If you're upgrading you'll need to update your code, see the Prefect 2.x migration guide:

<https://orion-docs.prefect.io/migration_guide/>

. Otherwise ensure that your code is pinned to the expected version.

However, the migration link gives me a 404 - Not found message.

Is it just me or are other people also experience broken links for prefect guides?

Hi! I’ve been trying to make a PoC pipeline with dbt transformation using AWS-based task runners, and the new version rollout took me by surprise :) In the v1 version, agent setup seems to be quite simple, and there is a guide - https://docs-v1.prefect.io/orchestration/agents/ecs.html#running-ecs-agent-in-production. Is there an Orion guide as well? I’ve managed to set up S3 storage and make a docker deployment, but a bit stuck after, as there is no agent to run.

My flow is failing because it can't find a file from my gitlab storage. I just added it so it is in the folder now, but the task is still saying that the file is missing. Is there a way I can have the task look at the most recent commit to the gitlab repo?

Hello folks. Can you answer or provide any example related to my question!. 1. How to run the cloud pipeline on prefect-server? 2. How to run the airbyte task with prefect pipeline Would highly appreciate help from this community 🎉

Hi Prefect Community! I'm trying out the new deployment/storage with a basic helloworld project and the "prefect deployment build" hangs after creating the manifest. This is for an s3 storage block. Does anyone have insight?

Dear prefect community, I am not sure whether prefect could be the solution for operational data pipelines / workloads triggered by some notification of an arrival of a new file which a microservice needs to manage. The latency requirement are not critical my concern ist data governance, monitoring, and Maintenance in operation The workloads need to check schemas, transform data into a common Schema (xml/xsd) , compare files / tables on record and field level for reporting # deltas records (reconciliation), and kind of operational impact analysis which would need to request another service with the current file as an input. It's quite operational… my concern with coding everything in Java using spring for developing micro service and Kafka for messaging different states is around data governance, cataloging data, and maintenance of the pipelines in operation when we have like hundreds of customers running these pipelines in parallel… what happens when a Pipeline gets broken?! So… reading about prefect made me thinking suggesting your solution instead. I guess airflow might not be an option here

Hi folks, Does anyone have any luck specifying a schedule in the deployment in 2.0? Also, how to specify environments? in 2.0 beta, I used to use CronSchedule, and SubprocessFlowRunner with a specific environment, but struggling to do the same in 2.0 GA The documentation seems to be missing some details - in the deployment.yaml, there are "schedule" and "infrastructure -> env" fields, but no specifications on what input is required to define them https://orion-docs.prefect.io/concepts/deployments/#deploymentyaml

How do we unit test tasks and flows in Prefect 2.0? I am getting the following error on my test cases after upgrading.

Tasks cannot be run outside of a flow. To call the underlying task function outside of a flow use

task.fn()

I tried

with prefect_test_harness():

but no luck... Ah. Never mind, I should have been reading the error more. Calling

fn

instead of

run

fixes old test cases.

Good Afternoon. Is there a way to get the current version of the flow running programmatically?

This message was deleted.

I am getting a cryptic error when trying to run flows in Prefect 2.0 Any idea why this is happening? I am using

.map

method on my tasks. I wonder if Prefect 2.0 does not support that

is there a way to correlate flow versions with code versions (git sha)?

Dear Community, I’m struggling in Prefect 1.0 for one problem, Thanks for any help!

@task('currency_orchestrator')
def call_currency_orchestrator():

    return create_flow_run(
        flow_name = "currency_orchestrator",
        project_name = "xxxx",
    )

why this throws me an error ??

aise TypeError("`fn` must be callable")
TypeError: `fn` must be callable

basically I wanna use that task to call a registered flow..

👋 I am still using Prefect 1 Is there any resource regarding how I should deal with different environments (prod / staging / proto)? I guess I can create multiple workspaces but then I have to onboard the same user in multiple places + setup multiple api_keys .... Looking for best practices here 🙏

Since API keys can be used across tenants if linked to a user account, you may also pass a tenant:

https://docs-v1.prefect.io/orchestration/concepts/api.html#authenticating-the-client-with-cloud

What are the difference between "Tenants" and "Workspaces"? I might not fully understand those 2 keywords the right way

Hey, I think in the docs here, the prefect api url should be api.prefect.cloud rather than app.prefect.cloud, right? https://docs.prefect.io/ui/cloud-getting-started/

How do we do

prefect.config

in Prefect 2.0? I tried using Settings but I cant put custom key value pairs in it.

When using the GraphQL API is there a certain way we should be passing our flow ID? I believe to be passing the correct flow ID but I get an API response of

Flow __ not found

so was not sure if passing the type wrong?

return create_flow_run(…) under a task, why it gives me this :

aise TypeError("`fn` must be callable")
TypeError: `fn` must be callable

Good morning community, we are really excited to start the upgrade to Orion but are having some issues with deployments. Previously during our beta PoC/migration development, we were using the python

Deployment

function to handle the deployment, as well as specifying

FilePackager

and

PickleSerializer

configurations so that we could create a single file from our (rather large) list of flow files and supporting library files. This file would then be pushed to an S3 bucket and pulled down during job execution on K8s using a standard docker image for the core dependencies. For our use-case, it doesn't make sense to upload the entire directory of 200+ files for each deployment, so we really like the ability to package our flow into a single file. We also don't want to have to create a single storage block for each flow, which would multiply the number of files we're storing in s3 (and therefore have to pull down for each flow run). The documentation now states that the

Deployment

function is no longer usable, but the YAML file and manifests provide no way to implement the

FilePackager

and

PickleSerializer

classes. My question for the prefect team - are the

FilePackager

and supporting serializers going to be available for deployments? Or is the new "upload all files" approach going to be the standard for Orion forever? If I were to hack the packager/serializer approach into our deployment methodology, will I run the risk of having the packager/serializer classes removed from the prefect package in the future? Anna has mentioned in previous comments that the prefect team is just getting started with deployments, but the lack of clarity and the fast breaking changes over the course of a week has made it quite difficult to plan our migration effectively - any clarity on this topic would be greatly appreciated!

Congrats on the 2.0 release! Just FYI, the "DockerFlowRunner" and "KubernetesFlowRunner" hyperlinks return "404" errors in the docs: https://docs.prefect.io/tutorials/docker-flow-runner/ https://docs.prefect.io/tutorials/kubernetes-flow-runner/

Are there instructions on how to set up Kubernetes flows in Prefect Cloud 2.0?

Has anyone done anything with clickhouse and prefect that minds sharing some of their tasks how they are inserting data? Current process is we have a spark job that does our ETL and inserts into our sql server database. but I'm currently exploring clickhouse as our new data store and looking at prefect to kinda orchestrate and monitor our ETLs

Hi all, prefect 2.0. I am getting the following error while building a deployment.

ValueError: Unable to find block document named mini1 for block type RemoteFileSystem

Hi. We have a number of flows that are running on Prefect 0.14.22. I am wondering if I can progressively upgrade them to 1.2. If I run a 1.2 agent, will it work with flows written using an older version of Prefect? I know it's a weird question to ask when 2.0 just went GA 🙂

This message was deleted.

hi everyone, I'm using Prefect 2.0 to orchestrate

xarray

dataset conversion, using

DaskTaskRunner

, the created dask client (

task_runner

) doesn't automatically pickup the

xarray

functions such as

xarray.dataset.concat()

or

<http://xarray.dataset.to|xarray.dataset.to>_zarr

to fill up the works to all CPU threads like without Prefect. To bring the tasks to dask client with Prefect, I have to explicitly submit the task with

@task

decorator and calling

.submit()

This way, however, disallow

xarray

functions to be executed in all CPU threads, only the wrapped task function to be executed in individual thread, which is not ideal. If anyone has come across this issue before and have a solution, much appreciated . Thanks a lot.