Hi all! According to Orion docs creating and running deployments using the (Python?) API is a "coming soon" feature. Do we have a rough estimate on that? Is this planned with the GA on July? I do not want to put any pressure, I am just asking to have a timeline in my mind for my project that would benefit from this

Hello, I am curious if anyone ran into the same issue I am facing with prefect server & prefect cloud. (when I use flow.run() directly in vs code it works fine), The issue happens when connecting to cosmos db, when using flow.register() then running the flow on the server or on the cloud. the task fails and I get the following error: "Unexpected error: TypeError("cannot pickle '_thread.RLock' object")" This is how I am connecting to cosmos db: URL = 'my cosmos db account uri' KEY = 'my cosmos db account primary key' client = CosmosClient(url=URL, credential=KEY) database = client.get_database_client('my database name') If anyone has any guidance as to how I can fix this, please let me know. I would highly appreciate it.

Hello - I’d like to automatically deploy a flow from within a CI/CD pipeline. I run into import errors and am curious if there is some registration pattern that doesn’t require the environment prefect is installed into to also have the flow dependencies installed for serialization? Ideally I’d like to run some CI/CD pipeline using the prefect docker image (but not necessarily install

sklearn

or similar into it) then run

prefect register --project "Project Name" -l label -p flow.py

from within the pipeline. Realistically I could probably use the container that I am running the flow from within but am curious if there is an easier way. Thanks.

Hi, I think there is a ghost task running behind which we cannot see on UI. All of our running task is queued but no job is seen running on UI. Can someone help?

Task 'RunNamespacedJob - DBT Run': Finished task run for task with final state: 'Queued'

Hello, in 2.0, is there a way to provide custom parameter values for a deployment from the UI?

Hello — we’re using Prefect Server 1.2.2, Docker storage (with

stored_as_script=True

). I’m attempting to use the ExecuteNotebook task and I’m getting the following only with debug logs on the Docker agent:

[2022-06-14 20:54:12+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'Test Docker Notebook Flow'
[2022-06-14 20:54:13+0000] INFO - prefect.CloudTaskRunner | Task 'notebook0': Starting task run...
Executing:   0%|          | 0/2 [00:00<?, ?cell/s]Operation not permitted (src/thread.cpp:309)

This is erroring out in the container when trying to access the notebook but not making it back to the Prefect API to change task state (the task run just hangs until it times out with a

No heartbeat detected from the remote task

error). Any ideas?

Orion's KV storage option is removed?

2.0b6

Hiya octavia hi Quick question on triggering Flows, any thoughts or ideas are appreciated and welcome 🙌 We have a Jenkins job extracting data from Postgres postg and loading into BigQuery (yep, aware it could’ve been done with certain ETL tools, but it wasn’t an option) We also have a few Airbyte syncs on the same VM that Prefect is running on too. How would it make the most sense to trigger dbt jobs after the Jenkins and Airbyte syncs are finished? 🤔

Hello Everyone, is it possible with prefect to run python script i.e. (abc.py) instead of refactoring my code to prefect tasks. I have already written my python script with multiprocessing functionality and would like to schedule it prefect.

is it possible to specify where the prefect local agent should clone gitstorage to? Right now it clones gistorage to AppData/Temp/[here] . Just wondering if it's possible to change that through a variable or config?

Hi all! Shouldn't the

config.toml

file be in the path where prefect is installed? There is only

backend.toml

file in the path where my prefect is installed, are those two the same? Also i can't read

backend.toml

. I got the error message like below:

TomlDecodeError: Found invalid character in key name: ':'. Try quoting the key name. (line 1 column 2 char 1)

What's the backend.toml file?

Setting the

apollo_url

under

[server.ui]

in the

.prefect/config.toml

file does not appear to be working. How do I change the host that the UI will call to access the API?

In Prefect 2.0b6 when using

DaskTaskRunner

tasks can stay in a pending state forever if they have custom dependencies (using _`wait_for`)_ on tasks that have failed. The failed state of the upstream tasks do not cause the downstream tasks to fail.. This isn't the case when using the default

ConcurrentTaskRunner

where the failed state correctly propagates downwards and causes the dependent tasks to fail by default

Good morning! I dont know if this is the place for questions, if It is not,sorry in advance. We are implementing prefect in our company and I would like to ask a question. We have deployed prefect Server on a Kubernetes cluster. We have several flows using different custom Python packages and different versions of these. The question I have is it best practice to deploy one agent per flow with a specific environment or have a single agent with all the packages and consume it from all the flows? On the other hand, is there any defined process for the CI/CD of the flows?

Morning all 👋 Any clue about the following error when using S3 as a storage option?

[2022-06-15 110852+0200] ERROR - prefect.S3 | Error uploading Flow to S3 bucket prefect-flows-proto: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Error the error + snippet of code

Hi everyone! I'm setting some state handlers for notifications in our "slack-compatible" (i.e. Zulip) streams. However I'm getting some odd behaviour with the updates. The Flow

with Flow('slack-test',state_handlers=[post_to_slack]) as flow_slack:
    ini_message="Some Init Message"
    #    
    post_slack_init(ini_message)
    #
    count()

We get the initial flow message start (

post_to_slack

notifies us in start/end/failure), but we don't get the "ini_message" until the flow ends. Even more if

count

task (wich also carries a state handler) fails we get the fail of the task in first place and then the

post_slack_init

(just a post request no state handler on this task) message. Any Clues? Thanks in advance!

Heyho 🐙 We tested out the Dask provider and think that Prefect version 2.0b6 maybe creates a Dask cluster that has a version mismatch between client and server when using the "aws.FargateCluster" cloudprovider. Steps to reproduce can be found in the thread 🙂

Hi all, has anyone tried using pydantic models as objects exchanged between tasks? It seems that it is not supported (exceptions are thrown). I am using the latest Prefect 2 release.

Hi there, I am a bit struggling to run two separate flows on ECS and thought to ask for some help 🙂 I would like to run one flow after another. Both flows are registered on prefect cloud, they have their own task_definition_arn and they both run fine separately. I have tried to register flow that would call these flows like in example ( code in the thread). But seems that ECS agent, doesn‘t take in consideration create_flow_run function run_config parameter and tries register new tasks definitions on ECS (or only one for „parent_flow“), although I would like to use existing task definitions plus our ECS agent doesn‘t have permission to register task definitions, therefore I am getting this error:

An error occurred (AccessDeniedException) when calling the RegisterTaskDefinition operation: User: is not authorized to perform: ecs:RegisterTaskDefinition on resource: * because no identity-based policy allows the ecs:RegisterTaskDefinition action

I have tried also to pass flow_a task_definition_arn to parent_flow, then it runs flow, but gets this error:`in create_flow_run ValueError: Received both

flow_id

and

flow_name

. Only one flow identifier can be passed.` Maybe someone can share the best practices how I could create flow that would run in order these two already registered flows ? Big thanks in advance ! 🙂

Hello friends, we’re running

flow.register

and getting an error from the

requests

module, presumably when it’s trying to register the flow to prefect cloud. It’s getting a webpage in the response, with a lot of obfuscated js. Any ideas why this is happening? We’re using prefect 0.14 on python 3.10.4

hello, I am seeing a strange behaviour when invoking an airbyte task from prefect, is this the right place to ask? using prefect 1.2.1

Hi, we are considering to upgrade our prefect server to 2.0. I see that it’s still not production-ready. What’s the timeline that it’ll be ready for production work? What are the consideration if we still do decide to use 2.0?

Hi Folks, I'm finding myself in a pickle I was in before, where I'm trying to just pull a secret from my cloud setup, and I did this in my python script in the last place I ran prefect, but I can't get it to work. While I was developing, I was able to simply set the use_local_secrets = False, and I could just pull down a secret, but I can't get it to work anymore. I only see one short blurb about getting cloud secrets in the prefect API docs around secrets. Can someone help me with the right commands? I'm getting a message that it keeps looking for it locally

Hi Everyone, we're using Prefect Server on premise V1.2.2 hosted on Kubernetes, and everything is working perfect. However, every now and then we get the occasional error

Error during execution of task: ConnectTimeout(MaxRetryError("HTTPConnectionPool(host='prefect-apollo.prefect', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f32b0d48910>, 'Connection to prefect-apollo.prefect timed out. (connect timeout=60)'))"))

This seems to affect all running tasks at the time, so for example if I have 2 or 3 tasks running, they all fail at the exact same time and give this error

Hello! I am trying Prefect2.0 for a simple flow from a docker container. I am getting an error and am stuck trying to resolve it - MissingContextError (“No settings context found”) Is there an explicit initial config required? Can I run flows from within a container? Any help would be really appreciated! Thanks!

Hi. I'm looking at Prefect 2.0. The documentation shows support for S3 for storage. By chance, does this S3 support extend to a self-hosted S3-compliant service, such as MinIO? Otherwise, what are the options for a self-hosted environment? The documentation states that remote storage is required when running in k8s or docker. Would the File Storage on a Remote File System option, such as SMB or NFS, work?

One more question, for Task retries, I see that a retry count and a retry interval are supported. Are you looking to add support for exponential back-off or might that capability already exist?

Hi all. Does anyone have a solid guide for getting up and running with Prefect Cloud and Azure Kubernetes Service for agents? I'm finding bits and pieces everywhere, but could really benefit from a comprehensive tutorial. I've found guides specific to using Docker agent on an Azure-based Ubuntu server, but not the same level of detail for a Kubernetes architecture. Thanks!

Hi everyone. I have a problem where my flow is unintentionally exposing secrets in the UI. I am passing secrets to the

env

argument of a class derived from

ShellTask

.

bash = LoggedShellTask(stream_output=True)

# more stuff here ...

with Flow(
    FLOW_NAME, storage=docker_storage, run_config=run_config, result=PrefectResult()
) as flow:

    run_results = bash(
        helper_script=get_helper_script(repo_path),
        command=get_command(ml_script_path, ml_script_arguments, conda_env=conda_env),
        env=dict(
            SNOWFLAKE_USER=snowflake_user,
            SNOWFLAKE_PASS=snowflake_pass,
            SNOWFLAKE_DATABASE=snowflake_database,
            SNOWFLAKE_OUTPUT_SCHEMA=snowflake_output_schema,
        ),
        log_file_path=get_log_file_path(output_dir),
    )

The issue is Prefect automatically creates a

List

and

Dict

task that have

PrefectResults

by default and this exposes the secrets in the UI. A couple of ideas that come to mind: 1. Assign a specific results objects to each task (not at the flow level) 2. Create a wrapper task that receives all the secrets and returns a dict that is passed to the

ShellTask

3. Create a

ShellTask

that accepts each secret as a parameter Any suggestions? One feature I'd like to preserve is the ability to restart the flow and I'd rather not persist the secrets anywhere. Thanks!

Hello, Have any one managed to run Shell Task via SSH ? or submitted a Spark job to a Yarn cluster ?