Hello again! Is there any way to do Bitbucket storage through SSH? There seems to be a generic Git storage module which supports it, but it's a bit unclear whether this is compatible with a Bitbucket remote

Hi I’m getting Storage permission while creating deployments with gcs Traceback (most recent call last): File “/usr/local/lib/python3.9/site-packages/google/cloud/storage/blob.py”, line 2567, in upload_from_file created_json = self._do_upload( File “/usr/local/lib/python3.9/site-packages/google/cloud/storage/blob.py”, line 2368, in _do_upload response = self._do_multipart_upload( File “/usr/local/lib/python3.9/site-packages/google/cloud/storage/blob.py”, line 1903, in _do_multipart_upload response = upload.transmit( File “/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/upload.py”, line 153, in transmit return _request_helpers.wait_and_retry( File “/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/_request_helpers.py”, line 148, in wait_and_retry response = func() File “/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/upload.py”, line 149, in retriable_request self._process_response(result) File “/usr/local/lib/python3.9/site-packages/google/resumable_media/_upload.py”, line 114, in _process_response _helpers.require_status_code(response, (http.client.OK,), self._get_status_code) File “/usr/local/lib/python3.9/site-packages/google/resumable_media/_helpers.py”, line 105, in require_status_code raise common.InvalidResponse( google.resumable_media.common.InvalidResponse: (‘Request failed with status code’, 403, ‘Expected one of’, <HTTPStatus.OK: 200>)

Is there a way to lock down which `Storage`s an agent is allowed to run? Our concern is if an API key is stolen, and someone submits a flow that e.g. uses a github storage pointing to their own repo and so runs arbitrary code on our ECS agent

hi team, 1.0 question, are task serializers stored in the Prefect database as part of the task state/result? I’m trying to use

get_task_run_result

to fetch task results from a different flow. The result location is correct but the serializer defaults to Pickle while that task uses PandasSerializer, so it errors out with Pickle

For Enterprise users of Prefect Cloud — is there any way we can determine when a licensed user was last active / logged in from the graphql API? Users are across multiple tenants, fwiw Querying

audit_log

returns this information in the

message

field, but looks to only be for the current tenant only. Wondering if there is a way to get this across all tenants/users associated with a license

how do I enable MFA for my account? Or is that not available in the Free plan?

In 2.0 how could I manually run a flow / replay a flow run to be able to use pdb for debugging purposes? I have tried

python -m prefect.engine <flow_run_id>

but I get

This run has already terminated

Hey folks Quick one from me. In Prefect 1.0, there are pre-configured tasks available, for example the AirbyteConnectionTask. I'm trying to trigger an Airbyte connection using Prefect 2.0, but from what I can tell, there's no out-of-the-box tasks available? Am I missing something, or do we need to build everything from scratch with Prefect 2.0?

since the update to prefect 2.0b7, I've been having issues with calling result() related to what I think is prefect storage to s3 broken.

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Hello everybody, I have two quick questions 1. Does a mapped task (of size n) count for 1 task or n tasks in terms of billing? 2. I have a workflow that generates a list of size n and then all subsequent tasks use

map

for each item. On the UI it says "No heartbeat detected from the remote task; marking the run as failed.", but the result of the workflow is marked as "Success". I checked my database and everything looks like the task succeeded. Any idea what is going on and how can I avoid it? Have a nice day!

Prefect 2 question. What are blocks?

Hi everyone, We have a failure callback on our flows that tries to report all of the exceptions that occurred in each task. We previously tried to access the

state.result

from the flow which works locally but not in the cloud explained in this answer https://prefect-community.slack.com/archives/CL09KU1K7/p1653606914417649. Is there a good way to collect exceptions from each task and report them in a state handler when the flow fails? We have have alerts on flow failures and we want to report the exceptions in the alert

Folks, Good evening/afternoon/morning! 🙂 . Does anyone has OpenShift Yaml templates to set up PREFECT ? Due to privacy reason we cant use Prefect Cloud. So need to set up everything on our private OpenShift.. Anyone has Openshift Build , Deploy yaml templates

Friends, again another quick question - Has anyone set up PREFECT on local Windows 10 machine with Docker / Docker Compose. One one of our machines - specially Windows, we are getting our Postegres DB shutting down. Anything specific we need to check while setting things up on Windows 10 machine with Docker Desktop. Thanks in advance

Is there a way I can detect a cancelled run for state handling? There doesn’t seem to be a

is_cancelled()

when looking here.

Hello, I am trying to setup prefect orion with postgress. My postgress db works with kerberso authentication, is there a way I can make it work with kerberos Authentication?

hello, is there any way can set task or flow priority when using dask executor on prefect 1.0 ( e.g., with dask.annotation( resource = { "GPU":1 } ) @Anna Geller. i konw prefect 2.0 support this but for some reasons i can't use 2.0

Hi, in Prefect 1.0+cloud I’ve triggered a flow run (“run1”, running v1 of the flow), say with tasks A->B->C->D. Task C has failed. Task B, a long-running task that assembled a dataset, succeeded. I’d like to de-bug C. Clicking into the task run for B in the UI, I see there is a path for “Result Location”. 1) What syntax can I use to load Task B’s result into a dataframe in an ipynb? (To help me try to re-create the error of Task C in an interactive environment.) 2) Assume I identify the problem in Task C and make the correct code changes. I need to re-register the flow, which updates the flow version to v2. Is there any way for me to partially re-run the flow, such that it starts with v2 Task C picking up the result from v1 Task B, so I don’t have to re-run the long process that succeeded? thanks for considering these!

I've asked this question in the Airbyte Slack, but got no traction so far, has anybody been using Airbyte + Prefect and know what the best practice is for communicating with the API for a deployed version of Airbyte? Hey folks I have an instance of Airbyte deployed on GCP following the tutorial below. https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine#connect-to-airbyte This uses ssh to connect to the instance and use it in my browser via

localhost:8000

. I'm trying to connect to the instance (to make an API call) through a Python script, but I have no idea how to think about the authentication layer. How do I send an API call to the server? The documentation suggests not to open the instance up to HTTP traffic but I'm guessing that's a requirement to be able to communicate with the API?

I have multiple tasks in a flow. Upon successful execution of all the main tasks, I am triggering a tasks called

success_task

that will send a SQS message to my backend server indicating that the flow ran successfully. Now, I want to send a message when there is a failure in execution of task as well. Since the amount of tasks can be high (more than 10), is there a better to do error handling of the tasks?

Hello! I’m just starting to attempt to deploy prefect open-source on GCP with kubernetes, I updated to

2.0b7

and noticed that

prefect orion kubernetes-manifest

is not a valid command. I assume I can just use the manifest from 2.0b6 and change the image version - so this is more of an FYI, since it’s still in the docs. Or is there something else I should be aware of here?

Hi Team, We have installed dask-gateway 2022.4.0 and PREFECT_VERSION=0.14.1, prefect helm chart 2022.01 Python version 3.8 (since dask-gateway 2022.4.0 required python 3.8 and above). When we are trying to register the prefect flow, we are facing the below error. Installed the above on the google cloud kubernetes engine cluster Error message below: "an integer is required ,got type bytes" error Could you please throw some light on where we went wrong. Could there be any version compatibility issue here

Morning! So Can someone help me understand the following - I have been looking at the data returned from the GQL api about flow runs an I have noticed a peculiar thing - I see two versions of the same flow (on a schedule) running at the same time - so I have a current version flow_run and then one also for the previous version - this continued to happen once per hour (2x runs with diff params per hour) for 5 hours after the version bump - I am stumped 🤫

Hi, my question is what “pickle” and “cloudpickle” mean？I am not a Python expert

How do I configure a flow to upload local python modules to S3? For example my directory structure looks like this:

project/
    flow.py
    scripts/
        util.py

And in flow.py, there is this import statement:

from scripts.util import Util

Also in flow.py, using this storage:

with Flow('some-name', storage=S3(bucket='some-bucket')):
    ...

But when run from the cloud, it complains

No module named "scripts.util"

Hey All - As a complete prefect novice can anyone give me a simple tldr of the flow from a github push of a new flow to a CI environment (teamcity in our case) to deployment of jobs on k8s. If a flow gets removed/renamed does the agent remove the existing job etc etc - using prefect cloud 2 and k8s agent

hello, we have a problem with scheduled flows, since 2 PM French time, no more than 10 runs are running

anybody having the same issue?