https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • o

    Oscar Krantz

    06/22/2022, 2:15 PM
    Hello again! Is there any way to do Bitbucket storage through SSH? There seems to be a generic Git storage module which supports it, but it's a bit unclear whether this is compatible with a Bitbucket remote
    k
    • 2
    • 1
  • a

    Amit Gupta

    06/22/2022, 3:58 PM
    Hi I’m getting Storage permission while creating deployments with gcs Traceback (most recent call last): File “/usr/local/lib/python3.9/site-packages/google/cloud/storage/blob.py”, line 2567, in upload_from_file created_json = self._do_upload( File “/usr/local/lib/python3.9/site-packages/google/cloud/storage/blob.py”, line 2368, in _do_upload response = self._do_multipart_upload( File “/usr/local/lib/python3.9/site-packages/google/cloud/storage/blob.py”, line 1903, in _do_multipart_upload response = upload.transmit( File “/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/upload.py”, line 153, in transmit return _request_helpers.wait_and_retry( File “/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/_request_helpers.py”, line 148, in wait_and_retry response = func() File “/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/upload.py”, line 149, in retriable_request self._process_response(result) File “/usr/local/lib/python3.9/site-packages/google/resumable_media/_upload.py”, line 114, in _process_response _helpers.require_status_code(response, (http.client.OK,), self._get_status_code) File “/usr/local/lib/python3.9/site-packages/google/resumable_media/_helpers.py”, line 105, in require_status_code raise common.InvalidResponse( google.resumable_media.common.InvalidResponse: (‘Request failed with status code’, 403, ‘Expected one of’, <HTTPStatus.OK: 200>)
    k
    • 2
    • 9
  • r

    rectalogic

    06/22/2022, 4:45 PM
    Is there a way to lock down which `Storage`s an agent is allowed to run? Our concern is if an API key is stolen, and someone submits a flow that e.g. uses a github storage pointing to their own repo and so runs arbitrary code on our ECS agent
    k
    • 2
    • 38
  • m

    Michał Augoff

    06/22/2022, 5:32 PM
    hi team, 1.0 question, are task serializers stored in the Prefect database as part of the task state/result? I’m trying to use
    get_task_run_result
    to fetch task results from a different flow. The result location is correct but the serializer defaults to Pickle while that task uses PandasSerializer, so it errors out with Pickle
    k
    • 2
    • 4
  • r

    Robert Spurrier

    06/22/2022, 5:36 PM
    For Enterprise users of Prefect Cloud — is there any way we can determine when a licensed user was last active / logged in from the graphql API? Users are across multiple tenants, fwiw Querying
    audit_log
    returns this information in the
    message
    field, but looks to only be for the current tenant only. Wondering if there is a way to get this across all tenants/users associated with a license
    k
    • 2
    • 1
  • r

    rectalogic

    06/22/2022, 5:51 PM
    how do I enable MFA for my account? Or is that not available in the Free plan?
    k
    • 2
    • 1
  • p

    Paco Ibañez

    06/22/2022, 6:45 PM
    In 2.0 how could I manually run a flow / replay a flow run to be able to use pdb for debugging purposes? I have tried
    python -m prefect.engine <flow_run_id>
    but I get
    This run has already terminated
    k
    • 2
    • 8
  • d

    Danilo Drobac

    06/22/2022, 6:46 PM
    Hey folks Quick one from me. In Prefect 1.0, there are pre-configured tasks available, for example the AirbyteConnectionTask. I'm trying to trigger an Airbyte connection using Prefect 2.0, but from what I can tell, there's no out-of-the-box tasks available? Am I missing something, or do we need to build everything from scratch with Prefect 2.0?
    ❤️ 1
    k
    m
    f
    • 4
    • 10
  • a

    Alex Tam

    06/22/2022, 8:17 PM
    since the update to prefect 2.0b7, I've been having issues with calling result() related to what I think is prefect storage to s3 broken.
    botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
    z
    • 2
    • 2
  • t

    Tilé

    06/22/2022, 8:33 PM
    Hello everybody, I have two quick questions 1. Does a mapped task (of size n) count for 1 task or n tasks in terms of billing? 2. I have a workflow that generates a list of size n and then all subsequent tasks use
    map
    for each item. On the UI it says "No heartbeat detected from the remote task; marking the run as failed.", but the result of the workflow is marked as "Success". I checked my database and everything looks like the task succeeded. Any idea what is going on and how can I avoid it? Have a nice day!
    k
    • 2
    • 4
  • b

    Bob Colner

    06/22/2022, 8:34 PM
    Prefect 2 question. What are blocks?
    k
    m
    +2
    • 5
    • 7
  • m

    Michael Duncan

    06/22/2022, 9:57 PM
    Hi everyone, We have a failure callback on our flows that tries to report all of the exceptions that occurred in each task. We previously tried to access the
    state.result
    from the flow which works locally but not in the cloud explained in this answer https://prefect-community.slack.com/archives/CL09KU1K7/p1653606914417649. Is there a good way to collect exceptions from each task and report them in a state handler when the flow fails? We have have alerts on flow failures and we want to report the exceptions in the alert
    k
    • 2
    • 4
  • a

    Abin Antony

    06/22/2022, 10:09 PM
    Folks, Good evening/afternoon/morning! 🙂 . Does anyone has OpenShift Yaml templates to set up PREFECT ? Due to privacy reason we cant use Prefect Cloud. So need to set up everything on our private OpenShift.. Anyone has Openshift Build , Deploy yaml templates
    k
    j
    +2
    • 5
    • 33
  • a

    Abin Antony

    06/22/2022, 10:20 PM
    Friends, again another quick question - Has anyone set up PREFECT on local Windows 10 machine with Docker / Docker Compose. One one of our machines - specially Windows, we are getting our Postegres DB shutting down. Anything specific we need to check while setting things up on Windows 10 machine with Docker Desktop. Thanks in advance
    k
    p
    • 3
    • 4
  • k

    Ken Nguyen

    06/23/2022, 12:52 AM
    Is there a way I can detect a cancelled run for state handling? There doesn’t seem to be a
    is_cancelled()
    when looking here.
    k
    • 2
    • 1
  • a

    Amol Shirke

    06/23/2022, 2:06 AM
    Hello, I am trying to setup prefect orion with postgress. My postgress db works with kerberso authentication, is there a way I can make it work with kerberos Authentication?
    k
    a
    • 3
    • 7
  • y

    yu zeng

    06/23/2022, 3:44 AM
    hello, is there any way can set task or flow priority when using dask executor on prefect 1.0 ( e.g., with dask.annotation( resource = { "GPU":1 } ) @Anna Geller. i konw prefect 2.0 support this but for some reasons i can't use 2.0
    k
    • 2
    • 10
  • l

    luke b

    06/23/2022, 5:57 AM
    Hi, in Prefect 1.0+cloud I’ve triggered a flow run (“run1”, running v1 of the flow), say with tasks A->B->C->D. Task C has failed. Task B, a long-running task that assembled a dataset, succeeded. I’d like to de-bug C. Clicking into the task run for B in the UI, I see there is a path for “Result Location”. 1) What syntax can I use to load Task B’s result into a dataframe in an ipynb? (To help me try to re-create the error of Task C in an interactive environment.) 2) Assume I identify the problem in Task C and make the correct code changes. I need to re-register the flow, which updates the flow version to v2. Is there any way for me to partially re-run the flow, such that it starts with v2 Task C picking up the result from v1 Task B, so I don’t have to re-run the long process that succeeded? thanks for considering these!
    k
    • 2
    • 2
  • d

    Danilo Drobac

    06/23/2022, 8:49 AM
    I've asked this question in the Airbyte Slack, but got no traction so far, has anybody been using Airbyte + Prefect and know what the best practice is for communicating with the API for a deployed version of Airbyte? Hey folks I have an instance of Airbyte deployed on GCP following the tutorial below. https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine#connect-to-airbyte This uses ssh to connect to the instance and use it in my browser via
    localhost:8000
    . I'm trying to connect to the instance (to make an API call) through a Python script, but I have no idea how to think about the authentication layer. How do I send an API call to the server? The documentation suggests not to open the instance up to HTTP traffic but I'm guessing that's a requirement to be able to communicate with the API?
    k
    n
    • 3
    • 5
  • r

    Raviraja Ganta

    06/23/2022, 9:58 AM
    I have multiple tasks in a flow. Upon successful execution of all the main tasks, I am triggering a tasks called
    success_task
    that will send a SQS message to my backend server indicating that the flow ran successfully. Now, I want to send a message when there is a failure in execution of task as well. Since the amount of tasks can be high (more than 10), is there a better to do error handling of the tasks?
    s
    k
    • 3
    • 9
  • d

    Dominik Wagner

    06/23/2022, 10:21 AM
    Hello! I’m just starting to attempt to deploy prefect open-source on GCP with kubernetes, I updated to
    2.0b7
    and noticed that
    prefect orion kubernetes-manifest
    is not a valid command. I assume I can just use the manifest from 2.0b6 and change the image version - so this is more of an FYI, since it’s still in the docs. Or is there something else I should be aware of here?
    k
    m
    +2
    • 5
    • 14
  • s

    Surya

    06/23/2022, 10:58 AM
    Hi Team, We have installed dask-gateway 2022.4.0 and PREFECT_VERSION=0.14.1, prefect helm chart 2022.01 Python version 3.8 (since dask-gateway 2022.4.0 required python 3.8 and above). When we are trying to register the prefect flow, we are facing the below error. Installed the above on the google cloud kubernetes engine cluster Error message below: "an integer is required ,got type bytes" error Could you please throw some light on where we went wrong. Could there be any version compatibility issue here
    k
    • 2
    • 11
  • j

    Joshua Greenhalgh

    06/23/2022, 11:01 AM
    Morning! So Can someone help me understand the following - I have been looking at the data returned from the GQL api about flow runs an I have noticed a peculiar thing - I see two versions of the same flow (on a schedule) running at the same time - so I have a current version flow_run and then one also for the previous version - this continued to happen once per hour (2x runs with diff params per hour) for 5 hours after the version bump - I am stumped 🤫
    k
    • 2
    • 9
  • z

    Zheng Cheng

    06/23/2022, 11:51 AM
    Hi, my question is what “pickle” and “cloudpickle” mean?I am not a Python expert
    k
    • 2
    • 4
  • z

    Zheng Cheng

    06/23/2022, 12:34 PM
    k
    • 2
    • 1
  • j

    jack

    06/23/2022, 1:31 PM
    How do I configure a flow to upload local python modules to S3? For example my directory structure looks like this:
    project/
        flow.py
        scripts/
            util.py
    And in flow.py, there is this import statement:
    from scripts.util import Util
    Also in flow.py, using this storage:
    with Flow('some-name', storage=S3(bucket='some-bucket')):
        ...
    But when run from the cloud, it complains
    No module named "scripts.util"
    k
    • 2
    • 10
  • r

    redsquare

    06/23/2022, 1:38 PM
    Hey All - As a complete prefect novice can anyone give me a simple tldr of the flow from a github push of a new flow to a CI environment (teamcity in our case) to deployment of jobs on k8s. If a flow gets removed/renamed does the agent remove the existing job etc etc - using prefect cloud 2 and k8s agent
    k
    • 2
    • 3
  • s

    Stéphanie Cérulli

    06/23/2022, 2:00 PM
    hello, we have a problem with scheduled flows, since 2 PM French time, no more than 10 runs are running
    k
    m
    j
    • 4
    • 24
  • s

    Stéphanie Cérulli

    06/23/2022, 2:00 PM
    anybody having the same issue?
    ✅ 2
  • m

    Michal Zawadzki

    06/23/2022, 2:24 PM
    FYI Something seems to have changed with the Docker image of the 2.0b7 release (
    prefecthq/prefect:2.0b7-python3.10
    to be exact); I was getting error code 255 at the
    apt-key add
    step in my custom image. Adding `apt install gnupg`fixed the issue.
    k
    m
    • 3
    • 2
Powered by Linen
Title
m

Michal Zawadzki

06/23/2022, 2:24 PM
FYI Something seems to have changed with the Docker image of the 2.0b7 release (
prefecthq/prefect:2.0b7-python3.10
to be exact); I was getting error code 255 at the
apt-key add
step in my custom image. Adding `apt install gnupg`fixed the issue.
k

Kevin Kho

06/23/2022, 2:44 PM
Thanks for raising. This looks like something we need to look into @Marvin open “Prefect image gives error code 255”
👍 1
m

Marvin

06/23/2022, 2:45 PM
https://github.com/PrefectHQ/prefect/issues/5931
View count: 7