https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • t

    Tim Enders

    06/09/2022, 3:10 PM
    What is the equivalent to this in 2.0?
    @task(checkpoint=False)
    If there is one currently
    ✅ 1
    k
    a
    m
    • 4
    • 5
  • a

    Andreas Nord

    06/09/2022, 3:13 PM
    What is the best way of restricting some tasks (with "tag2") to not run until all tasks (with "tag1") has succeeded? Does not have to be tag-based (Prefect 1)
    k
    • 2
    • 7
  • j

    jack

    06/09/2022, 3:39 PM
    How can I have the first task in a flow specify a timestamp that subsequent tasks can access? Basically I want all tasks to be able to read/write data to the same directory, with the directory have a name like 2022_06_09__113826 based on the approximate time the flow started.
    t
    k
    • 3
    • 2
  • t

    Thomas Opsomer

    06/09/2022, 3:43 PM
    Hello, We currently have no running flows however when looking at the "Task Concurrency" page, I have a tag for which it says that 2 task currently running 😮
    k
    • 2
    • 8
  • c

    chicago-joe

    06/09/2022, 5:13 PM
    RUN export PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS=$(echo `(aws secretsmanager get-secret-value --secret-id ${prefect_aws_secret_name} --region ${region}) | jq -r '.SecretString'`)
    I'm having a ton of trouble using ARG/ENV/RUN with a dockerfile, what I want is for PREFECT__CONTEXT__SECRETS_AWS_CREDENTIALS to show up as an env variable in the final image. Does anyone have experience setting env variables in a dockerfile from command output?
    ✅ 1
    k
    m
    a
    • 4
    • 15
  • d

    Dung Khuc

    06/09/2022, 6:01 PM
    Hello, we are trying to run prefect agent behind a proxy and encounter this:
    Error downloading Flow from Azure: ('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
    Following env vars are set
    HTTPS_PROXY=x.x.x.x
    AZURE_STORAGE_CONNECTION_STRING=xxxx
    We can connect to blob should just fine using
    az
    . Any pointer?
    ✅ 1
    k
    a
    • 3
    • 15
  • p

    parsa

    06/09/2022, 7:27 PM
    Hey I'm trying to run a mapped task and it looks like everything was successful but then right after reduce when I try to merge the results I'm getting this error:
    ✅ 1
    k
    c
    a
    • 4
    • 29
  • f

    Frederick Thomas

    06/09/2022, 7:35 PM
    Hi All, We've just migrated to Prefect Cloud from Server and are confused as to whether the scheduled times of our flows will remain or will we have to manually re-schedule them ourselves. Thanks
    k
    • 2
    • 2
  • n

    Nikhil Jain

    06/09/2022, 7:58 PM
    I am trying to use GIT storage with ECS run. The goal is to have a docker container which has all the python packages installed and then just pull the code from github prior to running the flow. This obviously requires that the container should be able to pull code from Github. Then, in
    task-definition.yaml
    I am providing
    GITHUB_ACCESS_TOKEN
    through linux ENVIRONMENT variables. But I am getting this error:
    'The secret GITHUB_ACCESS_TOKEN was not found.  Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>'`
    Am I missing something?
    k
    • 2
    • 24
  • d

    Darin Douglass

    06/09/2022, 8:30 PM
    is there any known issue with
    create_flow_run
    that will cause two flows to spawn from the one call?
    k
    • 2
    • 11
  • m

    Mike Larsson

    06/09/2022, 8:39 PM
    hi folks, i have a question about agent-specific flow config. is there a way to make config available in a flow where the value will depend on the agent it's running in?
    k
    • 2
    • 11
  • j

    Jessica Smith

    06/09/2022, 9:10 PM
    Can a templated task run name access attributes of objects? Like task_run_name="{my_object.name}"
    k
    • 2
    • 10
  • s

    Slackbot

    06/10/2022, 8:26 AM
    This message was deleted.
  • s

    Slackbot

    06/10/2022, 8:44 AM
    This message was deleted.
  • u

    Usman Abdulkareem Adedayo

    06/10/2022, 8:46 AM
    The error is simply sqlalchemy.exec.OperationalError: (sqlite3.OperationalError) table flow already exists
    ✅ 1
    a
    f
    • 3
    • 14
  • d

    David Maier

    06/10/2022, 8:58 AM
    Prefect Cloud 1.0 gives me the error message `Something went wrong. Please wait a few moments and try again.`when I try to enable a schedule. My flow was registered with a simple IntervalSchedule. How would I debug that?
    ✅ 1
    a
    • 2
    • 5
  • s

    Slackbot

    06/10/2022, 9:00 AM
    This message was deleted.
  • a

    Andreas Nigg

    06/10/2022, 9:06 AM
    Hey, for prefect 2.0 can I use *def my_flow(*kwargs) to get my flow parameters? Or do I need to specify each parameter individually as named parameter in my flow function?
    ✅ 1
    a
    • 2
    • 3
  • f

    Florian Guily

    06/10/2022, 10:18 AM
    Hello, i was wondering if there was a way to retrieve current flow run metadata such as id or time elapsed inside the code ?
    ✅ 2
    a
    • 2
    • 6
  • h

    Hemabh Kamboj

    06/10/2022, 11:24 AM
    Hi Folks! I am facing an issue in running a flow on schedule after registering it on prefect cloud. I am getting the following error in logs in prefect cloud
    Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'google.api\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
    I tried running the flow by putting the
    flow.run
    instead of
    flow.register
    and then ran the flow using
    python3 workflow.py
    . And its running fine. The agent is installed on the same machine from where I ran the flow manually. So registering and running are done from the same machine. So I cannot understand why such error is thrown. Can anybody help?
    ✅ 1
    a
    • 2
    • 2
  • d

    Danilo Drobac

    06/10/2022, 11:39 AM
    Has anybody come across an error when using
    prefect cloud login --key {key} --workspace {workspace}
    related to timeouts? I'm trying to run the command in a CI/CD pipeline through Google Cloud Build but it fails due to a timeout and I have very little information in the error.
    a
    • 2
    • 44
  • m

    Martin T

    06/10/2022, 12:00 PM
    Is it possible to set max_retries on Tasks from a Parameter, i.e. runtime?
    ✅ 1
    a
    • 2
    • 12
  • r

    Rajvir Jhawar

    06/10/2022, 12:37 PM
    Hi all loving the new look for prefect 2! I had a question about the storage options for prefect 2, it is possible for me to configure storage for flows and flow results separately? For instance if i wanted to store flow results in cloud storage and flows on local storage can i do that currently? Also curious about the roadmap for storage options. In prefect 1 you allowed for github and module storage just curious if that is coming back in 2?
    ✅ 1
    a
    • 2
    • 3
  • j

    Jessica Smith

    06/10/2022, 12:52 PM
    Curious if there is a best practice on something I'm struggling with. I have a task that creates a sql alchemy engine from a parameter, and this is passed on to the next set of tasks. However, you can't pickle a sql alchemy engine, which means I have checkpointing turned off for the Get Engine task. This means that if one of the queries fails and retries it will not have the input it needs. Do I need to just create the engine inside the task? Or is there another way people have used?
    ✅ 1
    a
    k
    s
    • 4
    • 8
  • r

    Ramzi A

    06/10/2022, 2:06 PM
    Is there a plan inbetween starter and enterprise to have more than 3 users?
    ✅ 1
    k
    c
    • 3
    • 3
  • s

    Sushma Adari

    06/10/2022, 2:36 PM
    Hi all, we have named volumes to which the agents are mounted to in prefect. We are facing an issue whenever a Flow Fails/Task fails involving a docker mount that the full server error message is returned with the full server values including themount location, user, address, and password. Obviously this is a massive security issue. Is there anyway to circumvent this output? SCRIPT ALERT: Intraday Strategy Greeks entered state Failed with message 500 Server Error for `http+docker://localhost/v1.41/containers/416e8db53c290dc44f8b1621c027e1930f07606101c82ed0ed059a367bcd8c13/start`: Internal Server Error ("error while mounting volume '/var/lib/docker/volumes/output/_data': failed to mount local volume: mount //XXXXXX.XXX.XXX/XXX:/XXX/lib/docker/volumes/output/_data, data: addr=XX.XXX.X.XX, username=XXXX, password=XXXXXX: no such file or directory"). (edited)
    k
    • 2
    • 18
  • i

    Ilya Sapunov

    06/10/2022, 3:31 PM
    Hi I have the same issue as in this thread with loosing value of “task_loop_result” on retries https://prefect-community.slack.com/archives/CL09KU1K7/p1650958214772249 More code inside
    k
    • 2
    • 12
  • z

    Zhibin Dai

    06/10/2022, 3:42 PM
    how do I add an additional slack notification action to my Automation Actions. I added a new webhook today but idk how to get it to show up in the UI. Do I need to reinstall the slack app?
    ✅ 1
    k
    • 2
    • 4
  • j

    Josh Paulin

    06/10/2022, 3:56 PM
    Hello. Very new to Prefect, and thinking about how to structure my project(s). Is there any way to create a master flow composed of flows that exist across different git repositories?
    ✅ 1
    k
    • 2
    • 1
  • y

    Yehor Anisimov

    06/10/2022, 4:42 PM
    Hello, I am new at Prefect, trying to run flow with dbt execution and in part of initializing GitHub Storage I got error in Cloud (I renamed secret here in error):
    Failed to load and execute flow run: KeyError('The secret ghp_pZdHbjkfhkjkefefiejfefnPd1VJikfn was not found.  Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>')
    I suppose issue in access_token_secret, I stored it in Cloud Secrets
    STORAGE = GitHub(
        repo="",
        path=f"flows/{FLOW_NAME}.py",
        access_token_secret=Secret('GITHUB_ACCESS_TOKEN').get()
    )
    I checked, github token is valid. Any ideas about this error? Thank you
    ✅ 1
    k
    • 2
    • 3
Powered by Linen
Title
y

Yehor Anisimov

06/10/2022, 4:42 PM
Hello, I am new at Prefect, trying to run flow with dbt execution and in part of initializing GitHub Storage I got error in Cloud (I renamed secret here in error):
Failed to load and execute flow run: KeyError('The secret ghp_pZdHbjkfhkjkefefiejfefnPd1VJikfn was not found.  Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>')
I suppose issue in access_token_secret, I stored it in Cloud Secrets
STORAGE = GitHub(
    repo="",
    path=f"flows/{FLOW_NAME}.py",
    access_token_secret=Secret('GITHUB_ACCESS_TOKEN').get()
)
I checked, github token is valid. Any ideas about this error? Thank you
✅ 1
k

Kevin Kho

06/10/2022, 4:44 PM
access_token_secret is a Secret name. Not the value. So no need to wrap it as a
Secret
and use
get()
. The name is enough. You can also edit your post to hide the secret value if that’s a real key
👍 1
y

Yehor Anisimov

06/10/2022, 4:45 PM
It is not the real key. I'll try now, thanks
It's working! Great thanks
View count: 4