https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • f

    Fina Silva-Santisteban

    06/23/2021, 11:27 PM
    Hi everyone, I’ve tried registering a prefect flow using docker storage and ECS run config today and received a
    pip
    and
    python 2.7
    error? (pls see screenshots) The dockerfile I’m using for the docker storage (
    flow.storage = Docker(dockerfile="Dockerfile", …)
    looks like this
    FROM python:3.9
    WORKDIR /app
    COPY . .
    ENV PYTHONPATH=$PYTHONPATH:`pwd`
    FROM <http://gcr.io/google-appengine/python|gcr.io/google-appengine/python>
    RUN apt-get update
    RUN apt-get install poppler-utils -y
    I’ve set the python version to 3.9 in the dockerfile, why doesn’t it use that?
    c
    • 2
    • 2
  • b

    Ben Muller

    06/24/2021, 5:01 AM
    Hey Prefect peeps, Do you have an example of how we can use the
    prefect.context.get("logger")
    to log throught our entire project ? eg. when sub modules are called by calling their classes in that flow?
    k
    • 2
    • 1
  • v

    Varuna Bamunusinghe

    06/24/2021, 5:36 AM
    I got the following error when I try to invite a new user in Prefect Cloud. (excuse if this is not the place for this).
    n
    • 2
    • 4
  • j

    Joël Luijmes

    06/24/2021, 7:26 AM
    Hi, I’m having a DAG with basic fan-out, fan-in idea. Some of the fanned out tasks may fail, but I still want to continue the fan-in task for the succesful tasks. Thus if I have upstream: [x, y, z, FAILURE], the downstream still should process for input [x, y, z]. What I did; • Added trigger=all_finished to downstream task • Iterate over the input, filter out exception Although this worked fine locally while developing, I now got an edge case that one of the upstream tasks is TriggerFailed. How can I mitigate this? And, still process the succesfull upstreams?
    e
    • 2
    • 12
  • a

    ash

    06/24/2021, 9:29 AM
    Hi I am looking forward to try prefect cloud for better understanding of its advantages over prefect core server. Can anyone please share where can i get the access to free developer account for testing and trying prefect cloud.
    c
    • 2
    • 3
  • b

    Ben Muller

    06/24/2021, 9:44 AM
    Hi Prefect, I'm trying to work out how to re run a task in the UI. I have a flow that is very straight forward and the third task failed with the other two passing. Is there a way to re run this individual task and not just re run the entire flow? 🙏
    b
    j
    • 3
    • 5
  • t

    Tim Enders

    06/24/2021, 12:23 PM
    Is there a way to edit the Automations Actions once they are created?
    j
    • 2
    • 2
  • b

    Bruno Murino

    06/24/2021, 12:43 PM
    Hi everyone — I have a flow that is ending with “Flow run SUCCESS: all reference tasks succeeded” even though one of the tasks failed! Am I missing something? The task that fails is a DBTShellTask.. Problem is that I have a flow state handler that sends slack alerts if the overall flow state is FAILED, but that’s no happening so we’re not getting any alert..
    z
    • 2
    • 12
  • m

    Matthias Roels

    06/24/2021, 1:02 PM
    Hi everyone, I have perhaps a more generic question, on AWS it is possible to run flows serverless using ECS (there is an appropriate agent/run_config for it). Are there alternatives on Azure/GCP to do the same (using e.g. cloud run or container instances)? And if so, how would you do that?
    c
    k
    • 3
    • 13
  • l

    Luis Henrique

    06/24/2021, 2:13 PM
    Hi, everyone! I have a question regarding performance of database operations: • Is every log entry stored on the database? • Is every task execution registered on the database? • Is there any built-in automatic history clean-up of old data? • Wouldn't it be better to use redis to avoid high frequency operations and thus high IO on the database? My concern is that if I proceed to break my flows in as many tasks as it might make sense I might cause a performance bottleneck on the DB, which seems counterproductive. Ideally (and it might not be possivle), I should be able to break any flow in as many tasks as I want without this decision affecting performance
    k
    • 2
    • 3
  • m

    Mark McDonald

    06/24/2021, 3:32 PM
    Hi - I'm adding an automation to some of my flows using your api, immediately after I've deployed the flow. I'm calling the "create_flow_run_state_changed_hook_input" mutation to create a new hook each time with the complete list of flow_group_ids. Is this the best way to do it? In other words, I'm looking to confirm that there is no way to update the hook with a new list of flow_group_ids and that I have to create a new hook each time with the updated list of flow_ids and then delete the old hook.
    k
    s
    j
    • 4
    • 10
  • n

    Nike Gurin-Petrovych

    06/24/2021, 3:37 PM
    Hi there! Sorry if it's a dumb question but I can't figure out what's wrong, I'm new to Prefect. My setup: • Prefect Cloud to orchestrate • a docker container with prefect installed In the container I registered a flow with
    prefect register --project nike_dev --path flows/test.py
    , it's sucessfuly registered, I can see it in my dashboard. I also have an agent running in the container
    [2021-06-24 15:27:12,585] INFO - agent | Starting LocalAgent with labels ['93d6256207bd']
    [2021-06-24 15:27:12,586] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    [2021-06-24 15:27:12,586] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
    [2021-06-24 15:27:13,124] INFO - agent | Waiting for flow runs...
    In the dashboard I manually scheduled the flow to run immediately but the agent doesn't pick it up https://d.pr/i/j55IrO. How can I debug it further? What can be wrong?
    k
    • 2
    • 3
  • j

    Justin Liu

    06/24/2021, 4:33 PM
    Hi all I followed this guide to set up my prefect enviornment with ecs fargate, and now I’m trying to use great expectations as part of the flow runs. As far as I know, the files for GE have to be on the same filesystem that’s running the flow, so does anyone know how I can get these files onto every ecs instance in a safe manner?
    k
    z
    s
    • 4
    • 7
  • g

    Gabriel Montañola

    06/24/2021, 9:27 PM
    Hi there folks, I need a little hand from the
    Prefect Cloud
    team. I want to upgrade my Legacy account but I need to add the CTO as admin because he owns the credit card. Long story short: I demoted my user (I know, it's dumb 😖*) and I have no way of inviting the CTO so he can pay for the upgrade.* If someone can help me with this, please reach me via DM! Thanks :)
    ✅ 1
    • 1
    • 1
  • m

    Mark McDonald

    06/24/2021, 9:53 PM
    Greetings again - I have another question about automations and the API mutations for it. For automations related to state changes, you have a mutation called "create_flow_run_state_changed_hook" in which you provide a list of flow_group_ids to assign the hook to an action. It seems sla automations are handled differently. For these, I have to create flow group sla per flow (you can't provide a list of flows) using the "create_flow_group_sla" mutation. Is this understanding correct? What do I call to assign this sla to an action?
    m
    • 2
    • 6
  • m

    Marko Herkaliuk

    06/25/2021, 8:04 AM
    Hi there. I have a similar question as Gabriel. I am now testing the cloud (proof of concept) on my work mail, with my card. When we create a company account and add my mail to it, will I have problems with accounts?
    z
    • 2
    • 2
  • a

    ash

    06/25/2021, 9:15 AM
    Hi Prefect Team , I want to do POC for prefect cloud and wanted to know difference between standard and starter features. Can someone please elaborate on 1st and 4th point of standard pack in below shown picture.
    z
    • 2
    • 3
  • a

    Anton Rasmussen

    06/25/2021, 12:45 PM
    Hi all, We use the Prefect cloud standard subscription and need invoices for the amounts charged to our credit card. I haven't been able to find this in the web UI. Can someone here help me get my hands on invoices? Thanks!
    z
    • 2
    • 3
  • m

    Mark McDonald

    06/25/2021, 2:40 PM
    Hi - I have another question about automations, specifically state change automations. When creating the hook through the API using the mutation "create_flow_run_state_changed_hook_input", it says that if you leave the states argument empty in the mutation, all states will trigger the hook. However, I've done this and then looked at the hook from the API and I don't see any states. When I create the automation through the UI for all states, and then I look at the hook in the API, all the states are populated.
    z
    m
    • 3
    • 12
  • n

    Nike Gurin-Petrovych

    06/25/2021, 3:11 PM
    Is there a way to register flows recursively from a folder?
    prefect register --project "my project" --path /api/flows/
    registers only flows that are in
    api/flows
    but not in
    api/flows/subfolder
    m
    • 2
    • 2
  • j

    Josselin Girault

    06/25/2021, 3:12 PM
    Hello ! I've found a possible bug, can't investigate a lot more so I'll be posting here. I've been using Gitlab storage, and after a package update I started getting
    GitlabGetError 404: Project Not Found
    errors when fetching flows. Turns out
    python-gitlab
    broke something from 2.7.0 onwards. I assume it's not an issue for prefect to solve but maybe it can help someone.
    k
    m
    t
    • 4
    • 6
  • a

    Anton Rasmussen

    06/25/2021, 3:35 PM
    Hi all, Using the Standard cloud subscription, somehow, we're now in a situation where some team members have the role Administrator others Read Only, and I'm unable to change these roles. Is it expected to not be able to change roles (as Administrator) on this plan? If so, how do we get all users to the same role (and what role would that be?). Thanks!
    k
    • 2
    • 3
  • t

    thebuleon29

    06/25/2021, 4:33 PM
    Hi guys, I am looking at the possibility to have loops in a flow. The existing LOOP feature that I found in the doc is interesting, but would it be possible to do the same with several tasks ? Like for example Task A produces a result, then Task B uses this result to produce another one, then Task C uses this second result, does its own thing with it and, if a condition is not met, calls A again. I know it sort of contradicts the meaning of a DAG, but it would be really useful for us... Is it an existing feature?
    k
    • 2
    • 6
  • m

    Mark McDonald

    06/25/2021, 5:59 PM
    Hi - sorry to keep peppering yall with questions about the automations api. I'm trying to add/configure automations into my flow deployment process. I've gotten it figured out with state change automations, but I'm struggling with SLA automations. Basically, I'm trying to write a query to see if a SLA automation already exists for a given flow_group_id and if it does not, I will create the flow_sla_config, and then the flow_sla_failed_hook. So my question is, how can I query if a flow_sla_config exists for a given flow (using the flow_group_id)?
    • 1
    • 2
  • n

    Nike Gurin-Petrovych

    06/25/2021, 7:17 PM
    Hey all! Is there a way to see all flows as a graph? Having a lot of dependencies between them (the dependencies are managed with StartFlowRun) I'd like to visually see them and trigger specific flows manually. I'm using Prefect Cloud.
    k
    j
    • 3
    • 4
  • m

    matta

    06/25/2021, 9:26 PM
    Double-checking - I need all the dependencies for a Flow installed on the machine that Registers it, right? Even if it runs elsewhere?
    k
    z
    c
    • 4
    • 7
  • m

    matta

    06/25/2021, 10:26 PM
    Seeing this.
    You have no currently running Agents configured to pick up flow runs without labels
    How do I set a Local Agent to pick up flow runs without labels? And/or see what labels my agents are looking for? I don't see where my Agent's labels are.
    k
    • 2
    • 11
  • m

    matta

    06/26/2021, 1:41 AM
    Hrm, just tried to use a flow with a
    StartFlowRun
    task, with the storage for both being in
    git
    . The underlying flow works, but the new flow fails with
    GitProtocolError('Repository not found.')
    k
    • 2
    • 6
  • c

    CA Lee

    06/26/2021, 10:19 AM
    Hello, has anyone run into Prefect registering the same script twice when calling
    prefect register
    from the CLI? Storage: S3 Execution: ECS Prefect version:
    0.14.22
    vscode ➜ /workspaces/flowstate/flows (main ✗) $ prefect register --project prefect_project_name -p project_path
    
    Collecting flows...
    
    Processing 'project_path/flow_one.py':
      Building `S3` storage...
    [2021-06-26 10:10:44+0000] INFO - prefect.S3 | Uploading script /workspaces/github_project/flows/prefect_project_name/flow_one.py to flow_one/2021-06-26t10-10-44-047892-00-00 in aws-ecs-flows
      Registering 'flow_one'... Done
      └── ID: 41b7ea82-6cee-47c9-871c-0155267e6373
      └── Version: 1
    
    Processing 'project_path/flow_two.py':
      Building `S3` storage...
    [2021-06-26 10:10:45+0000] INFO - prefect.S3 | Uploading script /workspaces/github_project/flows/prefect_project_name/flow_one.py to flow_one/2021-06-26t10-10-44-047892-00-00 in aws-ecs-flows
    -------------------------------------------------------------------------------------------------------------------- * SAME FILE * -------------------------------------------------------------
    [2021-06-26 10:10:45+0000] INFO - prefect.S3 | Uploading script /workspaces/github_project/flows/prefect_project_name/flow_one.py to flow_two/2021-06-26t10-10-45-180059-00-00 in aws-ecs-flows
      Registering 'flow_two'... Done
      └── ID: f03469a7-c463-486f-a032-53d0fd4f265c
      └── Version: 1
    
    ======================== 2 registered ========================
    This results in an error when attempting to execute `flow_two`:
    Failed to load and execute Flow's environment: ValueError("Flow 'flow_two' not found in file. Found flows:\n- 'flow_one'")
    k
    • 2
    • 13
  • b

    Ben Muller

    06/26/2021, 10:42 AM
    Hey prefect, me again, sorry! So, I have an Ecs service running with my agent. All good. I used the launch type argument as EC2 for starting my Ecs agent in my cluster, but when I start a flow with ECSRun it still starts it as a FARGATE task. I'm trying to utilise the compute that the service is running on. I've read the docs, what could I be missing?
    k
    k
    • 3
    • 17
Powered by Linen
Title
b

Ben Muller

06/26/2021, 10:42 AM
Hey prefect, me again, sorry! So, I have an Ecs service running with my agent. All good. I used the launch type argument as EC2 for starting my Ecs agent in my cluster, but when I start a flow with ECSRun it still starts it as a FARGATE task. I'm trying to utilise the compute that the service is running on. I've read the docs, what could I be missing?
Do I need to set launch-type on the ECSRun object too?
So when I add the kwarg in my
ECSRun
object for
launchType: "EC2"
I run into an error on the agent launching the job
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Assign public IP is not supported for this launch type.
Can someone help me out on this one?
Hoping there are some Aws experts that can help me on the. More information... The reason we want to launch the task onto the ec2 instance is so that we can enable caching on the instance of the Docker image, fargate pulling the image every time a task runs seems inefficient for our use case.
Hoping you can advise on this one @Kevin Kho 🙏
k

Kevin Kho

06/27/2021, 4:47 AM
I saw this but didn't reply cuz I'm not 100% sure but I think you want this?
Oh my bad I see you already use it on the agent
What Prefect version are you on?
What does your
requiresCompatibilities
look like here ?
b

Ben Muller

06/27/2021, 5:01 AM
I'm on the latest version.
I haven't used a task definition, just the default cli command `prefect agent ecs start \ --token $PREFECT__CLOUD__AGENT__AUTH_TOKEN \ --task-role-arn=$TASK_ROLE_ARN \ --log-level INFO \ --label prefect-agent \ --name prefect-ecs-agent \ --launch-type EC2``
Isn't the requiresCompatabilities the same as the launch type argument? Just to be clear, my agent is running on ec2 (within ecs), just my flows aren't deployed there. I have a feeling it's because the default network mode is awsvpc... But not exactly sure
Anyone that might be able to assist in your team @Kevin Kho?
k

Kevin Kho

06/27/2021, 6:00 AM
On Monday I can ping someone
b

Ben Muller

06/27/2021, 6:01 AM
That would be amazing. Thanks so much. It's kind of the last hurdle for us to get this into prod. 🙏
k

Kathryn Klarich

07/02/2021, 3:51 PM
Hi @Ben Muller and @Kevin Kho did you all find an answer for this problem? I am having the same issue
k

Kevin Kho

07/02/2021, 3:52 PM
There is a workaround posted at the bottom of my comment here
👍 1
View count: 5