https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • m

    Miguel

    03/23/2021, 5:25 PM
    Hello everyone, I'm trying to execute a flow using an existent dask cluster but I'm getting the following error:
    [2021-03-23 17:16:53+0000] ERROR - prefect.my-flow | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'prefect'")
    This is what I'm doing:
    executor = DaskExecutor("my-dask-cluster-ip")
    flow.run(executor=executor)
    If I use other executors (LocalExecutor or LocalDaskExecutor) it runs fine. Any idea on what I can do to address this issue?
    m
    • 2
    • 8
  • c

    Charles Leung

    03/23/2021, 9:34 PM
    hey guys! i have the following error:
    INTERNAL_SERVER_ERROR: Variable "$input" got invalid value null at
            "input.states[0].task_run_id"; Expected non-nullable type UUID! not to be null.
    
    The GraphQL query was:
    
        mutation($input: set_task_run_states_input!) {
                set_task_run_states(input: $input) {
                    states {
                        message
                        status
                        id
                }
            }
        }
    
    The passed variables were:
    
        {"input": {"states": [{"state": {"message": "Trigger was \"all_successful\" but some of the upstream tasks failed.", "_result": {"location": null, "__version__": "0.14.11", "type": "Result"}, "cached_inputs": {}, "context": {"tags": []}, "__version__": "0.14.11", "type": "TriggerFailed"}, "task_run_id": null, "version": null}]}}
    
    what is the issue? I'm not sure how to debug this
    j
    • 2
    • 9
  • m

    Michael Hadorn

    03/24/2021, 7:43 AM
    Hi there 🙂 Is it by design that scheduled flow will not use updated flow default parameters. e.g. • schedule a flow • let them run • update the default params • for the next run the old/wrong default values are used On Prefect local server 0.14.12 Btw: There is no way, to specify params for a specific schedule, right? That would be awesome and necessary in a way.
    a
    j
    m
    • 4
    • 6
  • c

    Clemens

    03/24/2021, 9:55 AM
    Hi, we are running prefect server with the helm chart installation on a GKE kubernetes cluster. We noticed that some flows fail without us being able to see why. In particular we receive as the final log that marks a working flow
    Finished task run for task with final state: 'Running'
    after that the flows state is completely off and it get’s hard to understand what happens. But already that log seems wrong. Any clue about why and when a task might finish with a running state?
    m
    • 2
    • 2
  • m

    Michael Hadorn

    03/24/2021, 11:10 AM
    Is it right, that a run with warnings will not be reflected in the gui (e.g. with a yellow color)? or is this a setting?
    j
    • 2
    • 2
  • j

    Josh Greenhalgh

    03/24/2021, 1:44 PM
    Anyone got any ideas of how to force the postgres chart used in the helm chart to deploy on a particular node pool? Have tried this;
    postgresql:
          postgresqlPassword: ***
          nodeSelector:
            <http://cloud.google.com/gke-nodepool|cloud.google.com/gke-nodepool>: some-node-pool
    m
    • 2
    • 14
  • j

    Josh Greenhalgh

    03/24/2021, 2:37 PM
    I have also just bumped my version to
    core-0.14.12
    and I am getting this error;
    {'type': ['Unsupported value: Module']}
    Has module storage been removed?
    j
    • 2
    • 2
  • c

    Charles Leung

    03/24/2021, 9:54 PM
    Hello Prefect team! I was curious about agent monitoring; do agents publish metrics? If so, how does that work? We're looking to start the hybrid model w/ cloud and wanted to see how this would work
    c
    m
    • 3
    • 4
  • m

    Milton

    03/24/2021, 10:30 PM
    Hi prefect team, I happen to see you “run prefect on prefect” in one of your job postings, so I am wondering how you run services via prefect? Are tasks/flows supposed to be short living?
    m
    • 2
    • 4
  • d

    Dominique

    03/25/2021, 2:51 AM
    Hello everyone! I am working with a server shared between JupyterLab and Prefect, and it happened to me recently that when running a code in JupyterLab, it turned out wrong and caused JupyterLab to crash, and along with it, Prefect UI as well. As a result of this I was investigating 2 things but I did not find much information in the documentation and that is why I turn to you. 1: Is it possible to have 2 servers, one for JupyterLab and the other for Prefect, but where you can still work hand in hand with both? (I would use this to solve the issue by ensuring that, if one of them falls, the other one doesn't, but if you come up with another solution I am all ears) 2: Is there a way for Prefect to notify me when the UI goes down?
  • j

    Jillian Kozyra

    03/25/2021, 7:32 AM
    Hey, i’m getting
    ModuleNotFoundError: No module named 'data_flows'
    when I try to register my flow locally using docker storage (specifically it’s failing the
    cloudpickle_deserialization_check
    healthcheck). we’re using fully qualified imports in our flows (e.g.
    from data_flows.utils.data_extraction tasks import thing
    ) to satisfy mypy, so i’m hoping there’s a way to get this working as well. i think the config matches the tutorial, and i’ve tried different values for the python path without any luck. Here’s our Docker setup:
    Docker(
        registry_url=ecr_registry_url,
        python_dependencies=python_dependencies,
        files={
            "/Users/jilliankozyra/projects/projectname/data_flows/utils/__init__.py": "/data_flows/utils/__init__.py",
            "/Users/jilliankozyra/projects/projectname/data_flows/utils/install_dependencies.py": "/data_flows/utils/install_dependencies.py",
            "/Users/jilliankozyra/projects/projectname/data_flows/utils/data_extraction_tasks.py": "/data_flows/utils/data_extraction_tasks.py"
        },
        env_vars={
            "PYTHONPATH": "$PYTHONPATH:data_flows/"
        },
    )
    m
    • 2
    • 13
  • m

    Milton

    03/25/2021, 3:24 PM
    Congrats on the release of 0.14.13! I am just wondering if we could use prefect build to validate the flows/tasks in our code before registering them? Or are you planing to do something like prefect validate?
    m
    j
    • 3
    • 5
  • c

    Charles Leung

    03/25/2021, 4:37 PM
    Hey Prefect Team! My team is moving towards Cloud, and we wanted to know if there was any docs on the following enterprise features: • SLA service for configuring special alerts • SSO integration • Special Types of Cloud Hooks for notificaitons • Audit trail of tenant activity • Secure Runtime Secrets
    c
    • 2
    • 2
  • a

    Adam Lewis

    03/26/2021, 7:29 PM
    What versions of kubernetes are supported for the helm chart deployment of prefect-server? I'm trying to target 1.14.8
    n
    • 2
    • 3
  • c

    Charles Leung

    03/26/2021, 7:43 PM
    Hey Prefect Team! I see that to use gitlab storage we have to use access tokens, which i've tested and worked great! However, our team's standards typically use deploy tokens for services. Is this supported?
    m
    • 2
    • 5
  • d

    Dana Merrick

    03/26/2021, 9:18 PM
    How do I get the k8s logs from a
    RunNamespacedJob
    task?
    n
    j
    • 3
    • 34
  • m

    Michael Hadorn

    03/29/2021, 12:57 PM
    Hi there Is there a way to disable the quick run? Also in the upcoming runs... It's quite dangerous...
    d
    m
    • 3
    • 7
  • g

    Gopinath Jaganmohan

    03/30/2021, 2:28 AM
    Hi Team, I'm evaluating Prefect as data flow pipeline and started learning and trying out. My setup is Prefect Server, LocalAgent running inside a docker and creating task from jupyter Notebook and registering flow using S3 storage. I could all but when I run getting error "Failed to load and execute Flow's environment: AttributeError("'NoneType' object has no attribute 'setup'")"I know making some simple mistake, any help would be appreciated. Flow is simple hello world.
    n
    • 2
    • 4
  • n

    Nathan Walker

    03/30/2021, 1:00 PM
    Hey folks, a question about Schedules here: When a Flow is run, if the hard-coded Schedule/Clock is different from the Schedule/Clock in the API, will anything strange/bad happen? Or is updating the API over the hard-coded values how the UI does scheduling? For example, let's say I code up a Flow with a Schedule and a few Clocks, then register it to Prefect Server. I can go to the API and query my Flow and see its schedule and all that. If I use the API to mutate my flow's Clock(s), I'm assuming that whatever data is in the API will be what the Agent will look at to see if it's time to run the Flow. But when the Flow actually goes to run, will the Schedule objects match whatever is coming from the API or will they come from the hard-coded Flow? I want to avoid having the API say "Run this flow at 3am with Params X, Y, and Z" but when the Flow actually goes to run, it says, "This flow will run at 2pm with Params A, B, and C."
    m
    • 2
    • 3
  • d

    Diego Alonso Roque Montoya

    03/30/2021, 3:38 PM
    Is there a way to specify a resource manager when using the imperative API?
    m
    • 2
    • 1
  • d

    Diego Alonso Roque Montoya

    03/30/2021, 6:26 PM
    I'm getting errors whenever I try pushing a prefect flow that's about 1500 tasks big. Is there a reason this happens?
    z
    • 2
    • 23
  • r

    Ross

    03/31/2021, 9:16 AM
    Hi, is it possible to use
    prefect.engine.flow_runner.FlowRunner
    or
    prefect.engine.cloud.flow_runner.CloudFlowRunner
    with a prefect server backend? I am wanting to run a flow and get task results so wanting to use the arg
    return_tasks=[task1,task2]
    . But I don't think I can do this with
    prefect.Client().create_flow_run(version_group_id="redacted")
    ?
    n
    j
    • 3
    • 7
  • t

    Tomás Emilio Silva Ebensperger

    03/31/2021, 1:38 PM
    Hello everyone! I built a simple ETL and it is working perfectly. When I run the agent in the cloud I get an error saying "cannot pickle psycopg2", any clues?
    n
    • 2
    • 8
  • j

    Jonathan Wright

    03/31/2021, 5:26 PM
    Hi, how do I delete a tenant which I created like this? Thank you.
    prefect server create-tenant -n data-eng
    • 1
    • 1
  • t

    Tomás Emilio Silva Ebensperger

    04/02/2021, 12:51 PM
    I set up a virtual environment in my ubunutu machine for prefect. my question is why are results saved in the home/ubuntu/.prefect/results and not on the environment
    z
    • 2
    • 2
  • s

    Sidd

    04/02/2021, 5:06 PM
    Hi folks, Im running into this weird error where the default prefect tables are not getting created when the server starts up for the first time. I actually got the whole thing working in our staging env, but when I pushed to our prod env, it seems like the default tables don’t exist. I don’t see the tables in prod hasura, like the ones I see in our staging hasura. Any insight/help would be much appreciated!
    m
    • 2
    • 55
  • e

    eric lumsden

    04/05/2021, 12:11 AM
    I was able to get prefect installed, register a flow on prefect cloud, and now I would like to use prefect cloud to run flows stored on my AWS instance. I am trying to follow this, but even something as simple as installing the package at step 1 isn't working for me
    pip install prefect[aws]
    zsh: no matches found: prefect[aws]
    k
    • 2
    • 4
  • l

    Luca Sorgiacomo

    04/05/2021, 7:32 AM
    Hi all, I’m new to Prefect. Is there anything I should know about using
    async def
     and
    await
    in defining and within tasks?
  • l

    Luca Sorgiacomo

    04/05/2021, 7:33 AM
    I haven’t found anything on the docs, besides using Dask for parallelism, but I would just want to not wait for I/O, it’s not about complex computing tasks.
    m
    • 2
    • 2
  • e

    eric lumsden

    04/05/2021, 5:30 PM
    hey guys, I am running prefect agent ecs start in terminal and getting this error. What is the syntax it wants me to use?
    ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
    k
    • 2
    • 78
Powered by Linen
Title
e

eric lumsden

04/05/2021, 5:30 PM
hey guys, I am running prefect agent ecs start in terminal and getting this error. What is the syntax it wants me to use?
ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
k

Kevin Kho

04/05/2021, 5:35 PM
Hey @eric lumsden, can you give me more details about your run configuration?
e

eric lumsden

04/05/2021, 5:43 PM
sure, I set up a new user under IAM in aws, the I used aws configure, set values for key ID, secret key, default region, and then left default output format as none. Ran the prefect agent ecs start... not sure what/where else it wants something specified
k

Kevin Kho

04/05/2021, 5:45 PM
Are you using the default VPC?
e

eric lumsden

04/05/2021, 5:47 PM
I'm not sure. How would I check?
k

Kevin Kho

04/05/2021, 5:50 PM
I guess this will give instructors are to check your VPCs in the UI. Do you have UI access? Just check if you only have the default VPCs or if you have additional ones.
e

eric lumsden

04/05/2021, 5:53 PM
I have UI access. I just browsed into "Your VPCs" and it is empty. What should my next step be?
wait no I had to select the region
we have additional VPCs
k

Kevin Kho

04/05/2021, 5:54 PM
Ok. Just making sure, the aws config is in the right region?
e

eric lumsden

04/05/2021, 5:56 PM
yes it is all set to us-east
k

Kevin Kho

04/05/2021, 5:57 PM
When you go to ECS, do you see the default cluster?
Or any cluster provisioned?
e

eric lumsden

04/05/2021, 6:01 PM
oh boy ECS doesn't look like it has been set up at all. There are no clusters. Do y'all have a doc?
k

Kevin Kho

04/05/2021, 6:02 PM
Let me look around
https://towardsdatascience.com/serverless-data-pipelines-made-easy-with-prefect-and-aws-ecs-fargate-7e25bacb450c#5cdb
aws ecs create-cluster
try this?
or really just create the default in the UI
e

eric lumsden

04/05/2021, 6:11 PM
oh man I was actually trying this guide but then I saw Fargate was deprecated for y'all
ok cluster is created after the aws create cluster but same error on prefect agent start. Is there work to do with configuring the VPC still?
so that error is "ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`"
k

Kevin Kho

04/05/2021, 6:15 PM
let me pull someone who can help better
e

eric lumsden

04/05/2021, 6:15 PM
ok, sorry. I am very new to aws config
k

Kevin Kho

04/05/2021, 6:17 PM
No worries. I pinged someone to help 🙂
Btw, the fargate agent is deprecated but you can still use the ECS agent to run ECS tasks on Fargate
e

eric lumsden

04/05/2021, 6:25 PM
thanks for your help. I'll sit on my hands so I don't mess up my config further
k

Kevin Kho

04/05/2021, 6:27 PM
Can you try doing something like this?
RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
                    run_task_kwargs={
                        "networkConfiguration": {
                            "awsvpcConfiguration": {
                                "assignPublicIp": "ENABLED",
                                "subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
                                "securityGroups": ["sg-123456"],
                            }
                        }
                    },
)
then pass it into your flow with
run_config=RUN_CONFIG
. get the VPC configuration from the default subnets
I assume you didn’t add a VPC when you created the cluster right?
e

eric lumsden

04/05/2021, 6:30 PM
I did not add a new VPC
I'm sorry I don't appear to be getting this. Where am I supposed to run the RUN_CONFIG declaration?
k

Kevin Kho

04/05/2021, 6:44 PM
Inside Python code and then pass this object to your
Flow()
call.
RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
                    run_task_kwargs={
                        "networkConfiguration": {
                            "awsvpcConfiguration": {
                                "assignPublicIp": "ENABLED",
                                "subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
                                "securityGroups": ["sg-123456"],
                            }
                        }
                    },
)

with Flow(name="ecs-test", storage=STORAGE, run_config=RUN_CONFIG, executor=EXECUTOR) as flow:
  print_log()
flow.run()
Let me know if you don’t know where do find the subnet and security group.
e

eric lumsden

04/05/2021, 7:42 PM
am I missing a dependency?
k

Kevin Kho

04/05/2021, 7:50 PM
is there a chance you named your file
flow.py
?
I think this might be an import conflict. Or do you have a folder named
prefect
inside?
Actually sorry I just noticed this is interactive mode. I don’t think you’re missing any modules. This error might be because the function inside your flow is undefined
Replace
print_log
with
print("1")
to test this block. You don’t have the STORAGE and EXECUTOR defined though so this will error. Do you have the original snippet that caused the error? Maybe we can take a look at that
e

eric lumsden

04/05/2021, 8:02 PM
yeah I was about to type up that I shoved it all into a script and was getting an exception about STORAGE being undefined
k

Kevin Kho

04/05/2021, 8:04 PM
The
STORAGE
is to point to where your code lives. For ECS it might point to the Docker image location. Something like:
STORAGE = Docker(
    registry_url="<http://1234.dkr.ecr.us-east-2.amazonaws.com|1234.dkr.ecr.us-east-2.amazonaws.com>",
    image_name="ecs-test",
)
e

eric lumsden

04/05/2021, 8:04 PM
the original snippet was me trying to run "prefect agent ecs start"
and I suppose I have to create the docker image location on AWS?
k

Kevin Kho

04/05/2021, 8:05 PM
oh yeah oof! let me give you a YAML file instead that gives the VPC info that we can attach to the command line command. will reply a bit later
e

eric lumsden

04/05/2021, 8:06 PM
thank you for your patience
k

Kevin Kho

04/05/2021, 8:06 PM
Regardless though, I think this ECS documentation is definitely something we need to improve on. I will note that.
and no problem!
https://prefect-community.slack.com/archives/CL09KU1K7/p1611215645018300?thread_ts=1610980811.204400&amp;cid=CL09KU1K7
This thread shows how to create a YAML file, add the taskRoleArn and the execurionRoleArn.
And then call it like this: https://docs.prefect.io/orchestration/agents/ecs.html#custom-runtime-options . Ultimately, there have been a couple of people who have run into this same issue. I have read that EKS was easier to setup so that may be an option. We have a couple of PR’s around this to make the experience easier and a minimum working example is definitely something we have in our minds
e

eric lumsden

04/06/2021, 3:38 PM
ok I could use a little help finding the value for networkConfiguration. Getting this error: ValueError: Failed to infer default networkConfiguration, please explicitly configure using
--run-task-kwargs
I'm guessing I need to set the networkMode to an existing vpc. Unfortunately my environment shifted it's hosting region overnight so I am pointed somewhere with no VPCs. How do I point to the new region and select a VPC?
k

Kevin Kho

04/06/2021, 3:41 PM
The region is on your CLI config. You start out with a YAML like this:
networkMode: awsvpc
cpu: 256
memory: 512
containerDefinitions:
  - name: flow
requiresCompatibilities:
  - FARGATE
executionRoleArn: arn:aws:iam::xxxxxxxxxx:role/tf-iam-role-prefect-ecs-tasks <--- change to whatever you need
and then you add VPC info (looking for an example)
e

eric lumsden

04/06/2021, 3:58 PM
ok I repointed my CLI to the new region, recreated the S3 bucket in the new region, grabbed one of the VPC ids from the new region and put it in the networkMode parameter in YAML... still get the same error complaining about my network configuration
k

Kevin Kho

04/06/2021, 4:00 PM
yep i pinged someone on the team to give me a more fleshed out example. will get back to you
e

eric lumsden

04/06/2021, 4:00 PM
ok, thank you. I'm probably missing something basic
it looks like networkMode should be awsvpc, I may be missing a parameter to specify the actual vpc
k

Kevin Kho

04/06/2021, 4:21 PM
ok sorry i mislead you from the python snippet yesterday. just to recap and start fresh, the problem is we need to give the VPC details
create a file 
run_task.yaml
 with this:
networkConfiguration:
  awsvpcConfiguration:
    subnets:
      - 'subnet-1234'
    securityGroups:
      - 'sg-1234'
    assignPublicIp: 'ENABLED'
run 
prefect agent ecs start --run-task-kwargs /ecs/run_task.yaml
ideally to simplify, subnet and security groups should be the default ones from your AWS account
ignore the previous YAML. that is when we start to get into tasks
e

eric lumsden

04/06/2021, 4:37 PM
ok... new error. I went for the subnet arn on the subnet parameter for the YAML file, hope that was correct over the subnet ID. I went the security group ID for lack of having another option. Now when I run prefect agent ecs start --run-task-kwargs s3locationwithyaml, I get prefect.utilities.exceptions.AuthorizationError: No agent API token provided. I went in to my cloud prefect account and generated an API token. Where do I specify its use?
k

Kevin Kho

04/06/2021, 4:59 PM
The orchestration tutorial has snippets how to configure: https://docs.prefect.io/orchestration/tutorial/overview.html#install-prefect
I also think you can revisit that tutorial you had now that we fixed the networkConfig 🙂
e

eric lumsden

04/06/2021, 5:12 PM
ok using that I was able to authenticate with my generated API token to prefect cloud. I was able to kickoff a flow within prefect cloud that does not do anything with our AWS. When I go back and run
prefect agent ecs start --run-task-kwargs <s3://bucketname>
I still get the same error
prefect.utilities.exceptions.AuthorizationError: No agent API token provided.
we've been talking a while so maybe let me restate what I want to do so we're on track: trying to spin up ECS on AWS and connect it to prefect cloud so that I can kick off and manage flows in prefect cloud that will run on AWS
so to that end we made a configuration file in YAML for ECS and have it on AWS, but I think I am missing the part where there is actual connection between the two cloud environments. Presumably there should be a listener set up on AWS waiting for inputs from Prefect Cloud that will then use the ECS to run flows. Where is the doc for that listener service/bridge between the two cloud environmnets
k

Kevin Kho

04/06/2021, 5:21 PM
Yeah I’m looking into it.
So I believe the issue here is that the agent needs the RUNNER token. Just to make sure, you’re doing the
prefect agent ecs start
on your local machine and you want the local machine to kick off ECS jobs right?
But you were able to get the basic flow running on your local agent from cloud?
Or are you using an EC2 instance to kick off the ECS job like the tutorial?
The agent is also hold the AWS credentials (like seen in the tutorial)
e

eric lumsden

04/06/2021, 5:53 PM
I was able to get a basic flow running on the cloud from my local agent. Basically, a hello world. I set my AWS creds in my local aws CLI and then called the prefect agent to start ec2 on my local. That's basically all the tutorial does, so I thought this was sufficient. What do I need to change so the prefect agent will know to use my aws credentials?
k

Kevin Kho

04/06/2021, 5:55 PM
Just to clarify, you mean start ECS from the local? or did you mean an EC2 instance?
e

eric lumsden

04/06/2021, 6:02 PM
At this point, I can't honestly say I am sure. Whichever will allow me to use prefect cloud to monitor and orchestrate data flows within my AWS environment
k

Kevin Kho

04/06/2021, 6:08 PM
I feel there are too many moving parts to ECS (we still have yet to get to Docker images and Execution/Task Role Permissions) that we might not be productive step-by-step debugging like this. What I’ll do is write an end-to-end tutorial around this (some time in the next week) and ping you when I complete it and that’ll be a much better starting point for us to get you up and running with ECS. Does that work?
This is a pain point on our end that a lot of people can’t get up and running on ECS independently. Having it in our docs is probably needed.
e

eric lumsden

04/06/2021, 6:30 PM
that'd be great, thank you Kevin
k

Kevin Kho

04/06/2021, 6:36 PM
Thank you for your patience as well 🙂
View count: 3