https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • y

    Yash

    01/21/2022, 12:08 PM
    Hi @Anna Geller @Kevin Kho, I have scheduled a flow programatically, but its not geting deleted when Im trying to delete that schedule using mutation
    delete_flow_group_schedule
    . Is there any way to delete a 'programatically created schedule' by using graphql mutation or client through code.
    a
    k
    • 3
    • 7
  • l

    Lana Dann

    01/21/2022, 6:43 PM
    hi! our team is currently doing a security audit on prefect right now. what’s the easiest way to get an example of the the metadata that’s being sent to prefect cloud? i.e. how can i view the API calls to prefect cloud that we’re making?
    k
    • 2
    • 6
  • j

    Jacob Longhurst

    01/24/2022, 6:20 PM
    Hello, My team is trying to get the k8s prefect agent to run in k8s. Whenever we spin up the agent it fails on
    registering agent
    . I took a look through the code and it seems that on spin up the agent attempts a query in the db to see if it already exists. If it doesn’t then it registers itself by adding a new record of itself to the db. We are also not currently using the prefect cloud server but rather local instances of apollo, graphql, hasura, etc running in our k8s cluster Here is the stack trace
    [2022-01-24 18:15:41,310] INFO - agent | Registering agent...
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1053, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.6/site-packages/click/core.py", line 754, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/prefect/cli/agent.py", line 331, in start
        start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/prefect/cli/agent.py", line 140, in start_agent
        agent.start()
      File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 189, in start
        self._setup_api_connection()
      File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 910, in _setup_api_connection
        self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
      File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 858, in _register_agent
        agent_config_id=self.agent_config_id,
      File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 2107, in register_agent
        agent_config_id=agent_config_id,
      File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 570, in graphql
        raise ClientError(result["errors"])
    prefect.exceptions.ClientError: [{'message': '"\'Box\' object has no attribute \'data\'"', 'locations': [{'line': 2, 'column': 5}], 'path': ['register_agent'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '"\'Box\' object has no attribute \'data\'"'}}}]
    I can also provide the yaml for the k8s
    a
    • 2
    • 8
  • r

    Romain

    01/25/2022, 8:46 AM
    Hello folks, I recently upgraded our prefect server to 2022.01.12. I also upgrade our prefect version to 0.15.12. While trying to register our flow, I get this error:
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "flow_registration.py", line 136, in <module>
        register_flows(args.local)
      File "flow_registration.py", line 128, in register_flows
        idempotency_key=flow.serialized_hash())
      File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1734, in register
        idempotency_key=idempotency_key,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1213, in register
        variables=dict(input=inputs),
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 555, in graphql
        retry_on_api_error=retry_on_api_error,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 459, in post
        retry_on_api_error=retry_on_api_error,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 740, in _request
        session=session, method=method, url=url, params=params, headers=headers
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 661, in _send_request
        raise ClientError(f"{exc}\n{graphql_msg}") from exc
    prefect.exceptions.ClientError: 400 Client Error: Bad Request for url: <http://prefect-server-apollo.emtrails:4200/graphql>
    
    The following error messages were provided by the GraphQL server:
    
        GRAPHQL_VALIDATION_FAILED: Unknown type "register_tasks_input". Did you mean
            "register_agent_input", "delete_agent_input", "delete_flow_input",
            "delete_tenant_input", or "log_insert_input"?
        GRAPHQL_VALIDATION_FAILED: Cannot query field "register_tasks" on type
            "Mutation". Did you mean "register_agent", "delete_task", "insert_task", or
            "update_task"?
    
    The GraphQL query was:
    
        mutation($input: register_tasks_input!) {
                register_tasks(input: $input) {
                    success
            }
        }
    Would anyone have encountered a similar issue?
    a
    j
    • 3
    • 3
  • a

    Alexander Melkoff

    01/25/2022, 2:24 PM
    Hello! I've just deployed Prefect Server into Kubernetes on AWS using official helm chart and I'm trying to make my first hello-world flow to work. I am using GitLab storage and I'm wondering what is the correct (and secure) way to pass
    GITLAB_ACCESS_TOKEN
    ?
    a
    s
    • 3
    • 13
  • j

    Jen Evenson

    01/25/2022, 6:35 PM
    Hello! I am trying to create Jira issues using the JiraTask and got this error (full text in thread):
    jira.exceptions.JIRAError: JiraError HTTP 400 url: <https://our_jira_url/rest/api/2/issue>
    	text: expected Object containing a 'name' property
    	
    	response text = {"errorMessages":[],"errors":{"assignee":"expected Object containing a 'name' property"}}
    k
    • 2
    • 2
  • l

    Lana Dann

    01/25/2022, 7:06 PM
    does prefect have anything built-in for datadog metrics? for example, if we wanted to track job success rate, job duration, etc. in datadog in addition to the cloud ui.
    k
    • 2
    • 1
  • a

    Anders Segerberg

    01/25/2022, 9:27 PM
    I have encountered a curious situation, where SOME mapped tasks appear to be "Zombies" -- they run for tens of minutes on end, when they average job is <1min; the UI shows the task status as "Running", with the run duration increasing. Of 40 Dask worker vCPUs, as Flow runtime increases, maybe 15 or so at any one point, will be these "zombies". What's strange is that it is possible to "jolt" these tasks out of this zombified Running state, by going into the UI for the Task. Once this happens, without fail, the task status will resolve to "Success" after a minute or so; and the Dask worker running it is freed up for a new task. (Usually, I find opening the Logs tab, the quickest way to do this.) It's clear that what we are doing, is indeed "jolting" the task awake, forcing it to finish its state transitions; this is observable in the increased workflow throughput, due to the now-freed vCPU Dask worker. So, I don't think this is simply a lag in the UI. Is this a known issue?
    k
    • 2
    • 18
  • b

    Ben Welsh

    01/26/2022, 12:17 PM
    Are there any examples of running local unit tests to verify your tasks are working? I'd like to have some pre-deployment checks in my GitHub actions.
    a
    • 2
    • 9
  • j

    Jonas Bolin

    01/26/2022, 12:37 PM
    Prefect beginner here. Can I set prefect server to run in the background, like
    docker-compose up -d
    would behave?
    a
    • 2
    • 2
  • b

    Ben Welsh

    01/26/2022, 2:09 PM
    What is the most straightforward example of creating your first simple cloud agent that can run a cloud registered flow without any help from my laptop?
    a
    • 2
    • 5
  • c

    Carlos Cueto

    01/26/2022, 9:22 PM
    I'm trying to deploy Prefect Server/UI to Azure Web App Service using Docker with docker compose and I'm trying to use the docker-compose.yml from https://github.com/flavienbwk/prefect-docker-compose/blob/main/server/docker-compose.yml. Has anyone successfully done this before? The UI is loading for me but I keep getting a "Couldn't connect to Prefect Server at *https://localhost:4200/graphql*" error.
    k
    • 2
    • 10
  • i

    Ian Armstrong

    01/27/2022, 12:14 AM
    Hello! I'm running prefect server on GKE using the helm chart. I have an ingress for the UI secured with oauth and a load balancer for Apollo. I'm able to use the graphql endpoint from Apollo and the CLI. However, when it comes time to connect to the graphql endpoint in the UI I'm getting an error message:
    Oops! It looks like something went wrong when trying to connect; make sure Prefect Server is running at the URL above and try again.
    Any advice on how to debug this? GKE already creates a firewall rule to allow ingress over tcp:4200. Maybe something else weird with the firewall? I previously had Apollo set up as in ingress, but I switched back to a load balancer in case that would simplify things.
    k
    a
    • 3
    • 4
  • y

    Yash

    01/27/2022, 11:04 AM
    Hi everyone, I trying to figure out how to register the prefect flows with hosted prefect server instead local. Is there any documentation for that?
    a
    • 2
    • 6
  • s

    Scarlett King

    01/27/2022, 12:15 PM
    Hi guys, I’ve just redeployed prefect server to AKS via helm chart and having trouble connecting to graphql from ui. How can I troubleshoot this?
    a
    m
    • 3
    • 11
  • b

    Ben Welsh

    01/27/2022, 6:17 PM
    Are there any examples of a Dockerfile for Prefect agents successfully deployed via Kubernetes?
    k
    • 2
    • 73
  • l

    Liam England

    01/27/2022, 7:30 PM
    Hi folks, Wondering if it's possible to launch gpu-enabled containers with a Docker Agent? Normally you would flag it when running a container like volumes, but I'm not seeing an obvious flag in the docs unless I've missed something.
    k
    • 2
    • 4
  • b

    Ben Welsh

    01/28/2022, 2:42 PM
    Thanks to all the help in this channel, yesterday I was able to launch my first cloud deployment with a Kubernetes agent running from a Docker source. I'm very happy with the progress but here's one thing I noticed: My tasks aren't running in parallel anymore. When I was in local mode, I had achieved that with a LocalDaskExecutor, which remains in my code after the switch to cloud hosting and execution. What is the simplest way to get my Kubernetes agent and Docker source doing Dask again? Is it as simple as replacing LocalDaskExecutor with DaskExecutor? Do I need to install some extra dependencies in my agent Dockerfile? Include more requirements in my Source's python_dependencies? Something else?
    🙌 2
    k
    • 2
    • 2
  • j

    Jamie Miller

    01/28/2022, 3:01 PM
    Hi guys, I'm working on my first prefect flow and keep running into this error below. I can't seem to find where it is referencing the missing argument, however, since it doesn't give me a specific line number
    TypeError: missing a required argument: 'target_date'
    k
    a
    • 3
    • 2
  • j

    Jack Chang

    01/28/2022, 6:59 PM
    Hey everyone! Ive set up an agent and my flow on prefect cloud. When I run my flow, I get:
    [28 January 2022 10:46am]: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/Users/jackchang/'")
    I went through some threads and added a gitlab repo as the storage and added that to my flow. Any ideas on what I should try next? 🙂
    k
    a
    • 3
    • 9
  • s

    Simon Hällqvist

    01/29/2022, 10:04 AM
    Hey guys - I’m new to Prefect but I’m considering using it at my startup, however, I have a question - how easy would it be to integrate Prefect with a web app for end user? Our users needs to be able to initiate a data processing job from our web app (not the Prefect UI) and also change some parameters of the job (what data to process). Does Prefect come with a REST API endpoint from which jobs can be ran and job parameters changed? Thanks! Simon
    a
    • 2
    • 3
  • p

    Paulo Benatto

    01/29/2022, 11:21 AM
    Good morning 🙂 I'm quite new in prefect and I would like an advice for a problem that I have. In the example below all numbers (the list) have to be processed before saving the number (save_number()). I would like to know how I can process one number per time. For the first number in the list i execute the whole flow, for the second number i execute the whole flow, and so on ... This is an simple example, I'm thinking more if i have to process a lot of data. Many Thanks
    from prefect import Flow, task
    import prefect
    
    import time
    
    from prefect.executors import LocalDaskExecutor
    
    
    logger = prefect.context.get("logger")
    
    
    @task
    def add_ten(x):
        time.sleep(1)
        return x + 10
    
    
    @task
    def save_number(y):
        <http://logger.info|logger.info>(f"Saving number: {y}")
    
    
    with Flow("reduce", executor=LocalDaskExecutor()) as flow:
        mapped_result = add_ten.map([1, 2, 3, 4, 5, 6, 7, 8, 9, 9])
        mapped_result_2 = add_ten.map(mapped_result)
        reduced_result = save_number(mapped_result_2)
    a
    • 2
    • 4
  • s

    Sanil Khurana

    01/29/2022, 10:28 PM
    Hi guys, I have been using Prefect but stuck at an issue. I am using ECSAgent and sometimes I get an error that says "An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.". I am running around 80 flows that all run every minute. I am planning to run more flows, maybe in 100s and all of them running per minute. Am I using Prefect incorrectly and can it handle this scale? I checked out this link from the docs - https://docs.prefect.io/orchestration/agents/ecs.html#throttling-errors-on-flow-submission. I am already running the ECS agent with the specified options for retry mode but these errors still happen. Is there any way to fix this? I don't really need Prefect to create new task definitions on every flow run, can I disable that behaviour? I am also using a custom task definition, could it be because of that?
    a
    • 2
    • 4
  • g

    Gagan Singh Saluja

    01/30/2022, 8:17 PM
    How can i use an ecs fargate cluster and run tasks in parallel? Like using fargate cluster for dask.
    a
    • 2
    • 6
  • b

    Ben Welsh

    01/30/2022, 10:48 PM
    Is there an established pattern for including dev/prod variations in a flow file. Basically, in dev I'd like to have the default local storage on my laptop and in prod I'd like to do a customized Docker storage with Google Artifact Registry. The hacky section of my brain is imagining some kind of if/else clause in my flow file that keys off an environment variable. But I'm not sure what that variable would be, and I wonder if there's a sturdier method for achieving this solution that I'm simply ignorant of.
    a
    • 2
    • 6
  • s

    Sylvain Hazard

    01/31/2022, 1:15 PM
    Hello there ! We're starting to see our Postgre database get a bit bigger than we would like it to be and were wondering how we could limit its size ? I suppose we could cleanup old flow runs and flow versions after a while but I have no idea how we could do that. Are there any good practices on how to do this ? Cheers !
    a
    • 2
    • 2
  • b

    Ben Welsh

    01/31/2022, 1:56 PM
    My next challenge trying to configure my Prefect
    LocalDaskExecutor
    to run mapped tasks in parallel inside a single GKE k8s container with what, if I understand the jargon correctly, is known as vertical scaling. Are there any resources that show the optimal way to do that?
    a
    • 2
    • 11
  • b

    Bo

    01/31/2022, 3:27 PM
    Hello everyone! I am trying to set up an ecs agent on fargate, but I don't have a "default" vpc set up, so I need to specify the network config, as I was getting this error:
    ERROR - agent | Failed to infer default networkConfiguration, please explicitly configure using
    --run-task-kwargs`` . However when I add the yaml path to the command, I start to receive this error:
    Error: No such option: --run-task-kwargs s3://*****/options.yaml
    Any thoughts?
    k
    • 2
    • 5
  • d

    Dotan Asselmann

    01/31/2022, 5:35 PM
    Hi everyone. Is there an equivalent for celery metrics exporter in prefect server? https://github.com/danihodovic/celery-exporter
    k
    • 2
    • 3
  • t

    Tony Liberato

    01/31/2022, 8:13 PM
    Good afternoon. I have been making great progress on our inital prefect workflow model. However, I ran into a snag I have no idea how to fix. I have a typical download/unzip/save to s3 workflow that I am trying to send to a generic docker agent (with Flow("<whatever>", storage = Docker()) as flow:). When I try to register, I get a healthcheck error: no module named bs4 / Flow uses module that is not importable. How do I get the flow to use bs4 on the docker agent?
    k
    • 2
    • 8
Powered by Linen
Title
t

Tony Liberato

01/31/2022, 8:13 PM
Good afternoon. I have been making great progress on our inital prefect workflow model. However, I ran into a snag I have no idea how to fix. I have a typical download/unzip/save to s3 workflow that I am trying to send to a generic docker agent (with Flow("<whatever>", storage = Docker()) as flow:). When I try to register, I get a healthcheck error: no module named bs4 / Flow uses module that is not importable. How do I get the flow to use bs4 on the docker agent?
k

Kevin Kho

01/31/2022, 8:16 PM
Is your storage definition just an empty
Docker()
? Maybe you can try adding Python dependencies as seen in the API to add the relevant Python packages?
There is an example here
t

Tony Liberato

01/31/2022, 8:56 PM
Thanks, Kevin. Yes, just and empty Docker() object. I'll try this.
That worked beautifully, Kevin. Thank you.
👍 1
k

Kevin Kho

01/31/2022, 9:05 PM
Nice!
t

Tony Liberato

02/03/2022, 8:39 PM
So one last question and then I think I have our system up and running: How do you start the agent and detach? I have a docker agent running on red hat, but I don't want the agent to stop if I close my putty session. Any suggestions?
k

Kevin Kho

02/03/2022, 8:40 PM
You need some other tool to do that. pm2 or tmux or something like that?
t

Tony Liberato

02/04/2022, 12:58 PM
Thank you, again, Kevin.
View count: 3