Hi @Anna Geller @Kevin Kho, I have scheduled a flow programatically, but its not geting deleted when Im trying to delete that schedule using mutation

delete_flow_group_schedule

. Is there any way to delete a 'programatically created schedule' by using graphql mutation or client through code.

hi! our team is currently doing a security audit on prefect right now. what’s the easiest way to get an example of the the metadata that’s being sent to prefect cloud? i.e. how can i view the API calls to prefect cloud that we’re making?

Hello, My team is trying to get the k8s prefect agent to run in k8s. Whenever we spin up the agent it fails on

registering agent

. I took a look through the code and it seems that on spin up the agent attempts a query in the db to see if it already exists. If it doesn’t then it registers itself by adding a new record of itself to the db. We are also not currently using the prefect cloud server but rather local instances of apollo, graphql, hasura, etc running in our k8s cluster Here is the stack trace

[2022-01-24 18:15:41,310] INFO - agent | Registering agent...
Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/prefect/cli/agent.py", line 331, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/prefect/cli/agent.py", line 140, in start_agent
    agent.start()
  File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 189, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 910, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 858, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 2107, in register_agent
    agent_config_id=agent_config_id,
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 570, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': '"\'Box\' object has no attribute \'data\'"', 'locations': [{'line': 2, 'column': 5}], 'path': ['register_agent'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '"\'Box\' object has no attribute \'data\'"'}}}]

I can also provide the yaml for the k8s

Hello folks, I recently upgraded our prefect server to 2022.01.12. I also upgrade our prefect version to 0.15.12. While trying to register our flow, I get this error:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "flow_registration.py", line 136, in <module>
    register_flows(args.local)
  File "flow_registration.py", line 128, in register_flows
    idempotency_key=flow.serialized_hash())
  File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1734, in register
    idempotency_key=idempotency_key,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1213, in register
    variables=dict(input=inputs),
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 555, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 459, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 740, in _request
    session=session, method=method, url=url, params=params, headers=headers
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 661, in _send_request
    raise ClientError(f"{exc}\n{graphql_msg}") from exc
prefect.exceptions.ClientError: 400 Client Error: Bad Request for url: <http://prefect-server-apollo.emtrails:4200/graphql>

The following error messages were provided by the GraphQL server:

    GRAPHQL_VALIDATION_FAILED: Unknown type "register_tasks_input". Did you mean
        "register_agent_input", "delete_agent_input", "delete_flow_input",
        "delete_tenant_input", or "log_insert_input"?
    GRAPHQL_VALIDATION_FAILED: Cannot query field "register_tasks" on type
        "Mutation". Did you mean "register_agent", "delete_task", "insert_task", or
        "update_task"?

The GraphQL query was:

    mutation($input: register_tasks_input!) {
            register_tasks(input: $input) {
                success
        }
    }

Would anyone have encountered a similar issue?

Hello! I've just deployed Prefect Server into Kubernetes on AWS using official helm chart and I'm trying to make my first hello-world flow to work. I am using GitLab storage and I'm wondering what is the correct (and secure) way to pass

GITLAB_ACCESS_TOKEN

?

Hello! I am trying to create Jira issues using the JiraTask and got this error (full text in thread):

jira.exceptions.JIRAError: JiraError HTTP 400 url: <https://our_jira_url/rest/api/2/issue>
	text: expected Object containing a 'name' property
	
	response text = {"errorMessages":[],"errors":{"assignee":"expected Object containing a 'name' property"}}

does prefect have anything built-in for datadog metrics? for example, if we wanted to track job success rate, job duration, etc. in datadog in addition to the cloud ui.

I have encountered a curious situation, where SOME mapped tasks appear to be "Zombies" -- they run for tens of minutes on end, when they average job is <1min; the UI shows the task status as "Running", with the run duration increasing. Of 40 Dask worker vCPUs, as Flow runtime increases, maybe 15 or so at any one point, will be these "zombies". What's strange is that it is possible to "jolt" these tasks out of this zombified Running state, by going into the UI for the Task. Once this happens, without fail, the task status will resolve to "Success" after a minute or so; and the Dask worker running it is freed up for a new task. (Usually, I find opening the Logs tab, the quickest way to do this.) It's clear that what we are doing, is indeed "jolting" the task awake, forcing it to finish its state transitions; this is observable in the increased workflow throughput, due to the now-freed vCPU Dask worker. So, I don't think this is simply a lag in the UI. Is this a known issue?

Are there any examples of running local unit tests to verify your tasks are working? I'd like to have some pre-deployment checks in my GitHub actions.

Prefect beginner here. Can I set prefect server to run in the background, like

docker-compose up -d

would behave?

What is the most straightforward example of creating your first simple cloud agent that can run a cloud registered flow without any help from my laptop?

I'm trying to deploy Prefect Server/UI to Azure Web App Service using Docker with docker compose and I'm trying to use the docker-compose.yml from https://github.com/flavienbwk/prefect-docker-compose/blob/main/server/docker-compose.yml. Has anyone successfully done this before? The UI is loading for me but I keep getting a "Couldn't connect to Prefect Server at *https://localhost:4200/graphql*" error.

Hello! I'm running prefect server on GKE using the helm chart. I have an ingress for the UI secured with oauth and a load balancer for Apollo. I'm able to use the graphql endpoint from Apollo and the CLI. However, when it comes time to connect to the graphql endpoint in the UI I'm getting an error message:

Oops! It looks like something went wrong when trying to connect; make sure Prefect Server is running at the URL above and try again.

Any advice on how to debug this? GKE already creates a firewall rule to allow ingress over tcp:4200. Maybe something else weird with the firewall? I previously had Apollo set up as in ingress, but I switched back to a load balancer in case that would simplify things.

Hi everyone, I trying to figure out how to register the prefect flows with hosted prefect server instead local. Is there any documentation for that?

Hi guys, I’ve just redeployed prefect server to AKS via helm chart and having trouble connecting to graphql from ui. How can I troubleshoot this?

Are there any examples of a Dockerfile for Prefect agents successfully deployed via Kubernetes?

Hi folks, Wondering if it's possible to launch gpu-enabled containers with a Docker Agent? Normally you would flag it when running a container like volumes, but I'm not seeing an obvious flag in the docs unless I've missed something.

Thanks to all the help in this channel, yesterday I was able to launch my first cloud deployment with a Kubernetes agent running from a Docker source. I'm very happy with the progress but here's one thing I noticed: My tasks aren't running in parallel anymore. When I was in local mode, I had achieved that with a LocalDaskExecutor, which remains in my code after the switch to cloud hosting and execution. What is the simplest way to get my Kubernetes agent and Docker source doing Dask again? Is it as simple as replacing LocalDaskExecutor with DaskExecutor? Do I need to install some extra dependencies in my agent Dockerfile? Include more requirements in my Source's python_dependencies? Something else?

Hi guys, I'm working on my first prefect flow and keep running into this error below. I can't seem to find where it is referencing the missing argument, however, since it doesn't give me a specific line number

TypeError: missing a required argument: 'target_date'

Hey everyone! Ive set up an agent and my flow on prefect cloud. When I run my flow, I get:

[28 January 2022 10:46am]: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/Users/jackchang/'")

I went through some threads and added a gitlab repo as the storage and added that to my flow. Any ideas on what I should try next? 🙂

Hey guys - I’m new to Prefect but I’m considering using it at my startup, however, I have a question - how easy would it be to integrate Prefect with a web app for end user? Our users needs to be able to initiate a data processing job from our web app (not the Prefect UI) and also change some parameters of the job (what data to process). Does Prefect come with a REST API endpoint from which jobs can be ran and job parameters changed? Thanks! Simon

Good morning 🙂 I'm quite new in prefect and I would like an advice for a problem that I have. In the example below all numbers (the list) have to be processed before saving the number (save_number()). I would like to know how I can process one number per time. For the first number in the list i execute the whole flow, for the second number i execute the whole flow, and so on ... This is an simple example, I'm thinking more if i have to process a lot of data. Many Thanks

from prefect import Flow, task
import prefect

import time

from prefect.executors import LocalDaskExecutor


logger = prefect.context.get("logger")


@task
def add_ten(x):
    time.sleep(1)
    return x + 10


@task
def save_number(y):
    <http://logger.info|logger.info>(f"Saving number: {y}")


with Flow("reduce", executor=LocalDaskExecutor()) as flow:
    mapped_result = add_ten.map([1, 2, 3, 4, 5, 6, 7, 8, 9, 9])
    mapped_result_2 = add_ten.map(mapped_result)
    reduced_result = save_number(mapped_result_2)

Hi guys, I have been using Prefect but stuck at an issue. I am using ECSAgent and sometimes I get an error that says "An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.". I am running around 80 flows that all run every minute. I am planning to run more flows, maybe in 100s and all of them running per minute. Am I using Prefect incorrectly and can it handle this scale? I checked out this link from the docs - https://docs.prefect.io/orchestration/agents/ecs.html#throttling-errors-on-flow-submission. I am already running the ECS agent with the specified options for retry mode but these errors still happen. Is there any way to fix this? I don't really need Prefect to create new task definitions on every flow run, can I disable that behaviour? I am also using a custom task definition, could it be because of that?

How can i use an ecs fargate cluster and run tasks in parallel? Like using fargate cluster for dask.

Is there an established pattern for including dev/prod variations in a flow file. Basically, in dev I'd like to have the default local storage on my laptop and in prod I'd like to do a customized Docker storage with Google Artifact Registry. The hacky section of my brain is imagining some kind of if/else clause in my flow file that keys off an environment variable. But I'm not sure what that variable would be, and I wonder if there's a sturdier method for achieving this solution that I'm simply ignorant of.

Hello there ! We're starting to see our Postgre database get a bit bigger than we would like it to be and were wondering how we could limit its size ? I suppose we could cleanup old flow runs and flow versions after a while but I have no idea how we could do that. Are there any good practices on how to do this ? Cheers !

My next challenge trying to configure my Prefect

LocalDaskExecutor

to run mapped tasks in parallel inside a single GKE k8s container with what, if I understand the jargon correctly, is known as vertical scaling. Are there any resources that show the optimal way to do that?

Hello everyone! I am trying to set up an ecs agent on fargate, but I don't have a "default" vpc set up, so I need to specify the network config, as I was getting this error:

ERROR - agent | Failed to infer default networkConfiguration, please explicitly configure using

--run-task-kwargs`` . However when I add the yaml path to the command, I start to receive this error:

Error: No such option: --run-task-kwargs s3://*****/options.yaml

Any thoughts?

Hi everyone. Is there an equivalent for celery metrics exporter in prefect server? https://github.com/danihodovic/celery-exporter

Good afternoon. I have been making great progress on our inital prefect workflow model. However, I ran into a snag I have no idea how to fix. I have a typical download/unzip/save to s3 workflow that I am trying to send to a generic docker agent (with Flow("<whatever>", storage = Docker()) as flow:). When I try to register, I get a healthcheck error: no module named bs4 / Flow uses module that is not importable. How do I get the flow to use bs4 on the docker agent?