prefect agent kubernetes install -k API_KEY | kubectl apply --namespace=my-namespace -f -

this command successfully creates the deployment and referrs to default prefect image so the question is how can i use my code with it?

hi, does anyone know how to get the flow running status inside a task? I need make some logics for different status like cancelled, running, etc.

How can I handle "internal error" in Prefect cloud?

Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
    task_run_info = self.client.get_task_run_info(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1798, in get_task_run_info
    result = self.graphql(mutation)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]

Potentially related to https://prefect-community.slack.com/archives/CL09KU1K7/p1634766491212000 on a long-running flow, we are no longer able to receive the logs for that flow. Any time I use

prefect get logs --id uuid_of_that_flow

, it gets killed with no logs retrieved. Is there another place to go look for logs?

I don’t know if this is a bug or not - I set a task concurrency to 1 with one label defined and testing, however, it keeps hanging at state:

Queued

with just one that task running. Only if I increase the limit to 2 it will start running. It looks a bug to me:

Hi! I'm trying to figure out what is the easiest way to restart the flow that crashed (but hasn't started)? For a context, we are facing concurrency issue

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.

and I'm aware there is open github issue (https://github.com/PrefectHQ/prefect/issues/4402). We did adjust params as recommended but it still happens. I want to be able to restart the flow but cannot use task retry feature as no tasks have been started (it is the only log line I see and there is no start time for the flow ):

Hey guys, i'm running prefect server on an azure vm. 2 issues, first is how I set the --expose flag in the docker-compose.yaml and i'm also struggling to connect to the ui remotely. The ports have been exposed and ip's have been whitelisted to our internal org only

Hi folks, I am looking at interfacing with prefect’s graphql API using Typescript and was wondering if the schema for Prefect’s API is available anywhere? Ideally I’d like to autogenerate typescript types for the queries, mutations etc. I’ve tried using

graphql-code-generator

but that hasn’t worked so far

What's the recommended way to add monitoring and observability to a prefect flow? I mean: • the ability to see input-output for tasks (at least for a while after running them; just not Results) • the ability to see timings for each task • any structured logging from inside tasks (e.g. especially important segment variables / which app / tenant is it running for) • integrating with Prometheus ?

Hi everyone, I am trying to define a new flow. In this flow I have a task that I map on thousands of values. The problem is that some children tasks fail because there are too many open connections simultaneously. When I run my flow on 1000 arguments it works fine but on 8000, some child tasks are stuck in a running state. Is there a way to do the mapping consecutively by chunks of 1000 arguments ? Basically, If I give a parameter list of 5000 arguments to my flow, I would like the flow to run the task five times consecutively. Each time the task would me mapped over 1000 arguments. I don’t have to use the results of the tasks if that would be an issue. The flow would run on a localdaskexecutor with 15 proceseses and use a prefect server as an endpoint/host. Thank you so much for your help guys

Can I access results of a flow run created with create_flow_run. I normally start child runs with

f_run = flow.run

method and access the results with

f_run.results[df].result

.

Hi. Sorry to ask such a basic question, but is it possible to delete a flow from the prefect server? I would like to do it programmatically, but I can't find any reference to it the documentation. I also can't see this option in the prefect cloud webUI.

Hi, I was just trying the prefect orion tutorial (https://orion-docs.prefect.io/tutorials/first-steps/) but it failed 😕 It failed on this line

state = my_favorite_function()

Hi everyone, when creating an

EmailTask

what is the

EMAIL_USERNAME

and

EMAIL_PASSWORD

used for? Are the notifications shipped from this email? Is this required, or is it possible to use the default notifications@prefect.io email address shown in the class definition?

Hi, I’m trying to ping the graphql API with python and javascript and running into some parsing problems. How would I convert the following to python using the requests package?

query {
  flow_run(where: {name: {_eq: "woodoo-leopard"}}) {
    id
    state
    start_time
  }
}

Hi, Is there an equivalent of Airflow’s `TaskGroup`’s in Prefect? Or is there any other way to build larger flows (100+ tasks) while making it easy to navigate the flow in the UI? I think I’ve seen something similar before, but I can’t find any references… Thanks

Hi, I created a flow of flows. Is there a way to call subflows with StartFlowRun depending on values from a input Parameter. For example input parameter has values [a, b, c] so I need to call StartFlowRun for flow “a”, “b” and “c”

What are our options for external monitoring or webhook integrations for total system uptime for Prefect? Looking for the ability to alert for outages.

I have many tasks distributed across files, and one slack_notifier handler that they all import and utilize. What is a good pattern for disabling/using a different handler for local dev. I cannot run flows locally unless i change the handler because i get a

ValueError: Local Secret "SLACK_WEBHOOK_URL" was not found.

Hey everyone, join us for a fireside chat with our CEO Jeremiah and CTO Chris about Orion. They will be answering community questions. Direct link

hi, is there a way to kill task in flow context? I know how to start a task in there, fairly easy. But not to kill a started one.

LiveStream starts now! 🦜

Hi Team, Is there a way in prefect.core library/api to de-register or delete stale flows?

Howdy Prefect team, I found you via Open MLOps and keep seeing you pop up in community discussion. Is there somewhere I can reference for how Prefect executes my code? Like a sequence diagram or something to help me wrap my head around it? Enjoyed the fireside, Orion sounds awesome!

So I'm refactoring a Flow into a Flow of Flows. It runs a bunch of Papermill notebooks. They have to be run in a certain order - but in some cases, a dependency isn't relevant, so we skip it. This currently takes the form of a bunch of Tasks along the lines of

if notebook_name in list_of_notebooks:
   run_notebook(notebook_name)
else:
    print("skipped")

How do I do that with the

StartFlowRun

task? Should I subclass it and replace the existing

.run()

method?

Hi, I’m running the same task multiple times in a flow using some Parameters that make different stuff happen each time, using the

map

method:

process_tasks.map(flow_description, start_task_id)

My problem is that if one of the mapped tasks fails, the other ones still subsequently run regardless, and I’d like for the whole flow to be cancelled programatically/automatically when that happens. I think the CancelFlowRun API method should be able to do this if I invoke it just before the failed task enters the failed state (via a raise exception). I’m struggling to figure out how to call this thing though, and was hoping someone might be able to help, please. If I try the following:

prefect.tasks.prefect.flow_run_cancel.CancelFlowRun.run(context.get("flow_run_id"))

My python engine says it can’t find the

flow_run_cancel

method in any imported Prefect module, and I’ve already imported

task, Flow, Parameter, context, Task, tasks

(for good measure). I’m clearly doing something very wrong and don’t know enough to figure out what it is. Would be very grateful for any help or suggestions to put me out of my misery. Thanks in advance!

Hi, what's a the recommended process of transitioning off Prefect cloud into something self-hosted that can run flows? E.g. is it recommended to set up the server software yourself? It's very important that we can run our flows.

Hey everyone, I am trying to run a simple flow with a

RunNamespacedJob

task over Kubernetes. I am using the following code:

storage = S3(bucket="gavriel-test", stored_as_script=True)
kubernetes_run_conf = KubernetesRun(
    env={
        "AWS_ACCESS_KEY_ID": AWS_ACCESS_KEY_ID,
        "AWS_SECRET_ACCESS_KEY": AWS_SECRET_ACCESS_KEY
    },
    labels=["prefect-poc-k8s-agent"]
)
body = {
    'apiVersion': 'batch/v1',
    'kind': 'Job',
    'metadata': {'name': 'echo'},
    'spec':
        {
            'template':
                {
                    'spec': {
                        'containers': [
                            {
                                'name': 'echo',
                                'image': 'alpine:3.7',
                                'command': ['sh -c  "echo Hello!!!"; sleep 10']
                            }
                        ]
                    }
                },
            'backoffLimit': 4
        }
}
with Flow("kubernetes-CreateNamespacedJob", run_config=kubernetes_run_conf, storage=storage) as flow:
    job = RunNamespacedJob(body=body, namespace="prefect", delete_job_after_completion=False)

But I keep getting the following error:

Error during execution of task: MaxRetryError("HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /apis/batch/v1/namespaces/prefect/jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8c8adeb650>: Failed to establish a new connection: [Errno 111] Connection refused'))")

Do you have any idea how to resolve this?

This issue, graphviz.backend.ExecutableNotFound: failed to execute 'dot', make sure the Graphviz executables are on your systems' PATH is occuring through my venv. It would seem to me that the executable would come prepackaged with prefect['viz'] or prefect['dev']. I'm using debian/bash. Anyone else come across this issue?

hi, how can I start two tasks simultaneously? Now I see it’s running at a single thread, even two tasks have no dependencies, they cannot run at the same time. Is this even impossible?