Hey all, please tell me how to set up the task launch sequence. For example at me one task deletes the data, the second task interposes the data. I need to run the delete first, and after insert data

hi all, I'm on a self hosted prefect server 15.11 running on AKS using the KubernetesRun() config, trying to create an Dask-cluster to use with the DaskExecutor. My flow works fine with a local cluster, however, as soon as I try to use an external cluster (using KubeCluster or Coiled) the flow stops with a message that "terminal tasks are incomplete" (see image). An hour or so later, the entire process failed with a connection-failed to my graphql-adress. Does this sound familiar to anyone?

Why would I get

The secret KUBERNETES_API_KEY was not found

if I’m running the prefect agent inside the cluster? According to this doc it will attempt an in cluster connection but my hello world task seems to keep failing 😭

What's the best way to replicate the

max_retries

feature but for an entire flow? I.e. if the flow fails for some reason, retry it 5 minutes later.

I have a task function which gets a tuple as input. The task_name for this task is to be derived from another function named _task_name_from_tup_ based on the tuple which it gets: example

@task(task_run_name="{task_name_from_tup(details)}", max_retries=3, retry_delay=timedelta(minutes=1))  
def processing(details):
	//some code//

I haven't dug into the API reference docs, but just looking at the high level docs for Flow-of-Flows, is it possible to have a flow-of-flows where FlowA is running on a schedule, and then FlowB only runs after FlowA, whenever that happens to be? edit: and these are in two different environments and two different agents so I can't make them one flow

Is it possible for a prefect flow to spin up a dynamic number of other flows based on the output of a task? I see in these docs some examples of spinning up a static number of flows but I would love to have the number of children flows vary from run to run.

Hey I’m passing a fairly large object from one task and then map a following task over an iterable with that big object being unmapped.

Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and 
keep data on workers

    future = client.submit(func, big_data)    # bad

    big_future = client.scatter(big_data)     # good
    future = client.submit(func, big_future)  # good

I was wondering, what would be the prefect pattern here to scatter the object ahead of time?

Is it possible to have more than 1 license admin per license?

I try to upgrade prefect but i faced this error. Pulling postgres ... done Pulling hasura   ... download complete Pulling graphql  ... download complete Pulling towel    ... download complete Pulling apollo   ... download complete Pulling ui       ... download complete ERROR: for graphql  failed to register layer: symlink ../753a933895769d16a9f50e547a17525d552c3f71551ee123b7fc73ca9fefbc69/diff /var/lib/docker/overlay2/l/B5VLSRNEDH3NABDXEFXCIRNLR2: no such file or directory ERROR: for towel  failed to register layer: symlink ../753a933895769d16a9f50e547a17525d552c3f71551ee123b7fc73ca9fefbc69/diff /var/lib/docker/overlay2/l/B5VLSRNEDH3NABDXEFXCIRNLR2: no such file or directory ERROR: for hasura  failed to register layer: symlink ../9e83ca61c81268bc9fb45be76dec1d0897716ce0633ce79aabbb49c98c9802bc/diff /var/lib/docker/overlay2/l/DQRRSWXFNYBRCBO2G4WTIKOQSO: no such file or directory ERROR: for apollo  failed to register layer: symlink ../0027aacacd89f06a1944e4198231f069a200aada88a290932f113a0cb9ab04b1/diff /var/lib/docker/overlay2/l/DAHDL3BQWHWRBVM7IV5DDI36H3: no such file or directory ERROR: for ui  failed to register layer: symlink ../ebf2b0dd143761035387a4197bd103c4238b4d3fe664cf622ea2b3126b2eaba7/diff /var/lib/docker/overlay2/l/AHLFWYQ3FLDKVRIL7PPFZM3XXZ: no such file or directory ERROR: failed to register layer: symlink ../ebf2b0dd143761035387a4197bd103c4238b4d3fe664cf622ea2b3126b2eaba7/diff /var/lib/docker/overlay2/l/AHLFWYQ3FLDKVRIL7PPFZM3XXZ: no such file or directory Exception caught; killing services (press ctrl-C to force) Removing network prefect-server WARNING: Network prefect-server not found. How can I fix it? Thank you.

Hey, Has there been any significant changes in Git storage between docker containers prefecthq/prefect:0.15.7-python3.9 and prefecthq/prefect:latest-python3.9? My flow runs fine under 0.15.7 but fails under latest with "No git repository was found at https://@git.*************" Update: After several hours of debugging I found that the issue was introduced in dulwich version 0.20.29. Downgrading dulwich to 0.20.28 in the docker image fixes it.

Hi prefects. I’m trying to execute a RunNamespacedJob task in my kubernetes setup but I’m running into this error

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "jobs.batch \"dummy\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"jobs/status\" in API group \"batch\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "name": "dummy",
    "group": "batch",
    "kind": "jobs"
  },
  "code": 403
}

For context, I’ve applied the yaml from

prefect agent kubernetes install --rbac

so all the permissions should work in theory. I'm stuck at what could be wrong

How to avoid this unwated task in logs and taskruns (in GUI as well):

[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'Tuple': Finished task run for task with final state: 'Success'
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'Tuple': Finished task run for task with final state: 'Success'
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'Tuple': Finished task run for task with final state: 'Success'
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'Tuple': Finished task run for task with final state: 'Success'
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'Tuple': Finished task run for task with final state: 'Success'
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'Tuple': Finished task run for task with final state: 'Success'
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'Tuple': Finished task run for task with final state: 'Success'
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'Tuple': Finished task run for task with final state: 'Success'
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'List': Starting task run...
[2022-01-19 14:45:49+0530] INFO - prefect.TaskRunner | Task 'List': Starting task run...
[2022-01-19 14:45:50+0530] INFO - prefect.TaskRunner | Task 'List': Starting task run...
[2022-01-19 14:45:50+0530] INFO - prefect.TaskRunner | Task 'List': Starting task run...

Hi, A question around best practices when setting up flows... In one scenario I have a flow which will typically run several steps. Some of those steps are used by other processes. To keep the Flows 'DRY' I have created three separate flows so I can call them in isolation. I have then created a parent flow which can run all three child flows. This works fine with wait_for_flow_run/set_upstream and allows the parent flow to give a view of the child flows and their progress/logs. I wonder if this is the best approach as despite it functioning well I lose observability. I've enabled stream_logs so the parent Flow sees all of the child logging, but the names aren't very intuitive and the cloud doesn't easily allow you to navigate to the child flows. See attached screenshot. You end up with a long list of 'create_flow_run'/'wait_for_flow_run' instead of the friendly task names. I can rename the create_flow_run however the new name only shows on the child flow instance, not on the parent. Because of this it feels I am going against the grain with Prefect. So am I barking up the wrong tree by keeping Flows DRY - should I instead create multiple flows and only worry about keeping the Tasks DRY? (they already are so this would be easy to change)

Hey! What is the most lightest/easiest way to run Prefect agent remotely? I have a script that runs within 20 seconds, doesn’t required too much computation power or storage. And it’s a personal project that have low stake. My current option is renting servers on digital ocean ($5/mo), but I wonder if there’s better option for that (maybe heroku dyno? haven’t explore it yet)

Hi i am trying to register a flow from agent . On the GUI the agent is connected but when i try to register the flow I get this error:

File "/home/infra/prefect_server/lib/python3.8/site-packages/prefect/client/client.py", line 603, in _send_request
    response = <http://session.post|session.post>(
  File "/home/infra/prefect_server/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/infra/prefect_server/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3be9b9e700>: Failed to establish a new connection: [Errno 111] Connection refused'))

Example: My GUI server is hosted on Machine A and one agent is Machine B, the code present on Machine B is to be run and registered on Machine B but should show its task on the Machine A GUI.

[solved] When I create a flow that runs a namespaced job, prefect actually creates 2 jobs.

dummy

runs on the

aks-spot

node but

prefect-job

runs on the

aks-system

node (and I don’t want it running on the system node pool). Is there a way to configure tolerations and affinities for the

prefect-job

pod?

Hello community ! Is it possible to limit/prevent some tasks to run at the same time ? I've read the doc on concurrency limits, but it doesn't seem to handle my use case. For instance if y a have 2 tasks A and B, I'd like to either prevent them to run at the same time or to limit the number of A and B that can run simultaneously.

Hi all, my team and I are developing a web-app to load and process dataframes, we're willing to use prefect orion instead of jupyter kernls. Is it possible to separate contexts/sessions using prefect? Example: User A creates a variable "var_a" and User B creates a variable "var_b". User A shouldn't be able to access to "var_b". Both variables should be accessible trough several flows from the same user until their session on our app is closed. Am I missing something?

I’m curious as to whether it is possible to update the state of a running Flow according the outcome of a Task. That is, if at some point during the execution of a Task a condition is met, can I update the state of the Flow/Task to Failure?

Hey community! I’m trying to set up Prefect with GreatExpectations, using the official task. I’m trying to run a validation on an in-memory Dataframe (result of previous task) but have trouble to set it up correctly. I’m trying to use v3 API of GE, set up the expectation suite, checkpoint according to the guide (until _context.run_checkpoint)_ but struggle to pass said dataframe. Would anyone be able to offer some guidance?

Is there a way just to execute a git pull command at the start of a run, assuming the local machine has authentication already set up? My company uses Azure Devops, and as discussed in https://github.com/PrefectHQ/prefect/issues/4850, I set up a storage to pull from this via a hard-coded URL. Unfortunately the personal access token (PAT) in that URL keeps expiring in only 30 days (think I'm hitting some corporate policy limit). The standard Azure approach is to use the Git authentication manager https://docs.microsoft.com/en-us/azure/devops/repos/git/set-up-credential-managers?view=azure-devops, not SSH. I have that set up on the machines already, and a git pull would use that more reliably. Possible?

Hi everyone, I’m new to prefect and I ran into an issue today. I have the kubernetes agent running in a cluster, the logs show that it is registered and waiting for flow runs. However, this agent does not appear on Prefect Cloud. Any reason why this might be?

Hi here the tasks on Left branch are running fine but the tasks on the Right Branch are stuck in Mapped state. Both have the same functions called and mapped to run a task.

Hi, I am running a flow in cloud which has two paralell task but only one task is running at a time, What might be the reason?

Hi prefectionists! I’m wanting to use a link to a flow run that I’m running in prefect cloud as metadata of sorts, so I can trace long-running queries back to the flow run easily. Are there env vars populated automatically (e.g. FLOW_RUN_ID, FLOW_ID, etc) that would allow me to construct this link, or more generally, is there any way for a task to automatically get info about it’s own identity?

Is anyone seeing issues with prefect cloud rn? I have a task run that says

Trigger was "all_successful" but some of the upstream tasks failed.

but all the upstream tasks were successful. Am I missing something?

it is possible for a task to get its own task run id? say if I wanted each task to send a notification with its task run id for example, would this be possible? and do task run ids persist across retries of the same flow run? (if a flow fails at a task and i restart the flow, will that task have the same task run id as before?)