@Marvin how does Prefect compare to Temporal?

Hi, does anyone know whether I can integrate Prefect on-prem, as part of my product, without paying for a subscription?

Hi all, I have an authentication problem:

prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.

However when I run

prefect auth status

it says I'm connected and authenticated. Any idea what's wrong?

Hi guys, hope someone can help me with my current problem. I have two virtual machines. One is running the Prefect server and I try to deploy a flow from the other one. I have started the Prefect server successfully

ubuntu in 🌐 instance-20230102-1221 in ~ via 🅒 prefect took 5m57s 
❯ prefect config set PREFECT_SERVER_API_HOST=0.0.0.0
Set 'PREFECT_SERVER_API_HOST' to '0.0.0.0'.
Updated profile 'default'.

ubuntu in 🌐 instance-20230102-1221 in ~ via 🅒 prefect took 3s 
❯ prefect server  start

 ___ ___ ___ ___ ___ ___ _____ 
| _ \ _ \ __| __| __/ __|_   _| 
|  _/   / _|| _|| _| (__  | |  
|_| |_|_\___|_| |___\___| |_|  

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>

View the API reference documentation at <http://0.0.0.0:4200/docs>

Check out the dashboard at <http://0.0.0.0:4200>

I am able to connect to the Prefect UI in the browser, by replacing

0.0.0.0

with the ip address of the vm. On my second vm, i have done the following:

ubuntu in 🌐 instance-20221129-2229 in yfin-db/yfin_db on  main [!?] via 🐍 v3.11.0 via 🅒 main took 6m37s 
❯ prefect config set PREFECT_API_URL=<http://xxx.yyy.zzz.xxx:4200/api>
Set 'PREFECT_API_URL' to '<http://xxx.yyy.zzz.xxx:4200/api>'.
Updated profile 'default'.

ubuntu in 🌐 instance-20221129-2229 in yfin-db/yfin_db on  main [!?] via 🐍 v3.11.0 via 🅒 main took 6s 
❯ prefect deployment apply healthcheck-deployment.yaml 
Successfully loaded 'healthcheck'
Deployment 'healthcheck/healthcheck' successfully created with id '52b59be9-fe11-49d1-acad-3257f3fc758d'.
View Deployment in UI: <http://xxx.yyy.zzz.xxx:4200/deployments/deployment/52b59be9-fe11-49d1-acad-3257f3fc758d>

To execute flow runs from this deployment, start an agent that pulls work from the 'test' work queue:
$ prefect agent start -q 'test'

As you can see, I have changed the

PREFECT_API_URL

to the Prefect server running on my first vm. Than I have applied the deployment and Prefect tells me, that the deployment was successfully created. However, the deployment does not appear in the Prefect UI. Did I miss anything?

Hi, How can I programmatically retrieve deployment id from deployment name. Eg, retrieve id 'b86c7264-beae-4485-a68c-290286e92f00' by providing 'LiveLots-ETL/dev', is there Python API to achieve this?

Hi, in prefect 2.0 - is there a way to perform 'yields' - we have some (unfortunately) long running tasks where we stream some content, which takes a while (+30 min, very large zipped json files) - and I can't seem to figure out if I can yield partial result futures in my task incrementally and start processing them in down stream tasks. Currently, the work flow is: • Stream read big annoying file into small chunks, save them in S3 incrementally in a separate thread inside the task, and return list of all the chunked file URI's as single result (= single

prefectfuture[list[str]])

object) ◦ this is where I would like to yield • Run some down stream processing on each chunk ◦ Here we use future.result() to loop over and start each task - this works 100%, but if we could use yield in the first task, we could begin processing immediately Is something like this supported / going to be supported? I tried making it work with contextmanager and tasks, but ran into nothing but trouble

I'm trying to dynamically add tags to a flow by iterating through a list. Tried a few different things, but keep getting

TypeError: 'list' object does not support the context manager protocol

or

TypeError: 'generator' object does not support the context manager protocol

Here's one of my attempts. Any suggestions on how to do this?

from prefect import flow, task, tags
from prefect.context import TagsContext

def add_tags(tag):
    with tags(tag):
        yield

@flow()
def my_flow():
    # print(TagsContext.get().current_tags)
    print('inside flow')


tag_list = ['mytag1', 'mytag2']

with [add_tags(x) for x in tag_list]:
    my_flow()

I am running a flow via kubernetes job. Sometimes when there are not enough nodes available to run the job pods, it will take a sec to scale up a new node. During this time, prefect seems to mark the flow as crashed (since it was not scheduled after some time), but eventually the new node does come up and the flow is able to run fine. However, prefect refuses to run the flow since since the run has already been marked as terminated:

aborted by orchestrator: This run has already terminated.

Is there some way I can configure the internal timeout for waiting for the pod to be scheduled? Configuring retries does not seem to make a difference. Thanks!

"Couldn't connect to Prefect Server at *http://localhost:4200/graphql*" i get this when i try to run docker version in my local how to fix this anyone please help

I'm running into some memory limits with a flow using an ECSRun config on a LocalCluster DaskExecutor. The first task in the flow is to generate a list of file keys, and then I map over that list to process the files. I would have expected that when an individual Mapped Child task completes, any memory used in that task would have been released (aside from the return value). But it seems like that is not the case, as the process eventually runs out of memory after several Mapped Children finish. Is there something I can do in the configuration of the flow or task to mitigate this? As an aside, when I run this same flow using a FargateCluster, it will complete without issue. I'm not sure if that's because the FargateCluster I'm using just has more memory, or if there's something inherently different between the two different DaskExecutor classes. Thanks as always for any help you can provide.