Hi - What is the suggested

target

pattern for a

task

that is called twice in the same flow but without mapping? I currently am mucking around with this pattern below and thought of using tags to differentiate them.

duplicate_task_target = "{parameters[<A FLOW_PARAM>]}/{task_name}-{???}"

But that would mean keeping track of duplicate tasks ... which is burdensome when working on multiple flows that then get updated. Any other thoughts? Aside: I think this goes to param based targeting mentioned in other threads.

when using the docker container, is there a way to not jump into python when it is launched?

Hi - I'd like to confirm that using

task_run_id

as a target is a bug using 0.11.3

from prefect import task, Flow
from prefect.engine.results import LocalResult

lcl_res = LocalResult(dir="~/prefect_guide/results/{flow_name}")

@task(target="{task_name}/{task_run_id}",
)
def return_list():
    return [1, 2, 3]
@task(target="{task_name}/{map_index}.prefect")
def mapped_task(x):
    return x + 1
with Flow("blah", result=lcl_res) as flow:
    mapped_task.map(return_list)


st = flow.run()
flow.visualize(flow_state=st)

Does Docker storage attempt to authenticate with the docker registry? I'm getting a "no basic auth credentials" error from

flow.register()

Hi, I'm trying to work through flow registration with cloud using Docker storage

Hello! Does Prefect has notion of priority? Something like priority_weight in Airflow. So I could raise some flow runs.

Where can I submit small Cloud UI-related bugs like this? Clicking on

Cloud Hooks

here opens https://docs.prefect.io/orchestration/concepts/cloud-hooks.html while the correct URL is https://docs.prefect.io/orchestration/concepts/cloud_hooks.html (notice the

_

instead of

-

)

Do you know where this error comes from?

Hello everyone - I think I found two issues in

prefect install agent kubernetes

•

--namespace

only applies to the rbac in the generated manifest but not to the deployment - [i.e agents always gets deployed to default namespace] • Can’t specify a

--name

to the cli call - so the agent’s name always gets registered as

agent

- apparently this is done in

prefect agent start

- I stand corrected

Hi all, I've started using prefect and I think its a great api for managing workflows (I see it as a human friendly wrapper around dask orchestration). Thanks for excellent documentation and a super useful tool! I'm trying to determine how best to use prefect within an orchestration workflow design. The workflow is essentially three generic steps: 1. spinning up various services (various web servers and processes) 2. manage these services: state, variables/properties, tasks (prefect seems to fit this, although I may need to find a good way to store state/data) 3. detect failure/errors in services and trigger redundancy/recovery tasks I am torn on how to use prefect with 1 and 3. I don't think using prefect alone is the best solution, but I also think prefect(dask scheduler/executor) is a good solution for starting services (1) and handling triggers (3). However I'm not sure if prefect is ideal for active monitoring. Is there a prefect usecase for monitoring log files/services, or if there is a better tool for this job? I didn't see any mention of monitoring within the prefect core. I'm not talking about monitoring specific prefect tasks which is obvious that prefect should monitor its own tasks, more so on monitoring errors/issues in services. If prefect isn't ideal for active monitoring of services, I envision using something like prometheus to watch data/logs from the services and trigger redundancy/recovery tasks in prefect if prometheus alerts or detects failures/issues. Are there any ideas for a better integration of prefect into this workflow?

Hey everyone got a question. I am currently working on building a flow via the

Docker

storage class. Right now my flow imports env vars from a

config.py

, found in the same directory with the file that holds my flow, which uses

os.environ

. I would like to set those environment variables at run time but when, the healtchecks run and deserialize my flow in the docker build step, it is trying to evaluate those environment variables. This causes an error since I don't have defaults set. Does anyone have any ideas of have to handle this without ignoring healthchecks and putting environment variables into the Dockerfile that gets generated?

When using a KubernetesExecutor, is it possible to have dynamic resource allocation? We can currently use the

job_spec

file to specify the K8 resources. But we have a use case where, in most situations the volume of work expected from our Flow fits within the specified resources, occasionally an input Parameter to the Flow will be such that it requires more resources. We're getting by this today by setting the resources to the maximum value, but (in theory) that restricts the number of concurrent Flows we can execute in the same environment. Curious if there's an ability to adjust the memory resources required in the

job_spec

based on the value of an input Parameter to the FlowRun?

Morning all! Curiosity question, anyone tried cross cloud platform usage? I.e. host Prefect server and agent on AWS and run a flow on Digital Ocean?

Hey 🙂 I am using a reverse proxy for accessing the UI, so my UI is accessible at

<https://domain.local/prefect>

. Is there a setting to tell the UI to load the CSS from

<https://domain.local/prefect/css>

instead of trying to get it from

<https://domain.local/css>

?

https://news.ycombinator.com/item?id=23349507 a thread rn that looks like a nice place to promote Prefect

Hi - I have two flows that I am trying to merge into one using

update

. They flows are located in separate modules and have some duplicate parameters. Unfortunately the example pattern for joining flows with duplicate parameters only compiles if all the duplicated parameters are removed from their respective modules. Which makes multiple flow development difficult. Is there another pattern or solution where flows can be developed independently and then merged? --thanks https://stackoverflow.com/questions/60679595/how-does-one-update-a-prefect-flow-with-duplicate-parameters .

I have two more questions running Prefect locally: • Is it possible to edit a schedule from the UI? • Is it possible to give tasks a name when they are called (not when the task class is instantiated) so I can distinguish them in the UI?

Hello Prefect community, I’m using a K8s agent with a Dask K8s environment and I’m finding about 1 minute pass by when a run state goes from “Scheduled” to “Running.” Are there any ways to can speed up scheduling? Maybe by configuring the agent?

When two Flow runs share same name, it appears that getting logs via prefect CLI shows only the logs from more recent run, is that right? Is there a way to access the logs from earlier run with same name via CLI?

Hi All, Is there a way to configure 'Flow Run name' for a flow that's set to run on a schedule ?

Is it possible to trigger a flow run after a different flow has finished?

Hi Team, how do I stop the prefect docker images

prefecthq/ui:0.11.2

prefecthq/apollo:0.11.2

etc from starting on boot. I.e uninstall?

This message was deleted.

Dask Scheduler: I’m using a

LocalDaskScheduler

with the default

scheduler=threads

but it only runs one task at a time (mapped tasks). Am I missing some extra argument that allows tasks to run concurrently?

Hi everyone. I am trying to get a better understanding of mapped tasks and memory implications/best practices when using a large number of them. I have a pipeline that 1) queries a DB for a list of app IDs. It usually gets about 25k 2) Calls an API for each app N times where N is currently 14 different countries. The json response is not too big for each app. A single dictionary with several columns. 3) combines and stores the output of all tasks in s3 This is currently implemented in Airflow with one branch per country. Each branch queries the API repeatedly for each app and stores the results in a single file for each country. If the task fails, all apps need to be reprocessed. What I am wondering is: * If I create a mapped task that gets the list of apps + countries, does it create all the 350k (14 * 25k) child tasks in memory at once and they are put in some sort of queue or are they lazily created? * I suppose that if I did nothing special regarding caching the results to an external system like s3, it would hold all the data in memory until it gets to the reducer task that dumps the output to a file. This may require a lot of memory because the reducer won't start until all children finish. Correct? * Would this be alleviated if I use caching to s3? Would the memory be released once each task results are persisted to s3? * Each child task output would be pretty small and it seems that having that may s3 files with a little data in each is not great. Would you recommend that instead of having the child task process a single app, it processes a small batch of apps say 50? * I suppose there is no garbage collection on persisted results. Is the recommendation to use s3 life cycle rules to clear old task outputs? Thanks!

Hey community! I’m having a problem with

LocalDaskExecutor

. I keep getting the following error messages, which are probably related:

Error message: can't start new thread

Error message: 'DummyProcess' object has no attribute 'terminate'

BlockingIOError: [Errno 11] Resource temporarily unavailable

Quick question - docs (https://docs.prefect.io/api/latest/environments/storage.html#docker) say the Docker parameter:

secrets(List[str], optional)

a list of Prefect Secrets which will be used to populate 

prefect.context

 for each flow run.

Used primarily for providing authentication credentials.

It's not clear how I can access these secrets inside my flow. I've tried, exploring the

prefect.context

object, but can't seem to find anything. Any examples or guidance?

Hi all. I'm new to Prefect (and development). I had a doubt. I understand that one can use prefect to set up flows that run tasks in a particular order among other things. Does it also implement queueing? So far in my projects I've only used redis-celery to run big tasks in a queue. PS, I've mostly worked in web dev.

Core UI - how do I stop a running flow? I couldn’t find any reference to it, and the only way for me to do this right now is to hard kill the agent itself. Any way of doing so from the UI / CLI?

Hi all, is there a recommended approach for objects that cannot be pickled? For example in a machine learning pipeline where you need to do some ETL. Some of the steps require tools that have objects that cannot be pickled so I imagine I might put all those items in one step (SFrames, Spark, etc). Then pipe the results to a CSV file, and the next step we can train a model using the file location of the CSV file, but then I may want one more step to create metrics and send the metrics to MLFlow, but it may be possible that the model cannot be pickled easily, is prefect the wrong tool for this job? But I also would like a cronjob and a UI to look at the results. It seems that I may need to engineer the problem around Prefect vs a single script and Linux crontab. Any suggestions?