itay livni
05/27/2020, 5:04 PMtarget
pattern for a task
that is called twice in the same flow but without mapping?
I currently am mucking around with this pattern below and thought of using tags to differentiate them.
duplicate_task_target = "{parameters[<A FLOW_PARAM>]}/{task_name}-{???}"
But that would mean keeping track of duplicate tasks ... which is burdensome when working on multiple flows that then get updated. Any other thoughts?
Aside: I think this goes to param based targeting mentioned in other threads.Alex Welch
05/27/2020, 5:08 PMitay livni
05/27/2020, 6:55 PMtask_run_id
as a target is a bug using 0.11.3
from prefect import task, Flow
from prefect.engine.results import LocalResult
lcl_res = LocalResult(dir="~/prefect_guide/results/{flow_name}")
@task(target="{task_name}/{task_run_id}",
)
def return_list():
return [1, 2, 3]
@task(target="{task_name}/{map_index}.prefect")
def mapped_task(x):
return x + 1
with Flow("blah", result=lcl_res) as flow:
mapped_task.map(return_list)
st = flow.run()
flow.visualize(flow_state=st)
Adam Roderick
05/27/2020, 7:29 PMflow.register()
Adam Roderick
05/28/2020, 4:51 AMIvan Shumilin
05/28/2020, 5:22 AMArsenii
05/28/2020, 6:09 AMCloud Hooks
here opens https://docs.prefect.io/orchestration/concepts/cloud-hooks.html while the correct URL is https://docs.prefect.io/orchestration/concepts/cloud_hooks.html (notice the _
instead of -
)Rafal
05/28/2020, 1:40 PMMarwan Sarieddine
05/28/2020, 3:46 PMprefect install agent kubernetes
• --namespace
only applies to the rbac in the generated manifest but not to the deployment - [i.e agents always gets deployed to default namespace]
• --name
to the cli call - so the agent’s name always gets registered as agent
prefect agent start
- I stand correctedMichael Reeves
05/28/2020, 4:18 PMGeoffrey Gross
05/28/2020, 8:44 PMDocker
storage class. Right now my flow imports env vars from a config.py
, found in the same directory with the file that holds my flow, which uses os.environ
. I would like to set those environment variables at run time but when, the healtchecks run and deserialize my flow in the docker build step, it is trying to evaluate those environment variables. This causes an error since I don't have defaults set. Does anyone have any ideas of have to handle this without ignoring healthchecks and putting environment variables into the Dockerfile that gets generated?Scott Zelenka
05/28/2020, 10:29 PMjob_spec
file to specify the K8 resources. But we have a use case where, in most situations the volume of work expected from our Flow fits within the specified resources, occasionally an input Parameter to the Flow will be such that it requires more resources. We're getting by this today by setting the resources to the maximum value, but (in theory) that restricts the number of concurrent Flows we can execute in the same environment.
Curious if there's an ability to adjust the memory resources required in the job_spec
based on the value of an input Parameter to the FlowRun?Darragh
05/29/2020, 10:22 AMMatthias
05/29/2020, 12:39 PM<https://domain.local/prefect>
. Is there a setting to tell the UI to load the CSS from <https://domain.local/prefect/css>
instead of trying to get it from <https://domain.local/css>
?Chris Hart
05/29/2020, 1:28 PMitay livni
05/29/2020, 1:55 PMupdate
. They flows are located in separate modules and have some duplicate parameters. Unfortunately the example pattern for joining flows with duplicate parameters only compiles if all the duplicated parameters are removed from their respective modules. Which makes multiple flow development difficult. Is there another pattern or solution where flows can be developed independently and then merged? --thanks https://stackoverflow.com/questions/60679595/how-does-one-update-a-prefect-flow-with-duplicate-parameters .Matthias
05/29/2020, 2:00 PMNoah Nethery
05/29/2020, 2:55 PMDan DiPasquo
05/29/2020, 3:51 PMHassan Javeed
05/29/2020, 4:21 PMWill Milner
05/29/2020, 6:05 PMGridcellcoder
05/29/2020, 7:50 PMprefecthq/ui:0.11.2
prefecthq/apollo:0.11.2
etc from starting on boot. I.e uninstall?Slackbot
05/30/2020, 7:38 AMAvi A
05/30/2020, 10:56 AMLocalDaskScheduler
with the default scheduler=threads
but it only runs one task at a time (mapped tasks). Am I missing some extra argument that allows tasks to run concurrently?Pedro Machado
05/30/2020, 5:39 PMAvi A
05/30/2020, 10:00 PMLocalDaskExecutor
. I keep getting the following error messages, which are probably related:
Error message: can't start new thread
Error message: 'DummyProcess' object has no attribute 'terminate'
BlockingIOError: [Errno 11] Resource temporarily unavailable
jars
05/30/2020, 10:43 PMa list of Prefect Secrets which will be used to populatesecrets(List[str], optional)
for each flow run.prefect.context
Used primarily for providing authentication credentials.It's not clear how I can access these secrets inside my flow. I've tried, exploring the
prefect.context
object, but can't seem to find anything.
Any examples or guidance?Sumant Agnihotri
05/30/2020, 11:55 PMAvi A
05/31/2020, 2:18 PMMax Lei
05/31/2020, 11:08 PM