Robin
10/16/2020, 10:22 AMk8s
= 3 and customer
= 1.
So, we thought that with a prefect agent that has the labels k8s
and customer
, we should be able to run several flows at the same time (e.g. up to 3 k8s
labeled flows) but always only 1 customer
labeled flow.
However, we were not able to run a ["customer", "k8s"]
and a k8s
flow at the same time. 😮
We then added another k8s
agent (without the customer
label) and now the flows indeed can run add the same time, 1 flow on each agent.
Is this intended behavior?
For us that does not feel right, because in that case one needs to update infrastructure whenever adding flow concurrency limits, right? 🤔Alexander
10/16/2020, 12:11 PMprefect server start
does not let you to detach?? It has so much control on how whole prefect server is deployed yet you cant use on anything rather then local console. Also i found that there was no actual documentation on all services of prefect server and how to connect them to each other. I was able to understand it only by reverse-engineering docker compose file and prefect server cli source code 😔 Other than that there is no really fancy stuff here, just regular compose file with some variables defined.
3. There is a big security hole in the whole setup called apollo. UI (read: your web browser) need access to apollo endpoint. There is no auth in both UI and apollo, which means you have to put the whole service behind the VPN, at least.
4. Then i have this nifty little dockerfile wich defines flow execution and registration environment. Nothing fancy too, except you have to somehow forward an apollo endpoint address. It also uses the same image as used by docker agent as its base.
5. On every commit i run this code:
docker build -t hs-databricks:build_runtime -f flows/Dockerfile --build-arg APOLLO_PUBLIC_URL=$APOLLO_PUBLIC_URL .
# Register all flows
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e APOLLO_PUBLIC_URL=$APOLLO_PUBLIC_URL hs-databricks:build_runtime \
python /src/hs-databricks/flows/registry.py
6. And there is a central flow registration script, which scans ./flows folded for python files, extracts flows from them, builds docker storage and registers the flows. It has no comments, but i wrote it just yesterday ☺️
My plans are, since this all is running in docker, try to deploy the prefect server + agent + flow storage in AWS Fargate, so i dont need to maintain any EC2 at all.
Hope this will help anyone 👍psimakis
10/16/2020, 3:18 PMwebhook_secret
on a SlackTask
using a PrefectSecret
? I have tried to provide the webhook_secret
on the instantiation but I get a TypeError
Given this flow:
with Flow(name='flow name') as flow:
slack_custom_webhook_url = PrefectSecret('SLACK_E_SPIDER_STATISTICS')
spider_name = Parameter('spider_name', default='spider a')
with case(spider_name, 'spider a'):
SlackTask(webhook_secret=slack_custom_webhook_url)(message='Test')
Check out the full trace-back in the thread:Aaron Y
10/16/2020, 7:30 PMKrzysztof Nawara
10/17/2020, 12:45 PMAlberto de Santos
10/17/2020, 2:00 PMAlberto de Santos
10/17/2020, 2:02 PMNicolas Bigaouette
10/17/2020, 6:21 PMFlow
seems to support this. But I have a specific requirement where a one (or more) command(s) are actually RPCs. Those RPCs are customs; the remote command to be executed is added to a queue and the remote worker polls that queue. After having executed a command, the remote worker will POST
its result to the backend. There is multiple instances of the backend running (for redundancy and scaling) so the result will probably not be received by the process that created the RPC call. This creates a discontinuity in the code; The task driving the workflow has to stop when enqueuing its remote command and another instance will pick up the work at a later time.
I can probably model my sequences with prefect just fines, except I am not sure how to handle the discontinuity described above. I though I could raise a "pause" signal in the task that initiated the custom RPC. Then the backend instance receiving the POST
could send a "continue" signal to the paused task. Is that even possible? Can such a workflow be modeled by prospect?
Thanks!!Bruce Haggerty
10/17/2020, 9:16 PMLeo Meyerovich (Graphistry)
10/18/2020, 12:19 AMfor p in configs: interval_flow.run(parameters=p)
<= was not concurrent
2. Docs: flow docs say to manually call `FlowRunner`: `for p in configs: FlowRunner(flow=interval_flow).run()`<= unclear if the intervals run concurrently, and unclear how to pass in different p
to different flows
Any pointers?Felix V
10/18/2020, 8:48 PMJasono
10/18/2020, 11:40 PMwith Flow('') as flow
r=task1()
r2=task2()
task3(r)
flow.run()
Lior
10/19/2020, 9:11 AMMagda Stożek
10/19/2020, 9:11 AMAlberto de Santos
10/19/2020, 10:00 AMAlberto de Santos
10/19/2020, 10:02 AMtrigger
setup to any_successful
), how could I deal with those tasks with the TRIGGERFAIL
result? Could I make something like if TRIGGERFAIL: then ...
?Avi A
10/19/2020, 11:50 AMale
10/19/2020, 11:57 AMAWSSecretsManager
in a flow which runs on Prefect Fargate Agent.
The flow works fine if the flow runs with launch_type="FARGATE"
If the flow runs with launch_type="EC2"
then I get the following error:
botocore.exceptions.NoRegionError: You must specify a region.
It seems that the flow cannot pick the region from the provided task/execution role…
Any suggestions?Adam
10/19/2020, 12:21 PMemre
10/19/2020, 1:45 PMRalph Willgoss
10/19/2020, 2:28 PMRalph Willgoss
10/19/2020, 3:04 PMNewskooler
10/19/2020, 3:26 PMValidationError: Value is not JSON-compatible
Does anyone know why?
I could not trace the source of the issue : /tarikki
10/19/2020, 3:34 PMprefect server start
in the background? Alternatively, is there a docker-compose.yml
file that can be used to start the local setup in the background? Cheers! 😊Jasono
10/19/2020, 4:08 PMContext object has no attribute 'today'
error when I try to get prefect.context.today
in the Flow block. According to the documentation, today
is one of those variables always available during task runs. What am I missing?Robin
10/19/2020, 4:15 PMKrzysztof Nawara
10/19/2020, 5:37 PMBob Colner
10/19/2020, 6:37 PMtom
10/19/2020, 9:02 PMLOOP(result=X)
in my task and at some point I pause and resume that task (e.g. using raise PAUSE()
within the task and then resuming in the UI). It seems like my task is resuming from the beginning and loses the loop context. Is there any way to make it keep the loop context? I tried doing raise PAUSE(result=X)
but it would still resume from the beginning.
Another question I have is if the result in LOOP
needs to be small or if it there is a way to have a growing Result
instance that potentially stores a lot of data.Rob Fowler
10/19/2020, 10:59 PM