Michał Junczyk
06/04/2020, 2:25 PMDarragh
06/04/2020, 2:31 PM[2020-06-04 11:01:26,972] INFO - agent | Deploying flow run 05ec3609-4457-4085-8cf3-d01a6d13ccb1
I can see that it’s created the TaskDefinition , but after that it just sits there and never runs the Task, so I’ve no idea what’s going on, and the UI and logs never show any update. Am I missing something? Some magic words to make the agent actually run the task as opposed to just registering?
Being able to turn on debug logging would also be a massive bonus 🙂 All I’m getting from the agent is INFO, and I’ve added the [logging] level="DEBUG"
to the config.toml, and added export PREFECT__LOGGING__LEVEL="DEBUG"
but nothings happening
UPDATE
In a fit of desperation I went on to my EC2 box and edited the fargate agent code to make all DEBUG statements INFO, and lo and behold, more log outpu.
BUT!!! As far as the deploy_flow
function is concerned everything succeeded. And I can see the task definition has been created, but nothing else ever happens in log s or the UI. So I’m slightly wiser but not by a lot. 🙂 Even digging into deploy_and_update_flow_run
in the root agent.py file isn’t giving me much, it seems like everything is succeeding, so I’m at a loss as to why nothing is coming in the UI to show failed, passed, stalled…
UPDATE 2
Ok, found something - I took the definition that prefect created and ran it manually, and it gets a CannotStartContainerError
failed to initialise logging. AccessDenied to create Cloudwatch log group. Fixing.
I’ll be back 😂
Does the fargate task need access to call back to the agent/graphql for updates? Finally found some logs and it seems like the flow could be trying to make a callback to localhost:4200, is that right? I realised this was stupid question as I wrote it.
UPDATE 3
Turns out it does. From the Fargte Agent docs: Core server
In order to use this agent with Prefect Core's server the server's GraphQL API endpoint must be accessible.
But nothing to say how to configure it? I’ve overridden the cloud.api endpoint in the config.toml and I can see the update coming out in the logs, but the task is still trying to call back to localhost, presumably because the api endpoint override needs to be injected into the task? As in, into the docker container running the flow?Dan DiPasquo
06/04/2020, 4:36 PMScott Zelenka
06/04/2020, 6:04 PMPrefectSecret
to read from?
Our use case is that we have different key/value pairs depending on the lifecycle the Flow is being executed in, so we split that up into two different Secrets, and use a Parameter to trigger which lifecyle it should attempt to process:
with Flow() as flow:
lifecycle = Parameter("lifecycle", default='dev')
secrets = PrefectSecret(f"SECRET_NAME_{lifecycle.run().upper()}")
That seems hacky, to have to call .run()
on the Parameter though. Wondering if anyone found a cleaner approach.Scott Zelenka
06/04/2020, 7:22 PMimagePullPolicy
when launching a Job through the Kubernetes Agent?
Since the image of the Flow to run will always have a tag for the specific version, this value could be changed from Always
to IfNotPresent
. This would decrease the startup time for Flows that get executed frequently, since there's a good chance the image will already exist on the K8 node where it's being executed.Skip Breidbach
06/04/2020, 11:23 PMLOOP
construct. The scenario I have in mind is that I have one task that is generating a series of chunks of data. Each chunk would ideally start to be processed by a different downstream task before the generator task completes. I don't know how many it will create in advance, so I don't think I can map
the task. (It's also possible that I'm just thinking about the problem the wrong way.)philip
06/05/2020, 1:58 AMDan DiPasquo
06/05/2020, 2:15 AMpython_dependencies=[
"kubernetes",
"docker",
- "prefect[gcp]",
- "prefect[kubernetes]",
+ "prefect[gcp]~=0.11",
+ "prefect[kubernetes]~=0.11",
(this may be redundant?). I can see in the output from the register/build process:
Step 1/36 : FROM prefecthq/prefect:0.11.5-python3.8
...
Step 30/36 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@0.10.7#egg=prefect[kubernetes]>
---> Running in e3773eba79d4
Name: prefect
Version: 0.11.5
...
Additionally I have pulled and run the docker image from the registered flow and inside:
root@b3fc72e3e101:/# python
Python 3.8.3 (default, May 20 2020, 20:38:46)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import prefect
pr>>> prefect.__version__
'0.11.5'
I've now even deleted the flow, all past versions, and registered again and still seeing Prefect Core Version: 0.10.7 in Cloud UI - what am I missing / doing wrong?An Hoang
06/05/2020, 3:12 PMAn Hoang
06/05/2020, 3:16 PMMatthias
06/05/2020, 4:12 PMschedule = IntervalSchedule(
start_date=datetime.utcnow() + timedelta(seconds=1),
interval=timedelta(hours=1),
)
flow.schedule = schedule
flow.register()
and it works, but after 10 intervals, it simply stops. Where am I wrong?Marwan Sarieddine
06/05/2020, 8:08 PMResult.validate
does - and where exactly it is being called (can’t seem to find when its invoked by the runner?) - what does it mean for a task to have an invalid Result ? (any explanation would be much appreciated - thanks)Luis Muniz
06/06/2020, 11:26 AMLuis Muniz
06/06/2020, 11:28 AMKai Weber
06/06/2020, 11:34 AMAlex Joseph
06/06/2020, 6:18 PMKevin Weiler
06/07/2020, 8:28 PMKevin Weiler
06/07/2020, 8:36 PMMatt Segal
06/08/2020, 4:49 AMprefect server
CLI does not seem to support a daemonized mode. Is there some standard way to use the docker-compose config to run the core server using something like Docker Swarm?Paul
06/08/2020, 5:16 AMprefect cloud login --token $TOKEN
results in
[WinError 10061] No connection could be made because the target machine actively refused it'))
More precisely other prefect auth commands yield the same result.
Did I miss anything in the docs in order to establish a connection to prefect cloud?Matthias
06/08/2020, 10:05 AMRafal
06/08/2020, 12:45 PMMatthias
06/08/2020, 1:28 PMDan Ball
06/08/2020, 2:23 PMZach
06/08/2020, 4:42 PMRon Van Buskirk
06/08/2020, 6:46 PM...
[2020-06-05 15:50:12,454] INFO - prefect.TaskRunner | Task 'task a': Starting task run...
[2020-06-05 17:15:04,576] INFO - prefect.TaskRunner | Task 'task a': finished task run for task with final state: 'Success'
[2020-06-05 17:15:04,576] INFO - prefect.TaskRunner | Task 'task b': Starting task run...
[2020-06-06 05:22:27,199] INFO - prefect.TaskRunner | Task 'task b': finished task run for task with final state: 'Success'
[2020-06-08 14:23:12,015] INFO - prefect.TaskRunner | Task 'task c': Starting task run...
Is there anything that I could have accidentally entered in the terminal that would have put it into some type of 'wait' state? Or any other suggestions about things I could check?Avi A
06/08/2020, 7:10 PMAvi A
06/08/2020, 9:08 PMPaul
06/08/2020, 9:19 PMSTOPPED (CannotPullContainerError: Error response from daem)
To my understanding the launched Fargate Instance does not have any access to the Docker Image of the Flow. Concerning the solution from
https://docs.prefect.io/orchestration/execution/storage_options.html#docker
# Non-Docker Storage for Containerized Environments
for rapid deployment, which image would the
metadata={"image": "repo/name:tag"}
refer to in the example given?Darragh
06/08/2020, 9:27 PMprefecthq/prefect:python3.7
and installs a bunch of other junk in there too. The Flow builds and registers, but when I kick it off I get the following:
Failed to set task state with error: ConnectionError(MaxRetryError("HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha
Searching in this channel I see an issue that was identified around this and fixed in April, so I’m not sure if I should still be seeing it, or if there’s config that needs to be passed to the Docker agent to run it? It’s running on Docker agent locally on Macbook if that’s any help.. Prefect version installed locally is 0.11.1, [upgrading now to test] and python in local and the docker image is 3.7
UPDATE: Never mind, upgrade to 0.11.5 did it!