Saatvik Ramisetty
11/05/2020, 11:54 PMAgent ID
from the server. A little confused around this. TIA!Jeff Brainerd
11/06/2020, 2:08 AMJoël Luijmes
11/06/2020, 9:12 AMCarlo
11/06/2020, 1:28 PM2020-11-06T07:56:19.166-05:00
Copy
containerDefinition.flow-container.command -> Given: ['/bin/sh', '-c', "python -c 'import prefect; prefect.environments.execution.load_and_run_flow()'"], Expected: ['/bin/sh', '-c', 'prefect execute flow-run']
containerDefinition.flow-container.command -> Given: ['/bin/sh', '-c', "python -c 'import prefect; prefect.environments.execution.load_and_run_flow()'"], Expected: ['/bin/sh', '-c', 'prefect execute flow-run']
Kamil Okáč
11/06/2020, 1:56 PMRobin
11/06/2020, 2:05 PMNo heartbeat detected from the remote task; marking the run as failed.
errors. 💔
(e.g. https://cloud.prefect.io/accure/task-run/8e44bd11-e7c3-4982-bcf2-7711bc1ca4a9?logId=83475348-2531-4efb-9663-33748515908e)
This is particularly unexpected, as most of those tasks seem to have already almost finished... 🤔
How do you approach these errors and debug them? 🙂Sergey Maslov
11/06/2020, 2:43 PMMike Marinaccio
11/06/2020, 3:33 PMECR Task
and run_config
recently added, which sound like a potential solution.
Thanks for the input!Maura Drabik
11/06/2020, 3:50 PMlifecycle = Parameter("lifecycle")
true_branch = PrefectSecret("SCM_AUTOMATION_PROD")
false_branch = PrefectSecret("SCM_AUTOMATION_DEV")
ifelse(tasks.get_lifecycle(lifecycle), true_branch, false_branch)
secrets = merge(true_branch, false_branch)
When I run this flow locally, setting the lifecycle parameter set to DEV it works as expected. However, on the cloud, it is only using the PROD secret. Any guidance on why this might be happening? I am positive the cloud secrets are all configured correctly and the issue seems to be with the IFELSE, MERGE
Thanks in advance!Zach
11/06/2020, 5:11 PMZach
11/06/2020, 5:11 PMZach
11/06/2020, 5:33 PMDJ Erraballi
11/06/2020, 5:54 PMDJ Erraballi
11/06/2020, 5:56 PMJasono
11/06/2020, 7:06 PMAndrew Hannigan
11/06/2020, 9:51 PMReadTable
which extends Task.
This class just SELECTs data from a big SQL table. I want to perform this SELECT in parallel with say n sub-Tasks. So the ReadTable(n)
object should spawn n sub-Tasks when it runs. How would this work within Prefect's OOP paradigm, where ReadTable
inherits Task
?Cab Maddux
11/06/2020, 10:25 PMDor Menachem
11/07/2020, 12:27 AMAlexander
11/07/2020, 2:57 PMJasono
11/07/2020, 4:50 PMsys.path.append()
. This seems to work fine when I run the flow locally with flow.run(), but not when running it from the Prefect Cloud/Server. (The error says the module can’t be found) Does it sound right? If so, why would it be the case?Omar Sultan
11/08/2020, 8:33 AMAmanda Wee
11/09/2020, 2:17 AMidempotency_key
option and serialized_hash()
method for flow registration makes this really nice, but I'll still get an error for the project creation when the ECS task restarts. I could ignore this error, but then that could be problem if somehow the error is not due to the project already existing, but a genuine failure to create a new project.JC Garcia
11/09/2020, 6:30 AMWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f74c98d0710>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/cryptography/
ERROR: Could not find a version that satisfies the requirement cryptography (from versions: none)
ERROR: No matching distribution found for cryptography
Removing intermediate container f73d8367b237
The command '/bin/sh -c pip install requests cryptography snowflake-connector-python wheel' returned a non-zero code: 1
Traceback (most recent call last):
File "./flows/frieze-viewing-room-flow/flow.py", line 113, in <module>
flow.storage.build()
File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 351, in build
self._build_image(push=push)
File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 418, in _build_image
"Your docker image failed to build! Your flow might have "
ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
##[error]Bash exited with code '1'.
any ideas on why this might be happening? it looks like it cannot connect to pypi for some odd reason. ideas/thoughts welcome. thanks!deltikron
11/09/2020, 9:41 AMsubmitted
. I originally started with a kubernetes agent and thought it was a configuration issue between prefect and kubernetes, but I get the same behavior with a docker agent. The agents pull the correct docker image and submit it, but the flows don't enter a running state. I'd be grateful for any ideas or pointers! CheersPhilip Bennett
11/09/2020, 3:40 PMflow.register()
method uses Docker to build the image. I'm not sure if this is possible.
2. We will need to create firewall rules that allow Google Cloud Build to communicate with Prefect server running as a Compute VM.
I'd be interested how people are automatically deploying flows from any CI/CD platform.DJ Erraballi
11/09/2020, 7:21 PMTask A:
run(self, x, y):
Task B:
run(self, a_result, x, y):
with Flow() as flow:
x_parameter = Parameter("x", required=True)
y_parameter = Parameter("y", required=True)
# question is how do setup the parameter pasing from teh parameters -> A -> B
# my attempt but still dont know how to keyword_tasks in this scenario to pass along results from A
flow.add_task(x_parameter)
flow.add_task(y_parameter)
a = A()
flow.set_dependencies(a , upstream_tasks=[x_parameter, y_parameter], keyword_tasks=??? )
flow.set_dependencies(B(), upstream_tasks=[a, x_parameter, y_parameter], keyword_tasks=????)
DJ Erraballi
11/09/2020, 7:28 PMJakub Hettler
11/09/2020, 11:10 PMAmanda Wee
11/10/2020, 1:34 AMprefect backend server
. I intend to run apollo directly as a container in an ECS service, taking reference from the docker-compose.yml
file that prefect provides. Consequently, what do I need to configure to make it such that the container (and of course the other containers for hasura, graphql, and towel) starts as if I had run prefect backend server
followed by prefect server start
?Michelle Wu
11/10/2020, 7:16 AM