Alex Welch
02/26/2021, 8:06 PMGitHub
storage? Reason for this is two-fold. Since we would be using the same block of code for just about every flow, it would help keep it semi-DRY. Second, it would allow us to pass different configs depending on environment variables (dev
vs prod
)
from prefect_utils import (
RUN_CONFIG,
STORAGE
)
flow.storage=STORAGE
flow.run_config=RUN_CONFIG
It works with the S3
storage. But when trying to use GitHub
I get the below.
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'prefect_utils'")
The module is in the same folder as the flow (flows/
). Is this an issue because when the flow is ran it is being ran from the parent directory? thus an issue with relative paths?Carlo
02/26/2021, 8:13 PMflow_a = StartFlowRun(flow_name="A", project_name="examples", wait=True)
flow_b = StartFlowRun(flow_name="B", project_name="examples", wait=True)
flow_c = StartFlowRun(flow_name="C", project_name="examples", wait=True)
flow_d = StartFlowRun(flow_name="D", project_name="examples", wait=True)
with Flow("parent-flow", schedule=weekday_schedule) as flow:
b = flow_b(upstream_tasks=[flow_a])
c = flow_c(upstream_tasks=[flow_a])
d = flow_d(upstream_tasks=[b, c])
Jack Sundberg
02/28/2021, 4:12 AMHiodo
03/01/2021, 7:19 AMSara Iris Valverde Bernuz
03/01/2021, 9:52 AMBeginning health checks...
System Version check: OK
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 151, in <module>
flows = cloudpickle_deserialization_check(flow_file_paths)
File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
flows.append(cloudpickle.loads(flow_bytes))
File "/usr/local/lib/python3.8/site-packages/cloudpickle/cloudpickle.py", line 562, in subimport
__import__(name)
ModuleNotFoundError: No module named 'gcloud'
Sara Iris Valverde Bernuz
03/01/2021, 9:58 AMMatt Liszewski
03/01/2021, 5:06 PMLaura Vaida
03/01/2021, 5:27 PM[2021-03-01 18:24:40+0100] ERROR - prefect.FlowRunner | Unexpected error: ModuleNotFoundError("No module named 'oauthlib'")
Traceback (most recent call last):
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\prefect\engine\runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\prefect\engine\flow_runner.py", line 619, in get_flow_run_state
final_states = executor.wait(
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\prefect\executors\dask.py", line 397, in wait
return self.client.gather(futures)
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\distributed\client.py", line 1993, in gather
return self.sync(
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\distributed\client.py", line 839, in sync
return sync(
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\distributed\utils.py", line 340, in sync
raise exc.with_traceback(tb)
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\distributed\utils.py", line 324, in f
result[0] = yield future
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\tornado\gen.py", line 762, in run
value = future.result()
File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\distributed\client.py", line 1858, in _gather
raise exception.with_traceback(traceback)
File "/opt/conda/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 75, in loads
ModuleNotFoundError: No module named 'oauthlib'
[2021-03-01 18:24:40+0100] ERROR - prefect.billwerk-contracts | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'oauthlib'")
Marwan Sarieddine
03/01/2021, 10:41 PMCarl
03/02/2021, 11:25 AMtasks.py
file with all your task functions? A .py
with a Flow
for each file you process? Interested to see how others work this.Sami Niemelä
03/02/2021, 12:13 PMHarshal Rane
03/02/2021, 3:07 PMRobert Bastian
03/02/2021, 3:27 PMrafaqat ali
03/02/2021, 3:45 PMwith Flow("Test run", run_config=DockerRun(labels=["dev"])) as flow:
flow.add_task(say_hello())
flow.register("Test")
@task
def say_hello():
print("Hello, world!")
When I run the flow from UI, I got the below error
Failed to load and execute Flow's environment: AttributeError("'NoneType' object has no attribute 'rstrip'")
Can anyone has faced such issue or help me in right direction?Laura Vaida
03/02/2021, 4:52 PM#define executor as dask executor with tcp from load balancer
flow.executor = DaskExecutor(address="35.198.104.230:8786") # tcp:port address of dask load balancer
flow.storage = Docker(registry_url="<http://gcr.io/keller-steering-enabling/flows|gcr.io/keller-steering-enabling/flows>", image_name="billwerk-contracts",python_dependencies=["pandas", "oauthlib ", "requests", "requests_oauthlib", "datetime", "oauth2client", "snowflake"])
flow.run_config = KubernetesRun()
flow.register('Billwerk')
Julie Sturgeon
03/02/2021, 4:53 PMmithalee mohapatra
03/02/2021, 9:13 PMHui Zheng
03/02/2021, 11:50 PM====== final results: <Success: "All reference tasks succeeded.">
or
====== final results: <Failed: "Some reference tasks failed.">
We have different types of successes/failures. Currently we could not differentiate them from the flow hook notification message. We would like to set customized messages to different scenarios and react differently.
for example
====== final results: <Failed: "Task_A failed due to timeout.">
Laura Vaida
03/03/2021, 11:44 AMJohn Ramirez
03/03/2021, 2:58 PMLaura Vaida
03/03/2021, 3:20 PMBrian Mesick
03/03/2021, 4:16 PMimage_tag
to a new version in order for K8s to pick up the new image. Does anyone have a more automated way of doing this? It’s not uncommon for someone to forget to make that change and leads to confusion about what’s running in prod.itay livni
03/03/2021, 4:22 PMflow
that runs successfully with a LocalExecutor
. However when the flow
is run using flow.executor = DaskExecutor()
the flow returns a Failed to deserialize
error on the final task
.
[2021-03-03 08:27:24-0600] INFO - prefect.TaskRunner | Task 'new-terms': Finished task run for task with final state: 'Success'
INFO:prefect.TaskRunner:Task 'new-terms': Finished task run for task with final state: 'Success'
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/distributed/protocol/core.py", line 151, in loads
value = _deserialize(head, fs, deserializers=deserializers)
File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 374, in deserialize
return loads(header, frames)
File "/home/ilivni/miniconda3/envs/py38lmap/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 70, in pickle_loads
return pickle.loads(x, buffers=buffers)
File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 75, in loads
return pickle.loads(x)
File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/tblib/pickling_support.py", line 26, in unpickle_exception
inst = func(*args)
TypeError: __init__() missing 1 required positional argument: 'request'
The last task is a FilterTask
filter_res = FilterTask()
res = filter_res(
keywords,
task_args={
"name": "new-terms"
}
)
Any suggestions?Laura Vaida
03/03/2021, 4:31 PMUnexpected error: TypeError("'NoneType' object is not subscriptable")
Sean Talia
03/03/2021, 4:49 PMtask_definition_arn
to the run configs), but I'm currently getting this error:
An error occurred (InvalidParameterException) when calling the RunTask operation: Override for container named flow is not a container in the TaskDefinition.
does this mean that my pre-registered task needs to have a container named flow
?Daniel Ahn
03/03/2021, 6:05 PMPrefect Core is licensed under the Apache Software License Version 2.0. Please note that Prefect Core includes utilities for running Prefect Server and the Prefect UI, which are themselves licensed under the Prefect Community License.• Is there a clearer documentation on what's under PCL vs Apache 2.0?
Prefect Core includes utilities for running Prefect Server and the Prefect UI• are these referring to the CLI wrapper around docker-compose (https://github.com/PrefectHQ/prefect/tree/master/src/prefect/cli)? • in other words, if I write my own docker-compose to deploy Prefect Core in my environment, the entire architecutre should be under Apache 2.0?
Jack Sundberg
03/03/2021, 7:49 PMBerty
03/03/2021, 8:25 PMSean Talia
03/03/2021, 9:43 PMDaniel Caldeweyher
03/03/2021, 10:30 PMDaniel Caldeweyher
03/03/2021, 10:30 PMJim Crist-Harif
03/03/2021, 10:35 PM