nick vazquez
05/27/2021, 1:12 AMFabrice Toussaint
05/27/2021, 9:51 AMJacob Baruch
05/27/2021, 12:30 PMChohang Ng
05/27/2021, 4:55 PMNikola Lusic
05/27/2021, 5:17 PMcpu
and memory
parameters of the ECSRun doesn't seem to have any effect. It seems that the agent always creates a task with 1CPU and 2GB RAM.
Do you maybe have some pointers for this issue?Irvin Tang
05/27/2021, 5:28 PMdef test_register_flow_docker(self):
register_flow(
flow=flow_docker,
flow_name="test",
project_name="test-project",
labels=["test-label"],
host_type="docker"
)
config = get_config("docker")
with set_temporary_config(config):
runner = CliRunner()
result = runner.invoke(get, ["flows"])
assert result.exit_code == 0
assert (
"test-docker" in result.output
and "test-project" in result.output
)
I was wondering if there was another way for testing flow registration? I want to be able to register the flow and the test should confirm that the flow successfully reigsteredFelipe Saldana
05/27/2021, 5:40 PMpost_runner.set_upstream(all_pushes)
post_runner.bind(mapped_run_name, mapped_data_root, mapped_collection_name)
post_runner(mapped_run_name, mapped_data_root, mapped_collection_name)
Using bind() works as I would expect: all_pushes completes entirely and then post_runner starts and completes.
Not using bind() post_runner does not wait for all_pushes to complete and then mayhem ensues and I get run() missing X required positional arguments messages.
Thoughts?Kathryn Klarich
05/27/2021, 6:19 PMAndrew Hannigan
05/27/2021, 7:58 PMPietro Immordino
05/28/2021, 4:46 AMChristian
05/28/2021, 7:11 AMThomas Hoeck
05/28/2021, 7:42 AMJohn Ramirez
05/28/2021, 1:17 PMNikola Lusic
05/28/2021, 2:18 PMECSRun
configuration be ran in a threaded mode, similar how LocalDaskExecutor
has the scheduled=threads
configuration?ciaran
05/28/2021, 2:23 PMciaran
05/28/2021, 2:32 PMCharles Leung
05/28/2021, 3:01 PMSean Harkins
05/28/2021, 4:25 PMOutOfMemoryError
.Jacob Goldberg
05/28/2021, 5:07 PM[2021-05-28 09:59:27-0700] INFO - prefect.Docker | Pushing image to the registry...
Traceback (most recent call last):
File "flow_registry.py", line 44, in <module>
build_and_register_all_flows(all_flows)
File "flow_registry.py", line 35, in build_and_register_all_flows
storage = storage.build()
File "/XXX/XXX/XXX/XXX/XXX/XXX/XXX/new_etl/venv/lib/python3.8/site-packages/prefect/storage/docker.py", line 303, in build
self._build_image(push=push)
File "/XXX/XXX/XXX/XXX/XXX/XXX/XXX/new_etl/venv/lib/python3.8/site-packages/prefect/storage/docker.py", line 378, in _build_image
self.push_image(full_name, self.image_tag)
File "/XXX/XXX/XXX/XXX/XXX/XXX/XXX/new_etl/venv/lib/python3.8/site-packages/prefect/storage/docker.py", line 586, in push_image
raise InterruptedError(line.get("error"))
InterruptedError: denied: Your authorization token has expired. Reauthenticate and try again.
This is confusing to me, because I have double checked that my local AWS authentication is all setup properly in the environment. Is this referring to something else? Relatively new to Docker, ECR, and Prefect. Any help appreciated! Any ideas?Tom Baldwin
05/28/2021, 8:26 PMrequests
library. Is there a way to trigger a flow run directly from ADF (without databricks as an intermediary)?itay livni
05/29/2021, 12:56 AMflow.result = LocalResult(
serializer=PandasSerializer(file_type="csv")
)
Task to pickle
num_clusters = value_count(df)
Thanks in advanceDaniel Bast
05/29/2021, 6:38 AMAurélien Vallée
05/30/2021, 7:07 AMwith Flow("myflow") as flow:
multiprocess = Parameter("multiprocess")
with case(multiprocess, True):
flow.executor = DaskExecutor()
PKay
05/30/2021, 9:57 PMUnexpected error: TypeError('no default __reduce__ due to non-trivial __cinit__')
Traceback (most recent call last):
File "/home/USERNAME/.local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/home/USERNAME/.local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 900, in get_task_run_state
result = self.result.write(value, **formatting_kwargs)
File "/home/USERNAME/.local/lib/python3.8/site-packages/prefect/engine/results/local_result.py", line 116, in write
value = self.serializer.serialize(new.value)
File "/home/USERNAME/.local/lib/python3.8/site-packages/prefect/engine/serializers.py", line 73, in serialize
return cloudpickle.dumps(value)
File "/home/USERNAME/.local/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/home/USERNAME/.local/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
return Pickler.dump(self, obj)
File "stringsource", line 2, in pymssql._mssql.MSSQLConnection.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
Tried both running on Ubuntu 20.04.02 and Windows 10, both are able to run flow locally, even tried using dockers and got the same error.
I am using python package pymssql to extract data from database and do some basic transformations with petl. Works fine when I run the flow without any agent/cloud. It's able to extract data and do transformations. Figure it's something to do with how it's passing data between functions.matta
05/30/2021, 10:02 PMFabrice Toussaint
05/31/2021, 8:46 AMitamar
05/31/2021, 9:51 AMAmit Gal
05/31/2021, 10:43 AMflow_a = StartFlowRun(flow_name="A", project_name="examples", wait=True)
flow_b = StartFlowRun(flow_name="B", project_name="examples", wait=True)
## Flow of flows
with Flow("parent-flow") as flow:
flow_b.set_upstream(flow_a)
flow.run()
as far as I can see, running this script waits with the scheduling of flow_b
, until flow_a
is completed. Is there a way to schedule them both, but wait with running flow_b
until flow_a
is completed?
My use case is I use a VM to run the flows, but my local machine (or phone) to run a small script that schedules them ad-hoc. flow_a
is often long, while flow_b
is a short one - and so as it currently stands any issues with my local machine might interrupt scheduling flow_b
.
There are a couple of ways I can imagine to solve this, but I wanted to make sure if my understanding is correct, or if there is a way to schedule both at and create a dependency for the running
of the flow?Jacob Blanco
05/31/2021, 10:51 AMfabian wolfmann
05/31/2021, 3:19 PM