Nitin Bansal
02/28/2022, 1:58 AMNitin Bansal
02/28/2022, 1:59 AMNitin Bansal
02/28/2022, 1:59 AMOvo Ojameruaye
02/28/2022, 6:40 AMTomer Cagan
02/28/2022, 7:38 AMStephen Lloyd
02/28/2022, 9:06 AMRUN_CONFIG = ECSRun(labels=['s3-flow-storage'],
task_role_arn=TASK_ARN,
execution_role_arn=EXECUTION_ARN,
image='<http://1111111.dkr.ecr.us-east-1.amazonaws.com/prefectdemo:latest|1111111.dkr.ecr.us-east-1.amazonaws.com/prefectdemo:latest>',
memory=512, cpu=256)
E Li
02/28/2022, 3:34 PMGreg Adams
02/28/2022, 4:07 PMDavid Michael Carter
02/28/2022, 5:03 PMfinalDF = finalDF.append(foo.map(aList))
TypeError: cannot concatenate object of type '<class 'prefect.tasks.core.function.FunctionTask'>'; only Series and DataFrame objs are valid
Am I on the right track here, or do I need a completely new approach?Dexter Antonio
02/28/2022, 5:27 PMEvgeny Liskovets
02/28/2022, 5:54 PMDaniel Komisar
02/28/2022, 6:01 PMflow_run(where: {state: {_eq: "Finished"}})
that returns nothing, I think because it is using string matching. Is the set of strings that can appear in state
listed somewhere? Thank you!Edvard Kristiansen
02/28/2022, 7:26 PMfrom prefect import Flow, task
from prefect.tasks.airbyte.airbyte import AirbyteConnectionTask
from datetime import timedelta
from prefect.schedules import Schedule
from prefect.schedules.clocks import CronClock
from prefect.executors import LocalDaskExecutor
schedule = Schedule(clocks=[CronClock("0 0 4 * * *")])
pipeline1 = AirbyteConnectionTask(
airbyte_server_host="localhost",
airbyte_server_port=8000,
airbyte_api_version="v1",
connection_id="xx-xx-xx")
pipeline2 = AirbyteConnectionTask(
airbyte_server_host="localhost",
airbyte_server_port=8000,
airbyte_api_version="v1",
connection_id="xx-xx-xx")
pipeline3 = AirbyteConnectionTask(
airbyte_server_host="localhost",
airbyte_server_port=8000,
airbyte_api_version="v1",
connection_id="xx-xx-xx")
pipeline4 = AirbyteConnectionTask(
airbyte_server_host="localhost",
airbyte_server_port=8000,
airbyte_api_version="v1",
connection_id="xx-xx-xx")
with Flow("dwh Sync", schedule, executor=LocalDaskExecutor()) as flow:
flow.add_task(pipeline1)
flow.add_task(pipeline2)
flow.add_task(pipeline3)
flow.add_task(pipeline4)
# Register the flow under the "airbyte" project
flow.register(project_name="airbyte")
I have already tried to use the @ task operator along with a function, but the task then finishes in a second without actually triggering the pipeline.Jacqueline Riley Garrahan
02/28/2022, 7:38 PMJosh
02/28/2022, 8:38 PMMeet Patel
02/28/2022, 8:40 PMDaniel Burkhardt
02/28/2022, 9:22 PMApoorva Desai
02/28/2022, 10:08 PMVictor Xie
02/28/2022, 10:11 PMScott White
02/28/2022, 10:59 PMveg_management
) was recently deleted with prefect delete project veg_management
because I had some old flows in there I wanted to clean-up. But now when I attempt to recreate it I get the errors below. Note that if I change the name of the project (e.g. veg_management_
) then it works.Chris Marchetti [Datateer]
02/28/2022, 11:01 PMprefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.
We changed from using an auth token to api key as a result of the error and are our flows are hanging. Are there changes that need to be made to our agents or to using service accounts? Thanks for the help.Apoorva Desai
03/01/2022, 9:08 AMprefect run --watch --id flowId --param strparam="paramValue"
. Then in the flow.py file, I am initializing outside the flow strparam = Parameter("strparam", default "hello")
but I'm unable to access "paramValue"
. Prefect only sees this <Parameter: strparam>
Yas Mah
03/01/2022, 9:31 AMAdam Roderick
03/01/2022, 1:31 PMdamien michelle
03/01/2022, 3:05 PMEdvard Kristiansen
03/01/2022, 3:05 PM@task
def run():
script....
key_location = "~/prefect/Prefect/keys/key.json"
with Flow("pipeline", executor=LocalDaskExecutor(), run_config=LocalRun(labels=["local"]), storage=Local()) as flow:
run()
I am getting error: FileNotFoundError: [Errno 2] No such file or directory:
Any ideas?YD
03/01/2022, 6:14 PMTony Yun
03/01/2022, 6:56 PM--param a=2
can accept a parameter name with a space in it? I only find this from the doc, where in my case the parameter is called “config list”:
$ prefect run --id "d7bfb996-b8fe-4055-8d43-2c9f82a1e3c7" --param a=2
Max Lei
03/01/2022, 7:23 PMDaskExecutor
, would DockerRun
be a good method to see if the dockerfile works or not?Stephen Herron
03/01/2022, 7:59 PMwait_for_flow_run
fails.
Not sure why that is - but the error is:
"An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid"
Something extra (or missing) required for the ECS task definition/role maybe?Stephen Herron
03/01/2022, 7:59 PMwait_for_flow_run
fails.
Not sure why that is - but the error is:
"An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid"
Something extra (or missing) required for the ECS task definition/role maybe?Kevin Kho
03/01/2022, 8:01 PMStephen Herron
03/01/2022, 8:06 PMKevin Kho
03/01/2022, 8:06 PM