Marwan Sarieddine
04/06/2021, 2:10 PMv0.14.15
basically the failure happens after the flow has completed running at the state handler level - please see the traceback in the threadFlorian Kühnlenz
04/06/2021, 3:00 PMprefect register --module
work.
My project looks like this:
flows
+ __init__.py
+ my_flow.py
+ shared_tasks
+ __init__.py
+ util.py
When I run prefect register --project 'Prefect Testing' -m '<http://flows.my|flows.my>_flow'
, I get No module named 'flows'
. What am I missing?Andor Tóth
04/06/2021, 4:01 PMSQL_DIR = Path('sql')
@task
def list_query_names():
return [f.name for f in SQL_DIR.glob('*.sql')]
@task(log_stdout=True, timeout=15, task_run_name='{name}-{date:%F_%T}', checkpoint=False)
def exec_query(name: str):
sql = Path(SQL_DIR / name).read_text()
print('Query name: %s' % name)
engine = sqla.create_engine(DSN)
rs = engine.execute(sql)
return dict(keys=rs.keys(), rows=rs.fetchall())
@task
def save_results(rs, name):
with (OUT_DIR / name).with_suffix('.txt').open('w') as f:
csv_writer = csv.writer(f, delimiter="\t")
csv_writer.writerow(rs['keys'])
csv_writer.writerows(rs['rows'])
with Flow("Queries") as flow:
query_names = list_query_names()
results = exec_query.map(query_names)
save_results.map(results, query_names)
flow.executor = LocalDaskExecutor(num_workers=2, schedule='processes')
flow.run()
Robin
04/06/2021, 4:13 PMHow do you build your prefect flows to ensure that the docker images run on different/the desired architectures/OS?
Tomás Emilio Silva Ebensperger
04/06/2021, 4:25 PMVincent
04/06/2021, 4:43 PMJacob Hayes
04/06/2021, 7:09 PMlocation="{flow_run_id}/{task_run_id}.prefect_result"
works for the unmapped tasks, but I don't see saved results for the mapped tasks (even though each one has a unique task_run_id
).Zack Novak
04/06/2021, 7:15 PMKieran
04/06/2021, 11:44 PMJonathan Chu
04/07/2021, 12:06 AMdata-ingestion-flow-staging
and data-ingestion-flow-production
?Ranu Goldan
04/07/2021, 1:48 AMtash lai
04/07/2021, 4:33 AMtarikki
04/07/2021, 5:26 AMTomás Emilio Silva Ebensperger
04/07/2021, 2:07 PMIgor Bondartsov
04/07/2021, 3:55 PMHawkar Mahmod
04/07/2021, 4:55 PMSatheesh K
04/07/2021, 5:15 PMxyzy
04/07/2021, 6:08 PMMatthew Millendorf
04/07/2021, 6:12 PMxyzy
04/07/2021, 8:06 PMPod prefect-job-6ab817cf-d6446 failed.
Container 'flow' state: terminated
Exit Code:: 1
Reason: Error
DockerRun doesn't even continue after pulling the image.Jay Sundaram
04/07/2021, 8:39 PMprefect register
and you don't get any error/warning messages but only:
================= 0 registered, 1 skipped =================
Jay Sundaram
04/08/2021, 2:07 AMRanu Goldan
04/08/2021, 2:18 AMPrefectSecret
Task
I want to pass the secret key via Parameter
. But it says that PrefectSecret argument should be statically defined so impossible to use parameter as secret key. Is that any workaround to solve that?
Thanks in advance!Igor Bondartsov
04/08/2021, 5:03 AMg.suijker
04/08/2021, 11:23 AMDomantas
04/08/2021, 1:37 PMhost
Parameter as MySQLExecute
argument?
Pseudo code:
from prefect.tasks.mysql.mysql import MySQLExecute
from prefect import Flow, Parameter
example_query = MySQLExecute(
name="example_query",
db_name="db name",
user="some user",
password="123456",
port=1234,
query="select * from example_table;"
)
with Flow("example") as f:
db_host = Parameter("db_host", default="host_address")
mysql_execute = example_query(host=db_host)
when I execute this pseaudo code, I get this error:
TypeError: __init__() missing 1 required positional argument: 'host'
According to documentation: https://docs.prefect.io/api/latest/tasks/mysql.html#mysqlexecute , host
argument is not optional, so by default it is required to be defined in a MySQLExcute
function.
Maybe there is a proper way to pass host
value to the MySQLExecute
function?Jay Sundaram
04/08/2021, 2:17 PMUnexpected error: ValueError('Could not infer an active Flow context.')
Jay Sundaram
04/08/2021, 3:08 PM--destination_bucket_name <Parameter: destination_bucket_name>
when was expecting:
--destination_bucket_name myorg-S3-bucket
Remi Paulin
04/08/2021, 4:30 PMRemi Paulin
04/08/2021, 5:24 PMRemi Paulin
04/08/2021, 5:24 PMKevin Kho
04/08/2021, 5:32 PMRemi Paulin
04/08/2021, 5:44 PMKevin Kho
04/08/2021, 5:47 PMRemi Paulin
04/08/2021, 5:55 PMKevin Kho
04/08/2021, 6:03 PMRemi Paulin
04/08/2021, 6:07 PM