Simon Macklin
12/05/2022, 6:50 PMMokshith Voodarla
12/05/2022, 7:13 PMChanaka De Silva
12/05/2022, 8:16 PMModleNotFound
error. I register the code as follows
prefect register --project CHX -p sparsify.py -n "sparsify" --label chx-2022-2.2
The folder where sparsify.py
sits has the missing module’s python code; masks.py. And I import the module and relevant functions as follows
from masks import MaskClient, combine_masks
Somehow when I run sparsify via UI it can’t find it. The code just runs fine in an ipython
sessionJean-Michel Provencher
12/05/2022, 8:22 PMKristian Andersen Hole
12/05/2022, 9:49 PMIf your deployment definesDoes this hold true forand doesn’t specify any remote storage block, Prefect will assume that the flow code is baked into the image.-ib docker-container/prod
-ib ecs-task/prod
as well?nchopra
12/05/2022, 10:59 PMdbt = DbtShellTask(
return_all=True,
profile_name="dbt_repo",
environment="dev",
# profiles_dir=".",
overwrite_profiles=True,
helper_script=f"cd {DBT_DIR}",
log_stdout=True,
log_stderr=True,
dbt_kwargs={ # removed for readability}
)(command="dbt docs generate")
The folder structure is like:
prefect-repo
|- containers
|- dbt-prefect-container
|-dbt-repo(submodule)
|- flows
|-flow.py (from where I want to access the location DBT_DIR)
My questions are:
1. Is it okay to use git submodule method inside Prefect?
2. How to pass correct repo location so that Prefect cloud doesn't start the root at local dir: /Users/my-usernameJohn
12/06/2022, 12:44 AMMichał Augoff
12/06/2022, 1:38 AMprefect.infrastructure
module)? I assume no since this needs to be present in the agent’s env but double checkingtsteijvers
12/06/2022, 8:39 AMFlow run RUNNING: terminal tasks are incomplete.
as the last log message, then flow stays in running state forever. Is this a known issue?Sunjay
12/06/2022, 9:00 AMJames Zhang
12/06/2022, 11:56 AMDeepanshu Aggarwal
12/06/2022, 1:52 PMfrom prefect import task, flow
from prefect.deployments import run_deployment
@task()
def task_A(event):
output = run_deployment(
name='some deployment',
parameters={'event': event}
)
if output.state_name != 'Completed':
raise Exception("flow run not completed")
return output
@flow()
def flow_B():
a = [1, 2, 3, 4, 5, 6, 7]
# method 1
for i in a:
task_A.submit(i)
# method 2
output = task_A.map(a)
for tasks in output:
tasks.wait()
#do something when all the parallel executions have completed for task A
this is what my flow looks like but in method 1 it runs next iteration of task_A only when previous iteration is complete
in method2 it doesnt wait for the tasks to completePatrick Tan
12/06/2022, 3:24 PMMarc Lipoff
12/06/2022, 3:57 PMAlexandru Anghel
12/06/2022, 4:08 PMdistributed.scheduler.KilledWorker: Attempted to run task prometheus_to_gcs-02d33e7b-92a6-4fe8-8258-ba9efaa3d609 on 3 different workers, but all those workers died while running it.
What could be the problem here? Other smaller datasets are running fine.
I am running it with adaptive workers (min 4, max 10). I am not sure though if I used correctly to dask configuration for what i am trying to do. Can you please have a look at my code in the thread?
I am able to run the same flow in Prefect 1.4.
Thanks!Prem Viswanathan
12/06/2022, 4:10 PMBob Colner
12/06/2022, 4:54 PMJoseph Loss
12/06/2022, 5:19 PMXavier Babu
12/06/2022, 11:07 PMZac Hooper
12/06/2022, 11:35 PM@task
def prod_9e_etl_asx_anns():
logger = get_run_logger()
res = invoke_lambda("prod_9e_etl_asx_anns", {}, True)
<http://logger.info|logger.info>(res)
return ""
@task
def prod_9e_etl_asx_ann_sentiment():
logger = get_run_logger()
res = invoke_lambda("prod_9e_etl_asx_ann_sentiment", {}, True)
<http://logger.info|logger.info>(res)
return ""
@flow(task_runner=SequentialTaskRunner(), retries=2)
def perfom_asx_anns_etl():
x = prod_9e_etl_asx_anns.submit(return_state=True)
y = prod_9e_etl_asx_ann_sentiment.submit(wait_for=[x])
I need to make sure the tasks happen in sequential order. Any idea how I prevent this error?Jarvis Stubblefield
12/07/2022, 12:26 AMprefect agent -q tenzinga_django_prod
on my server … I’m using a systemd
to run. It errors and lets me know that Orion (the server not agent) needs SQLite >= 3.24.0 but Amazon Linux 2 only has 3.7.17 as it’s latest version.Slackbot
12/07/2022, 3:14 AMSanthosh Solomon (Fluffy)
12/07/2022, 6:20 AMSanthosh Solomon (Fluffy)
12/07/2022, 6:21 AM陳柏翰
12/07/2022, 7:35 AMNic
12/07/2022, 11:03 AMArnoldas Bankauskas
12/07/2022, 12:17 PMLiz McCutcheon
12/07/2022, 12:34 PMJoão Coelho
12/07/2022, 1:33 PMZachary Loertscher
12/07/2022, 4:22 PMpyodbc
However, I continue to get the error Failed to load and execute flow run: ModuleNotFoundError("No module named 'pyodbc'")
I have..
• added pyodbc
to requirements.txt, which is installed using our Dockerfile
with RUN pip install -r requirements.txt
Info:
Prefect version: 1.2.4
Deployment: AWS CDK for AWS
Where does your docker container live?: an ECS Fargate deployment.
What OS runs on your docker container? Amazon Linux 2
Any ideas? Is Prefect just not finding the package?Zachary Loertscher
12/07/2022, 4:22 PMpyodbc
However, I continue to get the error Failed to load and execute flow run: ModuleNotFoundError("No module named 'pyodbc'")
I have..
• added pyodbc
to requirements.txt, which is installed using our Dockerfile
with RUN pip install -r requirements.txt
Info:
Prefect version: 1.2.4
Deployment: AWS CDK for AWS
Where does your docker container live?: an ECS Fargate deployment.
What OS runs on your docker container? Amazon Linux 2
Any ideas? Is Prefect just not finding the package?Mason Menges
12/08/2022, 7:44 PMZachary Loertscher
12/08/2022, 7:51 PMUniversalRun
? Also, not sure if it matters, but this is running via a Local Agent that gets booted up from the dockerfile
Mason Menges
12/08/2022, 8:08 PM