Anze Kravanja
10/01/2021, 10:00 PMTim Finkel
10/01/2021, 10:35 PMEssential container in task exited
. I’ve been trying to debug this for a bit and am not sure how to proceed. Any advice?Nikhil Acharya
10/02/2021, 1:14 PMflow.run(parameters=dict(studentId=student, data=data))
Tadej Svetina
10/03/2021, 8:25 AMFina Silva-Santisteban
10/04/2021, 3:10 AMsome reference tasks failed
error message but you can’t see the stack trace the way you can when you run a ‘simple’ flow. Do you have any suggestions for how to implement error logging for flow-of-flows? I was thinking of using a state handler but those only know the ‘state’ of the flow as the name suggests and don’t have access to a stack trace. Curious to hear your thoughts! 🤔Klemen Strojan
10/04/2021, 7:31 AMSuraj Mittal
10/04/2021, 11:09 AMVikram Thirumalai
10/04/2021, 2:43 PMbotocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeTasks operation (reached max retries: 4): Rate exceeded
We tried updating Dask but we're not super sure how to proceed since we're still getting the errorFilip Lindvall
10/04/2021, 2:57 PMAdam Brusselback
10/04/2021, 7:57 PMAndrew Hannigan
10/04/2021, 8:53 PM@task
decorator, or as classes by subclassing Task
. In general when programming in python and presented with these options, I typically reach for the subclassing option. I find I can think about problems at a higher level with OOP, easier to build up complex components from simpler ones, cleaner abstraction, etc. However I do find it’s a bit clunky with at times when building flows. Is it recommended to avoid the subclassing approach and use a functional approach instead when possible?Matt Alhonte
10/04/2021, 9:04 PMJacob Goldberg
10/04/2021, 9:19 PMChris Leber
10/04/2021, 9:30 PMfile_list = ['./file_1.txt', './file_2.txt', './file_3.txt', './file_4.txt', './file_5.txt', './file_6.txt', './file_7.txt', ..., './file_100.txt']
@task
def test_task(input_file):
result = some_function(input_file)
return result
def ecs_cluster(n_workers=4):
"""Start a ecs cluster using the same image as the flow run"""
return ECSCluster(
n_workers=n_workers, image=prefect.context.image, region_name="us-east-1"
)
with Flow(
"test flow",
storage=S3(bucket="storage"),
run_config=ECSRun(task_definition=task_definition),
) as flow:
test_task.map(input_file=file_list)
flow.executor = DaskExecutor(
cluster_class=ecs_cluster, cluster_kwargs={"n_workers": 10}
)
Ismail Cenik
10/04/2021, 11:11 PMMatt Alhonte
10/05/2021, 1:07 AMUnexpected error: ClientError([{'path': ['user'], 'message': 'field "user" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.user', 'code': 'validation-failed', 'exception': {'message': 'field "user" not found in type: \'query_root\''}}}])
(we're on 0.13.19). Anyone know what might be up?chicago-joe
10/05/2021, 2:03 AMChris White
10/05/2021, 2:32 AMBastian Röhrig
10/05/2021, 8:03 AMWilson Bilkovich
10/05/2021, 1:30 PMDask Performance Report
of a Prefect Flow that is using DaskExecutor
i.e. https://distributed.dask.org/en/latest/api.html#distributed.performance_report
It looks like the dask.distributed.performance_report()
needs to run inside the context of the temporary Dask Cluster that Prefect spins up, but I don’t see how to get into those internals.Kevin Weiler
10/05/2021, 1:56 PMdef nomad_batch_node_cancel_handler(task: Task, old_state: State, new_state: State):
if (new_state == Cancelled) & (old_state == Running):
<http://_LOGGER.info|_LOGGER.info>(f"killing job {nomad_job_name}")
Nomad().stop_job(job_id=nomad_job_name)
It gets nomad_job_name
from an enclosing function (it’s a closure).
When I click the “cancel” button in the UI - I don’t think this handler is getting executed - my log message isn’t there, and the task is not killed. Is there something wrong with how I’m testing for state change? It seems to me that the task is in a Running
state when I click cancel, and the UI seems to think it goes to a Cancelled
state thereafter.Vincent
10/05/2021, 2:12 PMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 929b8b8d-b75c-4a7b-b0ff-e0433a903fee: provided a running state but associated flow run 3775f8a8-6d8c-452b-a69f-e51eb6bc07e7 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 1917, in set_task_run_state
result = self.graphql(
File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 929b8b8d-b75c-4a7b-b0ff-e0433a903fee: provided a running state but associated flow run 3775f8a8-6d8c-452b-a69f-e51eb6bc07e7 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Tony Yun
10/05/2021, 3:29 PMRunNamespacedJob
task. I had set the log_level='info'
option, but when it’s failed, no any logs are being sent to Prefect. Instead, I only get this from UI.
FAIL signal raised: FAIL('Job dbt-run-from-flow failed, check Kubernetes pod logs for more information.')
After going to the k8s pod, I see exception logs:
dbt ls --models tm_snowflake.* --profiles-dir=. --profile default --target dev
Encountered an error while reading profiles:
ERROR Runtime Error
Compilation Error
Could not render {{ env_var('DBT_PASSWORD') }}: Env var required but not provided: 'DBT_PASSWORD'
Encountered an error:
Runtime Error
Could not run dbt
make: *** [Makefile:6: ls] Error 2
How could I pass any logs from k8s pods to Prefect UI?kiran
10/05/2021, 5:07 PMflow.visualize()
to run on a Linux server (running the code, which is on the server, from my Mac)? I’m having issues with xdg-open
and also gio
and have now gone down several google/stack overflow rabbit holes with only minor successesek
10/05/2021, 6:00 PMimport my_lib
from prefect import task, Flow
from prefect.storage import S3
@task
def my_func():
my_lib.func()
with Flow("myflow") as flow:
flow.storage = S3(
bucket="BUCKET_NAME",
stored_as_script="true",
local_script_path="main.py"
)
flow.register('myflow')
my dir looks like this:
.
├── main.py
└── my_lib
├── __init__.py
└── db.py
I'm trying to push my code up to s3 bucket but my my_lib
dir doesn't get package up along with main.py
what am I missing here?Kevin Weiler
10/05/2021, 6:38 PMKen Nguyen
10/05/2021, 7:04 PMBob Colner
10/05/2021, 7:34 PMAdam Brusselback
10/05/2021, 9:16 PMFrederick Thomas
10/05/2021, 9:52 PMsudo du -h /mnt/data/prefect/ > log.txt
and after reading the file found this:
217G /mnt/data/prefect/.prefect/results
My question being is it safe to delete the results in the folder, and if not can they be stored elsewhere safely? Thanks!Frederick Thomas
10/05/2021, 9:52 PMsudo du -h /mnt/data/prefect/ > log.txt
and after reading the file found this:
217G /mnt/data/prefect/.prefect/results
My question being is it safe to delete the results in the folder, and if not can they be stored elsewhere safely? Thanks!Kevin Kho
10/05/2021, 9:56 PMFrederick Thomas
10/05/2021, 10:00 PMKevin Kho
10/06/2021, 12:33 AM@task(checkpoint=False)
def ...