Hi I am using prefect to orchestrate my spark jobs. The spark jobs are submitted with

spark-submit

using prefect’s

ShellTask

. I have created a subclass of ShellTask to invoke the

spark-submit

.The spark jobs are running on k8s. There seems to be an issue especially with long running tasks where the spark jobs completes but the prefect task in unaware of the status and continues keep running. I have tried few suggestions including Added to the following configuration to job template

“<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>”: “false”

It looks like

ShellTask

has some issue in maintaining the status To overcome this I am thinking the following approach: - In map of jobs say

["j1", "j2", "j3"]

, submit all the jobs and not wait for execution completion (In my case since there is no dependency between tasks) - Separate task to monitor the status for all the tasks. This task runs in a loop say for max 1 hour before timeout. This task looks at status for each task and updates the overall status What is the best way to accomplish this looped task on prefect ?

@Marvin how do I test if a flow is Completed?

Hi I’m trying to use Blocks for using secrets in my ETL. However, I keep on running into errors when trying to load them:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [1], in <cell line: 6>()
      3 secret_block = Secret.load("etlharvestqaspassword")
      5 # Access the stored secret
----> 6 secret_block.get()

AttributeError: 'coroutine' object has no attribute 'get'

I defined the block within the Orion UI, because when I tried to define it via code in a simple script (as suggested here), I get the following kind of error:

prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for url '<http://ephemeral-orion/api/block_documents/>'
Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'loc': ['body', 'name'], 'msg': 'name must only contain lowercase letters, numbers, and dashes', 'type': 'value_error'}, {'loc': ['body', '__root__'], 'msg': 'Names must be provided for block documents.', 'type': 'value_error'}], 'request_body': {'name': 'test2_password', 'data': {'value': 'test2'}, 'block_schema_id': '8019abd6-409a-4f91-9367-bc8343c31763', 'block_type_id': '29fb0ec8-f7e9-4527-984c-48f8675f2bc4', 'is_anonymous': False}}
For more information check: <https://httpstatuses.com/422>

I’m mostly using Prefect within a jupyter notebook and from within a virtualenv. Thanks in advance for anyone who could point me to what’s going on 🙂

Hey folks 👋 I’m getting these warnings while running tests on a new collection I’m working on

../../../.pyenv/versions/3.10.9/envs/prefect-huggingface/lib/python3.10/site-packages/prefect/testing/standard_test_suites/task_runners.py:155
  /Users/alessandro.lollo/.pyenv/versions/3.10.9/envs/prefect-huggingface/lib/python3.10/site-packages/prefect/testing/standard_test_suites/task_runners.py:155: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo?  You can register custom marks to avoid this warning - for details, see <https://docs.pytest.org/en/stable/how-to/mark.html>
    @pytest.mark.flaky(max_runs=4)  # Threads do not consistently yield

Not sure what’s the issue here, anyone experiencing this?

tests/test_tasks.py::test_get_inference_result_fails
  /Users/alessandro.lollo/.pyenv/versions/3.10.9/envs/prefect-huggingface/lib/python3.10/site-packages/prefect/orion/database/migrations/versions/sqlite/2022_04_25_135207_b75d279ba985_replace_version_with_checksum.py:102: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: <https://sqlalche.me/e/b8d9>)
    meta_data = sa.MetaData(bind=connection)

tests/test_tasks.py::test_get_inference_result_fails
  /Users/alessandro.lollo/.pyenv/versions/3.10.9/lib/python3.10/contextlib.py:142: SAWarning: Skipped unsupported reflection of expression-based index ix_flow_run__coalesce_start_time_expected_start_time_desc
    next(self.gen)

tests/test_tasks.py::test_get_inference_result_fails
  /Users/alessandro.lollo/.pyenv/versions/3.10.9/lib/python3.10/contextlib.py:142: SAWarning: Skipped unsupported reflection of expression-based index ix_flow_run__coalesce_start_time_expected_start_time_asc
    next(self.gen)

-- Docs: <https://docs.pytest.org/en/stable/how-to/capture-warnings.html>

These seems to be related to SQLAlchemy 2.0. However, in my environment I have

prefect==2.7.11

and

sqlalchemy==1.4.46

Anyone experiencing the same? Is this something I should be worried about? 😅

Hi everyone, I am currently trying to setup an ECS agent with aws. However, the task seems to stop and exit with the error

essential container in task exited

. The prefect ecs agent also does not appear in the UI. I have been able to start the ECS agent locally, but am unable to start it through aws. Another thing to mention is that when I try to add the logConfiguration to try to see what could be going wrong with the service it will give me an error of

ResourceInitializationError: failed to validate logger args: : signal: killed

. I’ve double checked with devOps that the IAM roles and network configurations should be correct too. Any ideas on how to debug this or why this is happening?

Hi, is there a way to pull an image from AWS ECR and run the image? the only one I found is to pull from Docker server which doesn’t apply to my case.

I am looking for a comparison with Mage. I read a comparison that is really favourable to Mage with respect to Prefect and I would like to read an opposite one 😄

anyone ever get the 'Can't connect to Server API at http://127.0.0.1:4200/api. Check that it's accessible from your machine.'? After developing locally, I'm setting up a remote host for actual deployments and unable to get the server configured correctly it seems. I'm starting the server with:

prefect server start --host=0.0.0.0

Hi, I am trying to register a simple flow in a server that I run in a docker locally. I first built a docker image on top of

prefecthq/prefect:latest

image, which also copied my python script

/src/test.py

and ran

prefect backend server

. Then in the

docker-compose.yaml

generated by

prefect server config

, I added a service

client

whose command is

python /src/test/py

. After running

docker-compose up

, I got the following error from the

client

service:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f24823ea0d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

Meanwhile, I can access UI and on UI it’s connected to Apollo endpoint successfully. It seems like

client

fails to connect to server at localhost:4200?

Hey Guys I want to setup a flow dependency for my task, basically I am running a flow, inside which I have a flow which downloads some data and then after this data has been downloaded I want to run some tasks, but when I try to do so the download flow runs successfully but the tasks after that end up failing, any idea why could that be happening and how can I fix it?