Hey, all! I'm Luis and work at Intuitive Care, a Brazilian healthtech. We are considering using prefect opensource and i was is there a reference implementation for prefect on AWS using ECS and isolating server, database and workers?

Hi all! Any plans for a collection/task set that works with Tableau? Thinking of simple extract refresh of some datasets times against a series of other Prefect flows. If not, anyone have some logic or examples they've used to do this before?

Hi! I have a problem running a mapped "create_flow_run" on Prefect 1.0. This used to work for us and is working on a registered flow but when running local python, the task ends with status mapped and never executes the actual entities of the map. log from Prefect UI from a run on our fargate machine

16 September 2022,01:33:41 	prefect.CloudTaskRunner	INFO	Task 'create_ingestion_params': Starting task run...
16 September 2022,01:33:41 	prefect.CloudTaskRunner	INFO	Task 'create_ingestion_params': Finished task run for task with final state: 'Success'
16 September 2022,01:33:41 	prefect.CloudTaskRunner	INFO	Task 'create_flow_run': Starting task run...
16 September 2022,01:33:41 	prefect.CloudTaskRunner	INFO	Task 'create_flow_run': Finished task run for task with final state: 'Mapped'
16 September 2022,01:33:42 	prefect.CloudTaskRunner	INFO	Task 'create_flow_run[0]': Starting task run...
16 September 2022,01:33:42 	prefect.create_flow_run[0]	INFO	Creating flow run 'fierce-flounder-Tome Ingestion' for flow 'Tome Ingestion'...
16 September 2022,01:33:42 	prefect.create_flow_run[0]	INFO	Created flow run 'fierce-flounder-Tome Ingestion': <https://cloud.prefect.io/tome/flow-run/8d5af013-baa8-4830-be75-7d212a4f3e74>
16 September 2022,01:33:42 	prefect.CloudTaskRunner	INFO	Task 'create_flow_run[0]': Finished task run for task with final state: 'Success'
16 September 2022,01:33:43 	prefect.CloudTaskRunner	INFO	Task 'create_flow_run[1]': Starting task run...
16 September 2022,01:33:43 	prefect.create_flow_run[1]	INFO	Creating flow run 'fierce-flounder-Tome Ingestion' for flow 'Tome Ingestion'...
16 September 2022,01:33:43 	prefect.create_flow_run[1]	INFO	Created flow run 'fierce-flounder-Tome Ingestion': <https://cloud.prefect.io/tome/flow-run/7729a5c3-5fb3-474f-bfbc-7d2fbce841d8>
16 September 2022,01:33:44 	prefect.CloudTaskRunner	INFO	Task 'create_flow_run[1]': Finished task run for task with final state: 'Success'

log from my local python execution

[2022-09-19 23:44:08+0000] INFO - prefect.CloudTaskRunner | Task 'create_ingestion_params': Starting task run...
INFO     [task_runner.py:241] Task 'create_ingestion_params': Starting task run...
[2022-09-19 23:44:14+0000] INFO - prefect.CloudTaskRunner | Task 'create_ingestion_params': Finished task run for task with final state: 'Success'
INFO     [task_runner.py:335] Task 'create_ingestion_params': Finished task run for task with final state: 'Success'
[2022-09-19 23:44:14+0000] INFO - prefect.CloudTaskRunner | Task 'create_flow_run': Starting task run...
INFO     [task_runner.py:241] Task 'create_flow_run': Starting task run...
[2022-09-19 23:44:14+0000] INFO - prefect.CloudTaskRunner | Task 'create_flow_run': Finished task run for task with final state: 'Mapped'
INFO     [task_runner.py:335] Task 'create_flow_run': Finished task run for task with final state: 'Mapped'
[2022-09-19 23:44:15+0000] INFO - prefect.CloudTaskRunner | Task 'get_task_run_result': Starting task run...
INFO     [task_runner.py:241] Task 'get_task_run_result': Starting task run...

Is there anything comparing Prefect 1.0 vs 2.0 concepts? Are 2.0 work queues roughly equivalent to 1.0 projects?

As a related, but separate, question… What are the different ways someone can setup dev, uat, and prod environments?

In prefect 1 we could use checkpoints to persist results. Is that replaced with caching in V2?

How come there are “No Upcoming Runs” in the queue, when I’ve limited the “Flow Run Concurrency” to e.g.

10

but are plenty, when it is left to Unlimited? When there are no visible scheduled jobs, my agent polling that queue would not retrieve any jobs either 😕 As soon as I change it to

Unlimited

, it comes to life and starts running all the late jobs like nothing had happened. I’m using Prefect Cloud v2

Q,1

Hi! Has anybody ran Prefect with ECS ? I seem to not be able to change the size of the CPU/memory, within the prefect cloud GUI. I always get an error like: An error occurred (InvalidParameterException) when calling the RunTask operation: Invalid 'cpu' setting for task.

How can I see which agents are running? either in Prefect Cloud or through the agent environment

Hi! My flow runs all tasks but does not send the Flow Succesful status before going to 'exited', the agent log in the environment looks like this:

08:57:40.650 | INFO    | Task run 'task_name-6b80d8b6-24' - Finished in state Completed()
08:57:59.061 | INFO    | prefect.infrastructure.docker-container - Docker container 'adaptable-quoll' has status 'exited'

And the Prefect Cloud log like this:

Created task run 'task_name-6b80d8b6-24' for task 'task_name'
10:57:39 AM
Executing 'task_name-6b80d8b6-24' immediately..

The flow run is stuck at 'running' in the Prefect Cloud UI. This is only the case for one of my flows, but the final tasks of the flow are the same tasks as for my other flows.

Hey all, am currently running Prefect 2.3.2 on a minikube cluster and while the agent connects to Orion just fine, it does not seem to be reading jobs from the queue. I have a pending job and the agent just doesn’t pick it up. Any ideas?

Hello, Is it possible to implement something like DateClock in Prefect 2? I was trying to implement custom Schedule, but getting validation error:

pydantic.error_wrappers.ValidationError: 9 validation errors for Deployment
schedule -> interval
  field required (type=value_error.missing)
schedule -> calendar_id
  extra fields not permitted (type=value_error.extra)
schedule -> run_times
  extra fields not permitted (type=value_error.extra)
schedule -> cron
  field required (type=value_error.missing)
schedule -> calendar_id
  extra fields not permitted (type=value_error.extra)
schedule -> run_times
  extra fields not permitted (type=value_error.extra)
schedule -> rrule
  field required (type=value_error.missing)
schedule -> calendar_id
  extra fields not permitted (type=value_error.extra)
schedule -> run_times
  extra fields not permitted (type=value_error.extra)

Heya! Probably a really stupid question, but how do I pass arguments to deployment runs via the CLI? I couldn’t figure out a way to do it yet. What I am looking for is something like this:

prefect deployment run flow/deployment paramA=1 paramB=2

Am I missing something super obvious here? 😅 Thanks!

Hi prefect friends, I haven’t checked in in a bit. Do we have

Result

equivalent in prefect 2.0? 🙂

Hi. Does RRuleSchedule support something like? I was able to deploy the flow, but don't see any scheduled flow runs.

RDATE:20220920T200000Z,20220921T133000Z

UI says: 'Every year' schedule:

Is the helm prefect-agent version 1.0.0 released?

Hi everyone, I am trying to run a simple task against a spark cluster using Prefect 1.2 and i get this error:

TypeError: cannot pickle '_thread.RLock' object

I can see the application being registered inside the spark cluster. Thanks!

Greetings friends; great product! I’m having a lot of fun diving in. I was looking into using an ECSTask infra block, and I observed that there’s no

conda

package for

prefect-aws

. I can just use

pip

to keep moving forward, but I was curious about the inconsistency. Other prefect packages are available.

Hi Team - In Blocks - Email Server Credentials. What needs to be put into the Smtp Type and smtp server for office 365?

Hi team! How come Docker is no longer a storage option? Is the local file a replacement for this? I remember reading in the docs that Docker was no longer required but it looks like a replacement is still needed for storage https://docs.prefect.io/api-ref/prefect/filesystems/#prefect.filesystems.GitHub

I'm currently looking at this example https://discourse.prefect.io/t/how-can-i-run-my-flow-in-a-docker-container/64 which looks pretty similar to what I have in 1.0, except I use Docker for storage and ECSRun as my run config. What would this look like in 2.0?

with Flow("data_pipeline_test_new_account", storage=STORAGE, run_config=RUN_CONFIG) as data_pipeline_test_new_account:

Hi Team - I'm currently attempting to login to prefect via the docker image (DockerFile). I'm having an issue where it is asking for the profile name. Initially, there was an issue with the workspace, but I added that in the same line as the login. I'm not sure how to resolve this, because there is not a parameter for profile name like there is for workspace. Is there another way to do this?

RUN /bin/sh -c "prefect cloud login -k $(cat ./api_key.txt) --workspace $(cat ./workspace.txt)"

Hi All, With prefect 2.4.0, prefect-dask 0.2.0 I am receiving Internal server error and these errors are random, sometime the flow works fine next time flow crashes with

internal server error

. Other errors that appears from agent logs are,

RunTime error: The connection pool was closed while 9 HTTP requests/responses were still in flight.

Hi - we have an API that’s trying to register and run flows programatically via Prefect 1 Cloud, using the Prefect Python lib. We need to do this against two different Tenants in Prefect Cloud. This leads to an issue with credentials - since Prefect’s credentials seemingly need to be stored globally either in an env var or in a file, trying to switch between tenant auths while concurrently publishing multiple flows (eg one to tenant 1, one to tenant 2) is leading to a race condition. We might be able to work around this via multiprocessing but is there a cleaner way to do it, eg some way to set auth per call rather than globally?

Hi all, while evaluating orchestration engines, I discovered prefect today. So I decided to give it a whirl, and the pip install failed on both Windows 10 and Mac OS 12.4. I'm running python 10.3.6 on both This is the Mac output after running "pip install --pre prefect" for Mac: × Running setup.py install for aiohttp did not run successfully. │ exit code: 1 ╰─> [511 lines of output] ********************** * Accellerated build * ********************** /Users/danhorne/Env/prefect/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead. warnings.warn(msg, warning_class) running install /Users/danhorne/Env/prefect/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build/lib.macosx-12-arm64-cpython-310 creating build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_ws.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/worker.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/multipart.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_response.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/signals.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client_ws.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/test_utils.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/tracing.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_exceptions.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_middlewares.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http_exceptions.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_app.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/streams.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_protocol.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/log.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_urldispatcher.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_request.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http_websocket.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client_proto.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/locks.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/__init__.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_runner.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_server.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/base_protocol.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/payload.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client_reqrep.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_log.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/resolver.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/formdata.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_routedef.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/connector.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client_exceptions.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/typedefs.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/hdrs.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_fileresponse.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http_writer.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/tcp_helpers.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/helpers.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/frozenlist.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http_parser.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/cookiejar.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/pytest_plugin.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/abc.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp running egg_info writing aiohttp.egg-info/PKG-INFO writing dependency_links to aiohttp.egg-info/dependency_links.txt writing requirements to aiohttp.egg-info/requires.txt writing top-level names to aiohttp.egg-info/top_level.txt reading manifest file 'aiohttp.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'aiohttp' anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '*.pyd' found anywhere in distribution warning: no previously-included files matching '*.so' found anywhere in distribution warning: no previously-included files matching '*.lib' found anywhere in distribution warning: no previously-included files matching '*.dll' found anywhere in distribution warning: no previously-included files matching '*.a' found anywhere in distribution warning: no previously-included files matching '*.obj' found anywhere in distribution warning: no previously-included files found matching 'aiohttp/*.html' no previously-included directories found matching 'docs/_build' adding license file 'LICENSE.txt' writing manifest file 'aiohttp.egg-info/SOURCES.txt' copying aiohttp/_cparser.pxd -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_find_header.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_find_header.h -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_find_header.pxd -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_frozenlist.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_frozenlist.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_headers.pxi -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_helpers.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_helpers.pyi -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_helpers.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_http_parser.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_http_parser.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_http_writer.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_http_writer.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_websocket.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_websocket.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/frozenlist.pyi -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/py.typed -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/signals.pyi -> build/lib.macosx-12-arm64-cpython-310/aiohttp running build_ext building 'aiohttp._websocket' extension creating build/temp.macosx-12-arm64-cpython-310 creating build/temp.macosx-12-arm64-cpython-310/aiohttp clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/danhorne/Env/prefect/include -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c aiohttp/_websocket.c -o build/temp.macosx-12-arm64-cpython-310/aiohttp/_websocket.o aiohttp/_websocket.c:2166:22: warning: '_PyUnicode_get_wstr_length' is deprecated [-Wdeprecated-declarations] (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : ^ /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython/unicodeobject.h:261:7: note: expanded from macro 'PyUnicode_GET_SIZE' PyUnicode_WSTR_LENGTH(op) : \ ^ /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython/unicodeobject.h:451:35: note: expanded from macro 'PyUnicode_WSTR_LENGTH' #define PyUnicode_WSTR_LENGTH(op) _PyUnicode_get_wstr_length((PyObject*)op) ^ /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython/unicodeobject.h:445:1: note: '_PyUnicode_get_wstr_length' has been explicitly marked deprecated here Py_DEPRECATED(3.3) ^ /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10/pyport.h:513:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((deprecated)) ^ aiohttp/_websocket.c:2166:22: warning: 'PyUnicode_AsUnicode' is deprecated [-Wdeprecated-declarations] (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : ... lots more messages ... aiohttp/_http_parser.c:23276:19: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value); ^ aiohttp/_http_parser.c:23276:17: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion] ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ aiohttp/_http_parser.c:23360:19: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ret = _PyGen_Send((PyGenObject*)yf, NULL); ^ aiohttp/_http_parser.c:23360:17: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion] ret = _PyGen_Send((PyGenObject*)yf, NULL); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 21 warnings and 5 errors generated. error: command '/usr/bin/clang' failed with exit code 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> aiohttp note: This is an issue with the package mentioned above, not pip. Has anyone else received these messages and resolved the problem? Thanks

Hi 👋 We have a GitHub block defined in the Prefect Cloud UI and it’s working. But the password is embedded inside the link repository like so: https://yuvital-render:xxxxx@github.com/yuvital/prefect.git Is there a way to make it work without embedding the password like this, or at least hide it in some way (for example like it’s done in the S3 Block)?

Suggestion to improve the tutorial at this page. For someone who is tinkering with Prefect and Docker containers for the first time it might not be obvious that the agent cannot be running with the ephemeral API. Specifically, here, the documentation only says to start an agent polling from the “test” queue. This works if the PREFECT_API_URL has been set before hand. It would be nice for the tutorial to include a note about setting this environment variable using the CLI or starting the agent with the --api flag set to the API endpoint.