OK, I'm trying to get Prefect Cloud v2 working .... and learning as I go .... I have a script that is working perfectly (finally), now created a deployment with an "every 30 mins schedule" .... but does that mean that every workflow in the workspace runs on that schedule?

In one of my flows, I'm using a concurrency limit of 1 on a particular resource. This is working, but I've noticed that the relevant task is only being executed once every 30 seconds or so. The rest of the time my flow is just sitting idle. If I inspect the concurrency-limit via the CLI while my flow is running, "Active Task Run IDs" is empty most of the time. Here is a gist that demonstrates this. With a concurrency of N, you get batches of N tasks running with a 30 second delay between each batch. My case is just with N == 1. Is there some kind of polling interval built into prefect, such that concurrency limits are only checked once every 30 seconds? If so, is there any way to reduce this delay? Is this behaviour documented anywhere?

Hello! Early this morning Late last night we got another "Pending" alert misfire from Cloud's automation feature to our Slack workspace.

Flow stuck in PENDING state. Please fix me!

Flow run <redacted>/accurate-leech entered state

Completed

at 2023-03-28T220040.649242+00:00.

Flow ID: 82fbf550-6938-4f85-83f8-e279fd4ac048

Flow run ID: cfe1b7d8-e858-4399-983a-fa1dfffa4d0d

Flow run URL: https://app.prefect.cloud/account/c1397d5f-b9f3-49e8-abb6-bce7d7b1412e/workspace/32dfe242-315b-4405-b06d-8b6308d6b631/flow-runs/flow-run/cfe1b7d8-e858-4399-983a-fa1dfffa4d0d

State message: All states completed.

• scheduled to run at 220000 UTC • actual start at 220003 UTC • completed at 220040 UTC Why did we get a flow stuck in PENDING state around the time of flow's completion time? Even more curious is the fact we have set the alert to trigger when a flow has remained in pending state for 15 minutes 🤷 Automation's Trigger settings • Trigger Type: Flow Run State • Flows: All flows • Flow Run Tags: All tags • Flow Run: Stays in = (pending) • For: 15 Minutes Automation in question: https://app.prefect.cloud/account/c1397d5f-b9f3-49e8-abb6-bce7d7b1412e/workspace/32df[…]/automations/automation/0284734c-53f9-4a72-8351-fbe3ad50d19b

Hi! When we try to run code that authorizes with the Kubernetes Python Client in our cluster, using exec to generate a token with a specific utility, we get a 403 error very often. But when the same code is only run in a manually started Pod, we don't have any issues at all. So the big question is, what's the difference between running code with a Prefect flow and just running it in a Pod? And what can we do about the issue in general?

Hey all, I would like to trigger a prefect flow whenever a new file arrives in an S3 bucket. 1. What is the best approach to do so?

So far, I set up an AWS S3 bucket and a related AWS Lambda as well as a VPC that provides internet access to trigger the prefect flow via a request to the GRAPHQL API. 2. Is the approach

S3->Lambda->GRAPHQL API call

appropriate?

However, I get the following error (ACCOUNT_ID and WORSKPACE_ID as obtained from a prefect agent):

[ERROR] ConnectionError: HTTPSConnectionPool(host='api.prefect.cloud', port=443): Max retries exceeded with url: /api/accounts/{ACCOUNT_ID}/workspaces/{WORKSPACE_ID} (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f04a5873310>: Failed to establish a new connection: [Errno 110] Connection timed out'))

3. Any tips what could cause the error?

Follow-Up Question: I successfully triggered a flow via an API call 🙂 However, it remains in

PENDING

. I successfully submitted it with

deployment_id

, but it still remains in

PENDING

😕 I tried to submit it with

work_pool

but that does not seem to be a valid request? I read the docs but it is not clear to me, which role

work_pool

,

deployment

and

work queue

have 🤔 PS: The successful run was triggered in UI, without setting anything different than the flow defaults.

I tried to use the result_storage_key but got a

TypeError: task() got an unexpected keyword argument 'result_storage_key'

🤔 Are the docs outdated? Do I miss something?

Is there a way to return persisted results via the PREFECT API? I mean the actual results, not meta data 🙂

Hi, I have created a deployment flow retrieving currency rates in prefect 2. The flow takes several parameters including a date range to retrieve in the form of

start_date

and

end_date

. The default is set to today's date using the function

datetime.today().strftime("%Y-%m-%d")

. When I create a deployment, the above function executes and sets the date as today's date, e.g.

03-04-2023

. The problem is that this date will be set all the time every time I run the flow tomorrow, next week and next month. Is there any functionality in prefect where I could specify a parameter e.g. prefect.today() which would set the current date each time the flow is executed. Flow params set in deployment

{

"symbols": [

"USD",

"EUR",

"GBP",

"CHF",

"PLN",

"DKK",

"COP",

"CZK",

"SEK",

"NOK",

"ISK"

],

"currency": "USD",

"end_date": "2023-03-24",

"if_exists": "replace",

"start_date": "2023-03-24",

"databricks_table": "exchange_rates",

}

Hi guys, I'm a new user of prefect cloud. I've been running a server for more than a week now. Everything works fine and suddenly, I encounter this error. I search through some pages but there're no answers. If anyone have been through this or have any clues on how to solve it, please leave me a comment. Thank you all!!! ❤️

Hello, I am trying to a create a api_key for the existing service account. I am using this api call https://api.prefect.cloud/api/users/{id}/api_keys. {id} is the service_account_id. But I am getting the 404 error

Please help me, if you have any clues with this

Is it possible to get

flow_run.name

from inside a task? From inside a flow this works:

from prefect.context import get_run_context
get_run_context().flow_run.name

but inside a task, it throws

AttributeError: 'TaskRunContext' object has no attribute 'flow_run'

Hi, I am migrating to Prefect 2, and I am facing some troubles there. What the job does is: 1- Downloads a jar from somewhere 2- Uploads it to databricks file system (DBFS) through Databricks Rest API 2.0 3- Use it on databricks job (trough

/job/update

API) 4- Run a job The problem I am facing is on step 2, when I call

Add Block

Endpoint (through async method

execute_endpoint

), when I got a Time Out (no matter which timeout I set). Can you show me an sample of how can this get done with Prefect 2??? thanks in advance

Can someone from Prefect Team reach out to me (amit.mukherjee@nih.gov). I have sent a few emails to get Enterprise Prefect Version for our org. but have not received a response.

Hello,

I am trying to create a new account role in prefect cloud using https://app.prefect.cloud/api/docs#tag/Account-Roles/operation/create_account_role_api_accounts__account_id__account_roles__post. However, I am getting this message as response "detail": "\n Marvin says: Oh good. You're creating errors already.\n I know you're excited about Cloud 2.0 but could you be a little more careful? Or not.\n I'll just be here. Brain the size of a planet and they've got me cleaning up after you.\n Shoot me an email with subject line '418' at marvin@prefect.io for some swag.\n". Can some one from @Prefect look at this?

Hi, I am facing some troubles with a deployment with Prefect 2. The deployment succeed and the flow is on the API, but when I try to run it fails because the agent can’t download de flow from S3 storage I set up before (which path has been set to a folder where the flow file has been uploaded at the moment that the deployment happened). The error is FileNotFoundException. Can you send me an easy example that I can reproduce with S3 location (not cli command, but Deployment.build_from_flow line)? Thanks in advance. Regards

I'm playing with a free account on Prefect Cloud. I put a very simple test job up and they're both "late" to run, and it looks like my work pool is stuck in

unhealthy

. is there anything I can do to fix this, or is it just because I have a very new / unpaid account?

Hey folks, I'm going through the "getting started" with prefect cloud and I can't seem to be able to see any logs on the Cloud UI. It does show my flow runs, but logs are empty, no matter what. I used

log_prints=True

, as well as

get_run_logger

, and neither case worked. What am I missing? for the record, this is my .py module:

from prefect import flow, get_run_logger


@flow(log_prints=True)
def hi():
    logger = get_run_logger()
    logger.warning("Hit from Prefect! 😅")


if __name__ == "__main__":
    hi()

has anybody seen this error before?

cannot deploy project with work pool of type 'prefect-agent'

(main) $ prefect deploy flows/modeling.py:foo -n test_deployment -p default-agent-pool

17:00:16.247 | WARNING | ray._private.worker - Failed to set SIGTERM handler, processes mightnot be cleaned up properly on exit.
Cannot deploy project with work pool of type 'prefect-agent'.

when i start the prefect worker helm chart i get:

│ prefect.exceptions.PrefectHTTPStatusError: Client error '405 Method Not Allowed' for url '<http://prefect-server.prefect:4200/api/work_pools/>'                                           │

is this a known problem? and if so is there a way to correct it? this is with the latest version of the server and work helm charts

Hi, I have a flow that runs a databricks job, once it has finished, runs a shell command by calling

shell_run_command

task. When this task is called, I have the following error:

Encountered exception during execution:
Traceback (most recent call last):
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/engine.py", line 669, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 181, in aresult
    return await asyncio.wrap_future(self.future)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 194, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "flows/flows/landbaron.py", line 149, in generate_lb_sites_flow
    ps_details_timestamp = shell_run_command(command=get_ps_details_timestamp_command(export_version))
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/tasks.py", line 491, in __call__
    mapped=False,
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/engine.py", line 972, in enter_task_run_engine
    return from_sync.wait_for_call_in_loop_thread(begin_run)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/api.py", line 137, in wait_for_call_in_loop_thread
    return call.result()
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/engine.py", line 1137, in get_task_call_return_value
    return await future._result()
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/futures.py", line 241, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/engine.py", line 1551, in orchestrate_task_run
    result = await call.aresult()
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 181, in aresult
    return await asyncio.wrap_future(self.future)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect_shell/commands.py", line 98, in shell_run_command
    shell_command, env=current_env, cwd=cwd
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/_core/_subprocesses.py", line 135, in open_process
    start_new_session=start_new_session,
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 1112, in open_process
    start_new_session=start_new_session,
  File "/usr/lib/python3.7/asyncio/subprocess.py", line 217, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 1544, in subprocess_exec
    bufsize, **kwargs)
  File "/usr/lib/python3.7/asyncio/unix_events.py", line 193, in _make_subprocess_transport
    self._child_watcher_callback, transp)
  File "/usr/lib/python3.7/asyncio/unix_events.py", line 941, in add_child_handler
    "Cannot add child handler, "
RuntimeError: Cannot add child handler, the child watcher does not have a loop attached

Hi. I have found out that when I start a worker (with Prefect V2), after random time it get stuck, so I need to restart it again. Today happened to me at least 5 times

I'm trying to have my flow use a

ray.JobSubmissionClient

. how do I install

ray

(or pip packages more generally?) I tried a variety of things: 1. importing

prefect.settings

in my flow directly and setting `prefect.settings.EXTRA_PIP_PACKAGES = ["list", "of", "packages"] 2. adding it to the

requires

field of

push

and

pull

in my

prefect.yaml

3. adding

EXTRA_PIP_PACKAGES: ray

under

job_variables

in the

deployment.yaml

none of these worked. what is the recommended way to install packages for flows without burning a whole new image every time

Hello @Prefect, I am trying to create the service using. I am using python requests and running it in dokcer. I am getting this error. Could you please throw some light on this? :HTTPSConnectionPool(host='api.prefect.cloud', port=443): Max retries exceeded with url: /api/accounts/{account_id}/bots/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL : CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

Hi. I am running a flow with Prefect V2, and I am not able to see the logs until the flow has finished. Is that a known issue?