Hey everyone! Does anyone know what are the resource requirements when deploying the prefect-server helm chart? I tried to deploy it on two Raspberry Pi 4s with 8GB RAM that are running k3s, and most of the pods are unable to start, which results in 0 deployments working. All the dead pods are giving the

standard_init_linux.go:228: exec user process caused: exec format error

error in logs though, so not 100% sure the issue's hardware-related. Any suggestions would be appreciated 🙏

Hi, in prefect 2.0 - is there a way to perform 'yields' - we have some (unfortunately) long running tasks where we stream some content, which takes a while (+30 min, very large zipped json files) - and I can't seem to figure out if I can yield partial result futures in my task incrementally and start processing them in down stream tasks. Currently, the work flow is: • Stream read big annoying file into small chunks, save them in S3 incrementally in a separate thread inside the task, and return list of all the chunked file URI's as single result (= single

prefectfuture[list[str]])

object) ◦ this is where I would like to yield • Run some down stream processing on each chunk ◦ Here we use future.result() to loop over and start each task - this works 100%, but if we could use yield in the first task, we could begin processing immediately Is something like this supported / going to be supported? I tried making it work with contextmanager and tasks, but ran into nothing but trouble

Is there a way to specify a default job template for kubernetes agent in the manifests? So I am looking at https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template - and I tried to use the

--job-template

on the

prefect agent kubernetes install

command in the hope that this would work but in fact it does not - do I need to modify the containers args which are

prefect agent kubernetes start

to use the flag?

how does everybody separate production vs development environments? i have separate projects for each, with separate labels and an agent in each env querying for the specified label. but yesterday, i was testing a big process in dev where i tried to run a large amount of flows, and it impacted my production environment, where nothing ran. should i be separating envs in another way?

How can we read log of a task or flow/subflow using Orion REST APIs? Also, Do we need to do any specific configuration to achieve it via REST APIs?

Hi Prefect Team. Yesterday early morning, two of our flows failed at the same time at 3:00AM PST with the same error. Both the flows ran fine when we restarted it in the morning at around 9AM. This is a really weird non-deterministic issue. We didn’t change or modify the secret SLACK_WEBHOOK_URL in the morning when we re-ran the flow but still it the tasks ran fine. We did root cause analysis but still couldn’t find out why the flows failed. Can you please take a look at it and help us out over here? 1. https://cloud.prefect.io/immersa-prod/flow-run/0ae8fb55-fa89-4a1b-aa86-9d122f490751 2. https://cloud.prefect.io/immersa-prod/flow-run/7c59423a-f7b7-4438-959b-9259058d22b1

Exception raised while calling state handlers: ClientError([{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 137, in get
    value = secrets[self.name]
KeyError: 'SLACK_WEBHOOK_URL'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 64, in call_runner_target_handlers
    new_state = super().call_runner_target_handlers(
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
    new_state = handler(self.task, old_state, new_state) or new_state
  File "/usr/local/lib/python3.9/site-packages/toolz/functoolz.py", line 306, in __call__
    return self._partial(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/notifications/notifications.py", line 299, in slack_notifier
    str, prefect.client.Secret(webhook_secret or "SLACK_WEBHOOK_URL").get()
  File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 161, in get
    raise exc
  File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 145, in get
    result = self.client.graphql(
  File "
/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 570, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
03:02:29

I

Hi all, I’m trying to find an similar class to

PostgresFetch

in Prefect 2.0. Any pointers?

Hi all, I am experimenting with tasks and subflows in prefect 2.0. While in tasks I can use the wait_for keyword to enforce task dependency, the same keyword does not work for subflows. I did not found much on the subflows docs, but it seems that in prefect 1.0 dependency between subflows was possible. Is such a feature available also in prefect 2 ?

Where does the Docker storage inherit its credentials from? I assume boto3, but I'm getting a weird behavior on Github Actions. Using the manual docker build and push with the ECR Login action (https://github.com/aws-actions/amazon-ecr-login#usage), I can push my image just fine to ECR. However, when I run this with flow.register() it fails on no authorization.

meme Friday going on in #random and it is hothothot 🔥

I’m bumping into all sorts of issues with Deployments on 2.0, just wanted to check if these are known problems or something awry with my config?

Hey guys seems our dev tenant (id 023b51a2-d0f4-4b3c-9f36-6eb176e50e46) can only accept 10 users (notifies me about a '10-user limit' when inviting more users). Is this something you can help increase? (A 100-user limit should be nice : )

Hi all, here is what I’m trying to do with 1.0 version. I have a database table

jobs

like following:

id | name                 | query                                | template
-----------------------------------------------------------------------------------------
1  | user_count_query     | select count(*) as cnt from users    | user count is {{cnt}}
2  | projects_count_query | select count(*) as cnt from projects.| project count is {{cnt}}

I would like to 1. Use

fetch_jobs = PostgresFetch

task to get a list of all jobs in the table above 2. Fan out the results of

fetch_jobs

to new task say

run_job_query

which will for each row grab value of

query

column and execute another

PostgresFetch(query=query)

In a pseudo flow it would look something like:

fetch = PostgresFetch()

@task
def send_to_slack():
  pass

with prefect.Flow() as flow:
    jobs = fetch(query="select query from jobs"), fetch="many")
    for job in jobs: 
       query_result = fetch(query=job["query"])
       send_to_slack(query_result)

The above code does not obviously work because Tasks are not iterable. Mapping won’t work too, since I cannot call another task in a task Any help appreciated.

Does Orion support canceling/restarting runs?

Hey, new user here trying to get my head around Prefect. I'm using Orion. Can I return a value from a subflow and use it? I'm trying to do it by assigning the function call but the subflow is returning a State rather than the return value of the subflow function. Any ideas?

This message was deleted.

I'm getting the following error when registering a flow:

Traceback (most recent call last):
  File "/app/biomarkers/extract_markers_image.binary.runfiles/biomarkers/biomarkers/dockers/extract_markers.py", line 2, in <module>
    from biomarkers.batch_processing import orchestrator
  File "/app/biomarkers/extract_markers_image.binary.runfiles/biomarkers/biomarkers/batch_processing/orchestrator.py", line 130, in <module>
    seq_markers_extraction_flow.register("neuralight-research")
  File "/usr/local/lib/python3.9/site-packages/prefect/core/flow.py", line 1708, in register
    registered_flow = client.register(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 839, in register
    project = self.graphql(query_project).data.project  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 452, in graphql
    result = <http://self.post|self.post>(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 407, in post
    response = self._request(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 649, in _request
    raise AuthorizationError(
prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.

Any idea what can cause it? I've already authenticated using my API key

https://prefect-community.slack.com/archives/C036FRC4KMW/p1650831951700959 The Pizza Patrol strikes again! If you want to feed your open source meetup, send me a message here on Slack or fill out our form on our Community page!

How do I get

CronSchedule("0 11 * * *")

to fire at 11 and not 12?

HI everyone!. I am newbie here and have trouble signin in to the prefect.io cloud from chrome(100.0.4896.127). Stuck at loader but cannot open the dashboard. Yet from other devices(mobile same chrome version) is opening. Tried to delete cookies but not working so stuck here (lost 2 days). Any suggestions?

Hello. Is there a way to add a taint to a KubernetesRun without creating a custom job_template.yaml?

Morning 👋 Is there an equivalent of DbtShellTask in Orion (Prefect 2.0) ?

is there a way to register different flows so that they are run with different environment variables? something like

Flow.register("foo", extra_env_variables={"ENV": "foo"})
Flow.register("bar", extra_env_variables={"ENV": "bar"})

For some reason, from today onwards in our Orion server, every new execute is going as scheduled by default, not running the flow immediately. Please shed some light.

Is this heartbeat configuration only applicable to Prefect cloud setups? https://docs.prefect.io/orchestration/concepts/services.html#heartbeat-configuration I have a (expected) long running job on a Prefect server setup trying to not have the zombie killer terminate the job:

No heartbeat detected from the remote task; marking the run as failed.

This message was deleted.

Hi, we're looking into ways to reduce latency between Flow Run submission and getting results back. At the moment we're using KubernetesAgent which spawns k8s Jobs for each flow run. Job initialization is quite slow in our case due to image size. Question: if we setup permanent Dask cluster with appropriate image and set executor to DaskExecutor - would we be able to skip Job initialization step? i.e. is it Agent who sends specific commands to Dask cluster or k8s Job cannot be avoided?

Hey team! I am trying to deploy prefect-server on gke At this point--> vim ~/.prefect/config.toml I entered apollo IP and pressing esc-> :wq or esc-->:x ... Then I am getting the following message. Any guide on how to deal with it?

Hey.. I am registering my task on prefect-server to demo project. The task has spark and sparks context. The flow.run() is successful but flow.register() gives following error RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063. Any guide on this spark context setting.

Hello, Have anyone experienced problems when using:

prefect.engine.signals.LOOP

, where the task is suddenly is missing the context on retries. e.g:

index = prefect.context.get('task_loop_result', {}).get('index', 0)

See more code in the thread 🧵