Can someone explain how to use the GitHub block? When creating a block, we will definitely have to execute the get_directory method with the name of the folder where the flow is located, otherwise we will receive an error that the file was not found. Flows are launched via KubernetesJob.

I am seeing schedule gaps in a deployment that is supposed to run every hour 5 minutes past the hour. Is/has anyone experienced anything similar? I redployed and it reduced it, but there was still a 4 hour gap yesterday afternoon.

Hi again. In prefect 1.0 we used LocalDaskExecutor with parameters LocalDaskExecutor(scheduler="threads", num_workers=4). There may already be a recipe out there for using DaskTaskRunner for this.

Hello! I use the

ECSTask

inside

prefect-aws

, is there a way to configure AWS CloudWatch log retention?

I’m running Prefect 2.0 Flows over an AWS ECS stack (Fargate and EC2). The relevant infrastructure blocks are created by specifying a

task_definition_arn

corresponding to the desired infrastructure. Occasionally our flows fail with the “too many concurrent attempts”

ClientException

mentioned in this thread. https://discourse.prefect.io/t/i-am-using-ecsagent-and-sometimes-i-get-an-error-that-sa[…]empts-to-create-a-new-revision-of-the-specified-family/100. I have two questions: 1. Is it really necessary to register and de-register a task definition for every flow run? Can Prefect not just use the stable task definition? 2. If yes, is this agent-based solution for Prefect 1 still the recommended solution for Prefect 2? https://docs-v1.prefect.io/orchestration/agents/ecs.html#throttling-errors-on-flow-submission. That is, will the Prefect Agent recognize and use the

AWS_RETRY_MODD

and

AWS_MAX_ATTEMPTS

environment parameters?

If I use prefect cloud, does that mean that I don't need to set up a database then?

Is there a way I can get the deployment name from prefect.context?

Hello I'm trying to use Docker Registry block to authenticate with a Amazon ECR instance, but the credentials AWS provides only last 12 hours. Has anyone solved this? How do I get a Docker Registry block to work with ECR permanently?

In regards to prefect workers: How would you go about editing the base job template for a kubernetes work pool through the CLI? I am trying to do the following:

"job_configuration": {
    "env": "{{ env }}",
    "name": "{{ name }}",
    "labels": "{{ labels }}",
    "command": "{{ command }}",
    "namespace": "{{ namespace }}",
    "job_manifest": {
      "kind": "Job",
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "env": "{{ env }}",
                "args": "{{ command }}",
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "{{ image_pull_policy }}"
              }
            ],
            "completions": 1,
            "parallelism": 1,
            "restartPolicy": "Never",
            "imagePullSecrets": [
              {
                "name": "git-registry-token"
              }
            ],
            "serviceAccountName": "{{ service_account_name }}"
          }
        },
        "ttlSecondsAfterFinished": "{{ finished_job_ttl }}"
      },
      "metadata": {
        "labels": "{{ labels }}",
        "namespace": "{{ namespace }}",
        "generateName": "{{ name }}-"
      },
      "apiVersion": "batch/v1"
    },
    "stream_output": "{{ stream_output }}",
    "cluster_config": "{{ cluster_config }}",
    "job_watch_timeout_seconds": "{{ job_watch_timeout_seconds }}",
    "pod_watch_timeout_seconds": "{{ pod_watch_timeout_seconds }}"
  }

Here I am adding the imagePullSecrets attribute so that the kubernetes job can pull the custom built runner image; however, I have to go into the UI and add this through the advanced config. Is there any clean way to do this via the CLI, or by editing the deployment.yaml/prefect.yaml? I thought about creating my own worker with its own base job template by subclassing the default kubenetes worker, but I would rather not add unnecessary complexity to my project! Thanks so much!

Hi, is there some documentation on the latest (2.10.16) workflows on how to set up prefect? I’m looking specifically for details on the docker-git deployment config recipe. Thanks!

Hi! I've been running Prefect in a local docker, and I want to switch to the cloud version. How do I do it? I've logged in via the CLI, but the runs keep trying to access my old local server.

Hi! How to limit the number of simultaneously running tasks using DaskRunner?

Hello everyone, I didn't find it anywhere, but does the

task.map

keep order ? (Will the list of generated futures have the same order as the mapped over argument ?)

Hello, Trying to understand why when I run

prefect.deployments.run_deployment

from a flow, it shows the new flows that are being triggered by this function as subflows in the UI? Shouldn't they be

regular

flows instead of

subflows

? That's what I understood from the docs.

Maybe something to be aware of: as of v2.10.15 the Prefect docker image for Python 3.9 seems to be based on Debian Linux 12 (bookworm). The

adduser

command in Debian 12 works slightly different than the one in Debian 11, in the fact that on Debian 12 a home folder is not implicitly created any more for system users. Because of this upgrade, we have seen our Prefect jobs that use

persist_result=True

crash with this error:

12:07:48.020 | ERROR   | Flow run 'caped-dingo' - Crash detected! Execution was interrupted by an unexpected exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/pathlib.py", line 1323, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/nonexistent/.prefect/storage'

We fixed by adding

--home=/home/<my-user>

option to the

adduser

command in our Dockerfile.

Hi, does anyone know how to get an agent to run a specific docker image from a docker repo where the docker image is defined in the deployment? Hence is the agent able to start up different docker images?

Hello, I'm running into an issue where a flow is executing 2 flow runs. I noticed this bug has been documented here I'm using Prefect Cloud 2.10.11 with the k8s agent. No retries are set on the flow. There is an automation set to send a slack notification upon failure or crash state. The flow includes subflow runs The flow is auto-scheduled Attached are the state transitions for the flow run.

hi

Is there currently a block capable of configuring access to an Azure Keyvault from within K8s job infrastructure ?

Hi Team, Im using Prefect to coordinate some KDB calls via PyKX. When trying to deploy however I keep running into the issue:

Script at './Script/ibes_pykx.py' encountered an exception: ValueError('signal only works in main thread')

. The code is trying to establish a connection and then run these processes on the connection itself. Any advice would be a HUGE help. Source code is below:

def setup():
    token = subprocess.Popen('\\\\ccl\\data\\extlib\\KDBSecToken.exe', stdout=subprocess.PIPE).communicate()[0].decode('utf-8')
    masterConn = kx.SyncQConnection( host='v-kdbr-01', port=5000, username='cleung', password=token, timeout=3.0 ) 
    user = masterConn('.z.u') 
    print('User = ' + user.py())
    port = masterConn('getProcessClient[`prefect_testing_base;`pykx_test]')
    print('Port = ' + str(port.py())) 
    masterConn.close()
    global conn
    conn = kx.SyncQConnection( host='v-kdbr-01', port=port.py(), username='cleung', password=token) 
    user = conn('.z.u') 
    print('User = ' + user.py()) 

@flow()
def ibes_pykx():
    setup()
    logger = get_run_logger()
    <http://logger.info|logger.info>("Initializing initial arguments")
    dailyDates = conn('dailyDates: .dt.drb[.dt.shiftdateb[exec max date from QModelD;-3];  exec max date from QModelD];')
    logFile = conn('inLogFile:`')

   

    return finishIBES_result

    

if __name__ == "__main__":
    ibes_pykx()

bump

Hello, how can I set up a deployment of a flow that assigns the runs to a specific work queue with a desired concurrency limit using the Python SDK? I'm not seeing any examples/documentation of that in the docs which makes me wonder if it's even possible. Also a question about concurrency limits in general -- the docs say that it is a way to restrict concurrent runs of matching flows. What does "matching" mean here? That is basically limiting concurrency of multiple runs of the same flow? I hope that's what it means, because I'm interested in limiting the number of concurrent runs from a particular flow, (in this case at the work queue level). But I don't mind runs of different flows running at the same time.

Hello, is there a way to check if a key is found in a parameter? each time I try to do it like a dictionary, I get the error that parameter doesn't have the dictionary attributes.

Is it possible to start a prefect server within a python .py without calling subprocess.popen? I would like to avoid having a shell script / calling shell commands

Hi, We have deployed a prefect server and prefect agent using helm chart in Kubernetes environment. We are using the default Prefect image for both server and agent. We have a python dependency package that needs to be installed before the deployment. we tried specifying the dependency in EXTRA_PIP_PACKAGES env variable. but this is not being respected as we don't see any logs that it is trying to install the package when the agent spins up extraEnvVars: - name: EXTRA_PIP_PACKAGES value: "minio" Whether helm chart is not using the entryPoint.sh which ignores the EXTRA_PIP_PACKAGES variable set in values.yaml?

Hi there, Could anyone explain the best way to set up a Kubernetes Job manifest? I mean is it better to make "Base Job Manifest" with the necessary options or use a default one with "Customizations"?

Hi, I am facing the below error. How can I fix this ?

Are the task cache and results the same concepts?

Hi guys, with the transition to Prefect2 we have lately running into some problems with prefect-gcp. While trying to query from BigQuery and store in a dataframe, BigQuery states "success" long before this is handled in Prefect. As a result, it can take up to 2 hours before the next task is started. Has anyone else ran into these problems? And is there a fix to this?

Hi, a flow for downloading geospatial data seems to timeout after an unspecified amount of time but without having set any parameters in the @flow decorator (like timeout_seconds). Can you explain why? Has it ever occurred to you? We tested the flow on a smaller dataset (an area) and the code terminates without a problem, we believe that the area in question, which is much larger, may somehow cause the flow to timeout, but we don't understand why and, possibly, what to set on prefect to prevent this. Here the error: