Hey I have been using prefect for a bit now on AKS in Azure

Question

Hey I have been using prefect for a bit now on AKS in Azure for hosting our agents On the whole this seems to work well it creates K8s jobs for our flow runs but regardless of how we trigger them it only ever seems to to run the amount of concurrent tasks in the flow to be the amount of virtual or physical CPUs on the cluster I was under the impression if we set the scheduler to threads and the num workers to x2 4 the number of CPUs the flow tasks would look to create that amount of threads to run the flow I am comfortable we can scale these as we are simply submitting jobs to a databricks cluster so we have no CPU bound work at all happening on the cluster it is purely IO monitoring Any help would be appreciated I have attached a sample of how we trigger our flow in the thread Thanks in advance for any help which could be offered

Michael Law · Answer

```def run self flow Flow scheduler = threads num workers = 8 env = DATABRICKS CLUSTER ID self cluster id MOUNT self mount CODEMOUNT self codeMount ENVIRONMENT self environment RELEASE VERSION self release version APP IMAGE self kubernetes job image AZURE STORAGE CONNECTION STRING self storage connection PREFECT CONTEXT SECRETS DATABRICKS CONNECTION STRING f self databricks connection PREFECT PROJECT self prefect project DATABRICKS INSTANCE POOL ID self instance pool id SQL SERVER JDBC CONNECTION STRING self sql jdbc connection string flow executor = LocalDaskExecutor scheduler = scheduler num workers = num workers DEBUG is not set on the K8 cluster if os environ DEBUG == 1 flow run elif os environ DEBUG == 2 flow visualize else flow run config = KubernetesRun image=self kubernetes job image env=env flow storage = Azure container= fdpflows connection string=self storage connection ```

Anna Geller · Answer

How did you assess how many threads and processes your flow run was using One option that may help here is attaching the executor directly to your Flow object when it gets created ```with Flow your flow executor = LocalDaskExecutor scheduler = scheduler num workers = num workers as flow ``` This may help since the executor is retrieved from storage at runtime

Michael Law · Answer

I can see from the timeline it never runs more than 4 which matches the amount of processors on the machine

Michael Law · Answer

Thanks for getting back to me

Michael Law · Answer

I ll try that suggestion too

Anna Geller · Answer

Gotcha What may also help is to log the thread and process ID within your flow You could do that by attaching the thread and process to your log format ```export PREFECT LOGGING FORMAT= levelname s name s process s thread s | message s ```

Anna Geller · Answer

this could be set on the agent or attached to your run config afaik ```flow run config = KubernetesRun image=self kubernetes job image env=dict PREFECT LOGGING FORMAT= levelname s name s process s thread s | message s ```

Michael Law · Answer

Ah yeah great shout < Anna Geller> I ll add that now and trigger some of these jobs

Michael Law · Answer

Success

Michael Law · Answer

TYVM

Anna Geller · Answer

Glad to hear it works now raised hands