https://prefect.io logo
Join Slack
Powered by
Hi Folks, we are in the process of migrating to ma...
# ask-community
m
Hi Folks, we are in the process of migrating to make use of a KubernetesRun config and a DaskExecutor, alongside our Kubernetes agent on EKS. We seem to be running into issues running our flows with a custom context from the prefect UI.
👀 1
We have succesfully registered and run flows but we are encountering an issue when it comes to running a flow from the UI - specifically whenever we need to supply a custom context. Using the graphql API to run our flow works fine, specifically running this line
Copy code
client.create_flow_run(flow_id=flow_id_to_use, context={"min_workers": 3, max_workers: "3"})
Will result in running our flow with the context properly updated (see the attached screenshot)
However when we try to achieve the same from the UI - we bump into this error:
Copy code
Failed to load and execute Flow's environment: StorageError("An error occurred while unpickling the flow:\n  TypeError('code() takes at most 15 arguments (16 given)')\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n  - python: (flow built with '3.8.6', currently running with '3.7.10')")
What seems to be happening is that the UI has no way of loading our custom Kubernetes template that we used when we registered the flow, and so it falls back on the default prefect image (which uses python 3.7 and fails the flow).
How can we run a flow with a custom prefect context from the UI without overwriting our RunConfig and without having to have to input our RunConfig every time ?
n
Hi @Marwan Sarieddine - this sounds like the UI isn't properly populating your coded runconfig; let me open a ticket for this; I don't think there's a workaround at the moment for this though so we'll try to get a fix out asap
@Marvin open "K8s runconfig templates aren't properly populated" in UI
m
m
Hi @nicholas - thanks for promptly responding about this I am using a template dictionary to construct the KubernetesRun config in case this helps
n
That is helpful @Marwan Sarieddine - would you be able to share the runconfig you're setting in code in its entirety? (Feel free to redact any info you need to)
m
Sure - will try to follow up with it in a bit soon
n
Much appreciated 🙏
m
@nicholas - please find the job template below
Copy code
{'apiVersion': 'batch/v1',
 'kind': 'Job',
 'metadata': {'labels': {'app': 'prefect-dask-job'},
  'name': 'prefect-dask-job'},
 'spec': {'template': {'metadata': {'annotations': {'<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>': 'false'},
    'labels': {'app': 'prefect-dask-job'}},
   'spec': {'affinity': {'nodeAffinity': {'requiredDuringSchedulingIgnoredDuringExecution': {'nodeSelectorTerms': [{'matchExpressions': [{'key': '<http://alpha.eksctl.io/nodegroup-name|alpha.eksctl.io/nodegroup-name>',
           'operator': 'In',
           'values': ['prefect-m5-xlarge']}]}]}}},
    'containers': [{'env': [{'name': 'PREFECT__CONTEXT__ng_name',
        'value': 'prefect-m5-xlarge'},
       {'name': 'PREFECT__CONTEXT__add_not_safe_to_evict', 'value': 'True'},
       {'name': 'PREFECT__CONTEXT__dask_memory_management', 'value': 'True'},
       {'name': 'PREFECT__CONTEXT__worker_memory_gb', 'value': '3.6'},
       {'name': 'PREFECT__CONTEXT__worker_cpu', 'value': '0.75'},
       {'name': 'PREFECT__CONTEXT__scheduler_memory_gb', 'value': '12.0'},
       {'name': 'PREFECT__CONTEXT__scheduler_cpu', 'value': '3.0'},
       {'name': 'PREFECT__CONTEXT__min_workers', 'value': '1'},
       {'name': 'PREFECT__CONTEXT__max_workers', 'value': '1'},
       {'name': 'PREFECT__CONTEXT__death_timeout', 'value': '600'},
       {'name': 'PREFECT__CONTEXT__use_nanny', 'value': 'False'},
       {'name': 'PREFECT__CONTEXT__use_dashboard', 'value': 'False'},
       {'name': 'PREFECT__CONTEXT__num_proc_per_worker', 'value': '1'},
       {'name': 'PREFECT__CONTEXT__n_threads_per_process', 'value': '1'},
       {'name': 'PREFECT__CONTEXT__dask_memory_limit', 'value': 'None'},
       {'name': 'PREFECT__CONTEXT__scheduler_name', 'value': 'None'},
       {'name': 'PREFECT__CONTEXT__scheduler_port', 'value': 'None'}],
      'image': 'xxxxxxx:0.20.1',
      'imagePullPolicy': 'IfNotPresent',
      'name': 'flow',
      'resources': {'limits': {'cpu': '1000m', 'memory': '3000M'},
       'requests': {'cpu': '1000m', 'memory': '3000M'}}}],
    'restartPolicy': 'Never'}}}}
I only redacted the image url value
n
that's perfect, thank you marwan! that'll let me test this really well
👍 1
m
Hi @nicholas - following up on this - is there an issue opened that I can track for progress ? (thanks)
n
Hi @Marwan Sarieddine - yes, the issue linked above is the one you can track
m
Sounds good - thanks
Open in Slack
PreviousNext
Powered by