Lon Nix
11/03/2021, 2:47 PMFROM prefecthq/prefect:latest
RUN apt update && apt install -y openssh-client
However now I get this every run Failed to load and execute Flow's environment: HangupException('Host key verification failed.\r')
I created a known_hosts file by adding this to the Dockerfile but it didn't help.
RUN mkdir ~/.ssh && ssh-keyscan -t rsa <http://github.com|github.com> >> ~/.ssh/known_hosts
Any thoughts?Santiago Gonzalez
11/03/2021, 5:35 PMLon Nix
11/03/2021, 5:56 PMjob_template
job_template = '''
apiVersion: batch/v1
kind: Job
spec:
template:
spec:
containers:
- name: flow
volumeMounts:
- name: ssh-key
readOnly: true
mountPath: "/root/.ssh"
volumes:
- name: ssh-key
secret:
secretName: prefect-ssh-key
optional: false
defaultMode: 0600
'''
but what shows up for the job is actually
volumeMounts:
- mountPath: /root/.ssh
name: ssh-key
readOnly: true
volumes:
- name: ssh-key
secret:
defaultMode: 384
optional: false
secretName: prefect-ssh-key
It's not keeping the same permissions for defaultMode
and I think that's why I'm getting an error about Failed to add the RSA host key for IP address '140.82.114.4' to the list of known hosts
It is using the correct secret name though. Why would it not take the correct defaultMode?Santiago Gonzalez
11/03/2021, 9:46 PMjack
11/04/2021, 1:47 AM/bin/sh: prefect: command not found
The Dockerfile used to build the image based on amazonlinux:2 runs pip install prefect[aws]
, and when I run the image locally prefect
is in the path.Chris Arderne
11/04/2021, 10:07 AMstate_handler
to name a flow run based on context, but haven't figured out how to name it based on passed parameters.
Eg if I ran a Flow with a animal=cat
I'd like that Flow run (as viewed in the UI) to be named run-cat
or something…Mike Lev
11/04/2021, 2:59 PMLocalRun
with LocalExecutor
how can I do the equivalent of sys.path.append
to the run config working dir
currently flows are working without a backend but then when I start to run on my server I get a an error with ModuleNotFound
currently my structure is as such
MainProject/
|-coreLogicModule/
|-PrefectStack/
flows/ -need access to coreLogic
runflows.py
Lawrence Finn
11/05/2021, 12:09 PMLawrence Finn
11/05/2021, 12:32 PMek
11/05/2021, 3:09 PMLawrence Finn
11/06/2021, 1:07 PMAdam Everington
11/08/2021, 4:15 PMLawrence Finn
11/09/2021, 1:03 PM08:01:52.893 | Submitting flow run 'f37fba9f-4280-431d-9f43-889e53192f24'
08:01:52.894 | Completed submission of flow run 'f37fba9f-4280-431d-9f43-889e53192f24'
08:01:52.901 | Finished monitoring for late runs.
08:01:54.187 | Flow run 'f37fba9f-4280-431d-9f43-889e53192f24' exited with exception: KeyError('__main__')
Adam Everington
11/09/2021, 3:00 PMhelm install prefectprod prefecthq/prefect-server -n prod --set agent.enabled=true --set postgresql.postgresqlUsername=admin%40my-server --set postgresql.postgresqlPassword=p%40ssword123 --set postgresql.externalHostname=<http://my-server.postgres.databases.azure.net|my-server.postgres.databases.azure.net> --set postgresql.useSubChart=false
when looking at my pods I'm getting the followingPrasanth Kothuri
11/09/2021, 6:30 PM-p 12001:12001
Prasanth Kothuri
11/09/2021, 6:43 PMhost_config
of DockerRun
?Payam Vaezi
11/09/2021, 9:32 PM0.15.7
from 0.14.22
we are seeing this error on registration of flow to the server. Any idea why this is happening? Has API contract changed? Also tagging @David Harrington for visibility. Traceback in thread.Ryan Sattler
11/10/2021, 12:34 AMCompleted deployment of flow run
), but no job container ever appears in k8s and the UI just hangs forever at “Submitted for execution”. Restarting the agent does not help. Does anyone know how to debug this?Ege Demirel
11/10/2021, 2:25 PMPedro Martins
11/10/2021, 6:32 PMflow_run_id
during runtime? I will run this flow many times and for run I’d like to save the flow_run_id
associated with it along with other metadata.Pedro Martins
11/10/2021, 6:34 PMKelby
11/11/2021, 2:40 PMwait_for
argument, Orion seems to ignore the wait_for and processes the tasks even when the subflow fails.jack
11/11/2021, 5:26 PMFlowRunView
to access the logs of a flow run?
Sometimes it works for me, but other times I keep seeing stale logs.Sylvain Hazard
11/12/2021, 7:56 AMSylvain Hazard
11/15/2021, 9:47 AMAqib Fayyaz
11/15/2021, 11:33 AMPedro Martins
11/15/2021, 11:57 AMraise KeyError(
KeyError: 'Task slug RunSimulationTask-1 is not found in the current Flow. This is usually caused by a mismatch between the flow version stored in the Prefect backend and the flow that was loaded from storage.\n- Did you change the flow without re-registering it?\n- Did you register the flow without updating it in your storage location (if applicable)?'
simulation_flow.py
SimulationFlow = Flow(
"RunSimulationFlowDemo-3",
storage=Local(stored_as_script=True, path=__file__),
)
----
simulation_task.py
class RunSimulationTask(Task):
def __init__(
self,
simulation_run_parameters: SimulationRunParameters,
simulation_gateways: SimulationGateways,
factory_scheme_gateways: FactorySchemeGateways,
map_gtw: MAPDataGatewayBase = None,
**kwargs,
):
super(RunSimulationTask, self).__init__(name="RunSimulationTask")
self.simulation_run_parameters = simulation_run_parameters.to_dict()
self.simulation_gateways = simulation_gateways
self.factory_scheme_gateways = factory_scheme_gateways
self.map_gtw = map_gtw or MAPDataGateway()
def run(self) -> str:
flow_run_id = prefect.context.get("flow_run_id")
<http://self.logger.info|self.logger.info>(f"flow_run_id: {flow_run_id}")
simulation_controller = SimulationController(
**self.factory_scheme_gateways.to_dict(),
**self.simulation_gateways.to_dict(),
map_gtw=self.map_gtw,
)
<http://self.logger.info|self.logger.info>(f"Run simulation task. [{self.simulation_run_parameters}]")
simulation_controller.run(
**self.simulation_run_parameters, flow_run_id=flow_run_id
)
return flow_run_id
----
simulation_manager.py
task = RunSimulationTask(
simulation_run_parameters=simulation_run_parameters,
simulation_gateways=self._simulation_gateways,
factory_scheme_gateways=self._factory_scheme_gateways,
map_gtw=self._map_gtw,
)
SimulationFlow.add_task(task)
SimulationFlow.run_config = self.flow_config.run_config
SimulationFlow.executor = self.flow_config.executor
SimulationFlow.state_handlers = self.flow_config.state_handlers
I’m trying to register and run this flow using the client.register()
and client.create_flow_run()
.
One weird thing I noticed is that the Flow is not being saved to my local ~/.prefect/flows
directory. Could be that?Lukas Brower
11/15/2021, 3:43 PMmapped
field has recently started always showing up as false.
Example response for one of our mapped tasks from the interactive API:
"task": [
{
"id": "ad65f438-062b-4f9d-ba8e-826edce68d74",
"name": "<task_name>",
"mapped": false,
"task_runs": [
{
"name": null,
"created": "2021-11-11T01:08:13.326713+00:00",
"map_index": -1,
"version": 2
},
{
"name": "<task_run_0>",
"created": "2021-11-11T01:31:55.062178+00:00",
"map_index": 0,
"version": 3
},
{
"name": "<task_run_1>",
"created": "2021-11-11T01:31:55.062178+00:00",
"map_index": 1,
"version": 3
}
]
},
We have some logic which depends on the value of mapped. We could use map_index != -1
in place of this I guess, but we’re trying to determine why the mapped
field is always false despite there being mapped task_runs
jack
11/15/2021, 6:26 PMFailed: "An error occurred (ThrottlingException) when calling the DeregisterTaskDefinition operation (reached max retries: 2): Rate exceeded"
Rob Fowler
11/16/2021, 1:19 AM