Stéphan Taljaard
03/14/2022, 10:14 AMChristian Nuss
03/14/2022, 6:20 PMcreate_flow_run.run(flow_name="flow-2")
within a task created by a KubernetesRun
...
assuming i have
with Flow("flow-1", run_config=KubernetesRun(...)):
... creates task that does a create_flow_run.run(flow_name="flow-2") ...
should flow-2
receive/inherit the run_config
from flow-1
?jack
03/15/2022, 1:15 PMTimeout waiting for network interface provisioning to complete.
?Sharath Chandra
03/15/2022, 3:16 PMspark-submit
using prefect’s ShellTask
.
I have created a subclass of ShellTask to invoke the `spark-submit.`The spark jobs are running on k8s.
I am facing an issue where prefect tasks are not completing and continuously running. I see this happening on the tasks which are slightly long running (> 10 mins).
The master flow maps over list and orchestrates the prefect ShellTask
e.g.
K8sSparkSubmitTask.map(
id = ["id1", "id2", "idx"]
)
The task starts running for id1
and pod status is observed to be successful
. However prefect does not move to the next task. Why are these tasks not getting completed?Madhup Sukoon
03/15/2022, 3:20 PMhelm pull --destination /tmp/c68c98cd-c919-46b6-9e17-924f22aa2dd3 --version 2022.01.25 --repo <https://prefecthq.github.io/server> prefecthq/prefect-server
results in an error :
Error: chart "prefecthq/prefect-server" version "2022.01.25" not found in <https://prefecthq.github.io/server> repository
However, I do see 2022.01.25
as a version when I do helm search repo prefecthq/prefect-server --versions
.
Can someone please point out where I am going wrong?Lukas Brower
03/15/2022, 4:10 PMSlava Shor
03/15/2022, 5:28 PMTypeError: Task is not iterable
exception.
@task(timeout=timedelta(seconds=30))
def fetch_symbols(...) -> tuple[MyCustomType, ...]:
...
Jacob Longhurst
03/15/2022, 7:17 PMDarren Burton
03/16/2022, 3:24 PMLana Dann
03/16/2022, 4:55 PMBruno Nunes
03/17/2022, 3:22 PMprefect server create-tenant --name default --slug default
I'm getting the following error:
prefect.exceptions.ClientError: [{'message': 'Uniqueness violation.', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_tenant'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Uniqueness violation.'}}}]
It seems that the tenant already exists. Do you know what might be the problem?Ross Rochford
03/17/2022, 5:41 PMRoss Rochford
03/17/2022, 5:56 PMRoyzac
03/17/2022, 8:11 PMJacob Longhurst
03/17/2022, 10:05 PM@task()
def my_task():
import prefect
logger = prefect.context.get('logger')
I’ve tried it both with having the import prefect
at the top level outside of the task and in the task like in that example and both times it gives me this
TypeError: Pickling context objects is explicitly not supported. You should always access context as an attribute of the `prefect` module, as in `prefect.context`
Anyone have any ideas why it’s failing when i try to register the flow?Sam Brownlow
03/18/2022, 7:51 AMtypical Server
is deployed to a single node via docker-compose with a single agent, instead of being deployed via something like Helm, which horizontally scales scheduled flows across a cluster of agents?
How quickly does the size of the PostgreSQL database generally grow, relative to the number of flows run? Is there any reason that regularly deleting old runs would be any more complicated than as suggested here?
Are there any case studies with Prefect Server being used to run 10s of thousands of concurrent flows?
I see that there used to be a Nomad Agent, are there any helpful resources for running Prefect on a nomad cluster?
Thanks for any advice you are able to share. I have been diving into Prefect for only the past couple of days so greatly appreciate any referential pointers here.Gabriel Milan
03/18/2022, 1:17 PMhasura:
image:
name: hasura/graphql-engine
tag: v1.3.3
pullPolicy: IfNotPresent
pullSecrets: []
service:
type: ClusterIP
port: 3000
labels: {}
annotations: {}
replicas: 2
strategy: {}
podSecurityContext: {}
securityContext: {}
env: []
resources:
limits:
cpu: "500m"
memory: "1Gi"
requests:
cpu: "100m"
memory: "256Mi"
nodeSelector: {}
tolerations: []
affinity: {}
both the pods crash at the same time and keep restarting a few times before it's back up, so my Server gets a significant amount of downtime. I don't think the resources are underestimated, but I do see a pattern on RAM usage when this happens (I've sent a screenshot). I've also attached some logs for the pods, but I can't find any relevant information on them. Is there anywhere else I could gather information for this issue?Aaron Ash
03/21/2022, 5:48 AMprefect run --name ...
but the agent doesn't seem to actually run the flow logic.
Watching the prefect agent logs i see:
agent_1 | [2022-03-21 05:41:42,906] INFO - agent | Registering agent...
agent_1 | [2022-03-21 05:41:42,926] INFO - agent | Registration successful!
agent_1 |
agent_1 | ____ __ _ _ _
agent_1 | | _ \ _ __ ___ / _| ___ ___| |_ / \ __ _ ___ _ __ | |_
agent_1 | | |_) | '__/ _ \ |_ / _ \/ __| __| / _ \ / _` |/ _ \ '_ \| __|
agent_1 | | __/| | | __/ _| __/ (__| |_ / ___ \ (_| | __/ | | | |_
agent_1 | |_| |_| \___|_| \___|\___|\__| /_/ \_\__, |\___|_| |_|\__|
agent_1 | |___/
agent_1 |
agent_1 | [2022-03-21 05:41:42,933] INFO - agent | Starting DockerAgent with labels ['default']
agent_1 | [2022-03-21 05:41:42,933] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
agent_1 | [2022-03-21 05:41:42,933] INFO - agent | Waiting for flow runs...
agent_1 | [2022-03-21 05:44:39,049] INFO - agent | Deploying flow run 71f0c5d3-b73b-4537-8267-b0fd30325fd3 to execution environment...
agent_1 | [2022-03-21 05:44:39,082] INFO - agent | Pulling image localhost:5000/my-flow:2022-03-21t05-43-58-036580-00-00...
agent_1 | [2022-03-21 05:44:39,284] INFO - agent | Successfully pulled image localhost:5000/my-flow:2022-03-21t05-43-58-036580-00-00
agent_1 | [2022-03-21 05:44:39,689] INFO - agent | Completed deployment of flow run 71f0c5d3-b73b-4537-8267-b0fd30325fd3
Aaron Ash
03/21/2022, 5:49 AMJulian
03/21/2022, 9:56 AMPREFECT__EXECUTOR__DEFAULT_CLASS
and PREFECT__EXECUTOR__DASK__ADDRESS
in our agent.
Unfortunately, this approach doesn't seem to work anymore. I'm pretty sure we could initialize a DaskExecutor in our code
and attach it to our flow somehow, but this would be undisireable, as it requires changing each of our existing flows, but also
for each flow to come.
Is it possible to achieve the same kind behaviour as before, i.e. defining the execution engine for the agent, by simply setting 1 or 2 envs?Aniruddha Sengupta
03/21/2022, 1:14 PMBruno Nunes
03/21/2022, 1:53 PMSylvain Hazard
03/21/2022, 2:38 PMStorage
is quite different from Prefect 1.0's Storage
as it does not only store flow code but also results. Does this mean that GitStorage
is not coming back and the new recommended process is to deploy from the CI on validated changes ? Also, this is quite specific to my use case but having to connect to a cloud storage resource to run Orion on Kubernetes could be an issue. We are currently running with GitStorage
and a Kubernetes-hosted PostgreSQL instance which means we can manage every Prefect component from Kubernetes configuration.
• The current version of the KubernetesFlowRunner
seems to use a fixed specification for the created job. Reading the code, I could not find a way to specify a template the way 1.0's KubernetesRun
allows us to. Being able to create precisely sized pods to run flows has been really useful for us and I'd love to be able to keep doing it. Is it a planned feature for the future ?
🙏 !Lana Dann
03/21/2022, 5:59 PMFROM prefecthq/prefect:latest-python3.9
but when i start up my container and try to use the prefect cli, it says prefect isn’t installed
$ prefect
/bin/sh: 5: prefect: not found
Michael Aldridge
03/21/2022, 9:59 PMSen
03/22/2022, 8:50 AMDavid Charles
03/22/2022, 9:38 AMmutation {
set_secret(input: { name: "FOO", value: "BAR" }) {
success
}
}
Result:
{
"errors": [
{
"message": "Cannot query field \"set_secret\" on type \"Mutation\".",
"locations": [
{
"line": 2,
"column": 3
}
],
"extensions": {
"code": "GRAPHQL_VALIDATION_FAILED"
}
}
]
}
Any ideas?Daniel Nilsen
03/22/2022, 9:39 AMssl
?Architha Rao
03/23/2022, 6:41 AMVladislav Moiseev
03/23/2022, 9:56 AM