John Mizerany
07/28/2022, 9:01 PMFlow __ not found
so was not sure if passing the type wrong?Chu
07/28/2022, 9:13 PMaise TypeError("`fn` must be callable")
TypeError: `fn` must be callable
Sam Dyson
07/28/2022, 10:10 PMDeployment
function to handle the deployment, as well as specifying FilePackager
and PickleSerializer
configurations so that we could create a single file from our (rather large) list of flow files and supporting library files. This file would then be pushed to an S3 bucket and pulled down during job execution on K8s using a standard docker image for the core dependencies.
For our use-case, it doesn't make sense to upload the entire directory of 200+ files for each deployment, so we really like the ability to package our flow into a single file. We also don't want to have to create a single storage block for each flow, which would multiply the number of files we're storing in s3 (and therefore have to pull down for each flow run). The documentation now states that the Deployment
function is no longer usable, but the YAML file and manifests provide no way to implement the FilePackager
and PickleSerializer
classes.
My question for the prefect team - are the FilePackager
and supporting serializers going to be available for deployments? Or is the new "upload all files" approach going to be the standard for Orion forever? If I were to hack the packager/serializer approach into our deployment methodology, will I run the risk of having the packager/serializer classes removed from the prefect package in the future? Anna has mentioned in previous comments that the prefect team is just getting started with deployments, but the lack of clarity and the fast breaking changes over the course of a week has made it quite difficult to plan our migration effectively - any clarity on this topic would be greatly appreciated!Austin Anderson
07/28/2022, 10:13 PMChris Hansen
07/28/2022, 10:52 PMGeoffrey Greenleaf
07/29/2022, 1:36 AMFaheem Khan
07/29/2022, 2:30 AMValueError: Unable to find block document named mini1 for block type RemoteFileSystem
Pedro Machado
07/29/2022, 2:49 AMSlackbot
07/29/2022, 4:42 AMViet Nguyen
07/29/2022, 6:25 AMxarray
dataset conversion, using DaskTaskRunner
, the created dask client (task_runner
) doesn't automatically pickup the xarray
functions such as xarray.dataset.concat()
or <http://xarray.dataset.to|xarray.dataset.to>_zarr
to fill up the works to all CPU threads like without Prefect. To bring the tasks to dask client with Prefect, I have to explicitly submit the task with @task
decorator and calling .submit()
This way, however, disallow xarray
functions to be executed in all CPU threads, only the wrapped task function to be executed in individual thread, which is not ideal. If anyone has come across this issue before and have a solution, much appreciated . Thanks a lot.Matthias
07/29/2022, 8:03 AMMatthias
07/29/2022, 8:05 AMflurven
07/29/2022, 9:20 AMChristian
07/29/2022, 9:29 AMRajvir Jhawar
07/29/2022, 10:14 AMLeonardo Melo
07/29/2022, 11:57 AMDaniel Manson
07/29/2022, 12:29 PMGit
storage clones the full repo, so I'm trying to use that rather than the GitHub
storage as that looks to just clone the one flow file (having looked at the code). However, when using Git
storage I get the following error as the only log in my flow:
{'type': ['Unsupported value: Git']}
Tom Klein
07/29/2022, 12:55 PMJoshua Greenhalgh
07/29/2022, 1:04 PMcreate_flow_run
?
Want to do something like;James Phoenix
07/29/2022, 1:24 PMEli Treuherz
07/29/2022, 2:00 PMTRIGGERFAIL
passed to them as their first argument. If this sounds completely bizarre I can see if I can narrow down a minimal exampleR
07/29/2022, 2:13 PMSebastian
07/29/2022, 2:27 PMorchester
in which I organize all my orchestration code.
In Prefect 1.x the deployment pattern was akin to this:
from orchester.flows import hello_flow
hello_flow.storage = Module(__name__ + ".flows")
hello_flow.register()
In Prefect 2.x-beta I was able to achieve something similar like this:
from orchester.flows import hello_flow
Deployment(
name="entw",
flow=hello_flow,
packager=OrionPackager(serializer=ImportSerializer()),
)
# After that I created the deployment on the local Orion server with the CLI.
Now with Prefect 2.0.0 I am unsure what the deployment pattern would look like. My guess would be to run prefect deployment build .\orchester\flows\hello_flow.py:hello_flow --name entw
from the cli and to then manually edit the hello_flow-manifest.json
. I am not sure whether this is the the best approach or whether it needs to be done differently.
{
"flow_name": "hello_flow",
"import_path": ".\\orchester\\flows\\hello_flow.py:hello_flow", # Do I just need to change this line to a module path?
"parameter_openapi_schema": {
"title": "Parameters",
"type": "object",
"properties": {
"name": {
"title": "name",
"type": "string"
}
},
"required": [
"name"
],
"definitions": null
}
}
TLDR; Does anyone have a good suggestion how to deploy flows that live in their own Python module with Prefect 2.0.0?Zach Schumacher
07/29/2022, 2:54 PMMuhammad Mikaal Saleem Anwar
07/29/2022, 3:27 PMHafsa Junaid
07/29/2022, 3:43 PMJeffrey Lai
07/29/2022, 4:27 PMAndrew Richards
07/29/2022, 5:28 PMAccess to XMLHttpRequest at '<https://api.prefect.cloud/api/accounts/><MY ACCOUNT ID>/workspaces/<MY WORKSPACE ID>/block_documents/<MY BLOCK ID>' from origin '<https://app.prefect.cloud>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
It seems like CORS may be unhappy with the UI trying to do a cross-domain request from app.prefect.cloud to api.prefect.cloud?Oleg Sheyner
07/29/2022, 5:29 PMJake
07/29/2022, 6:46 PMprefect.exceptions.ClientError: Malformed response received from API.
Here is the full stack trace:
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
agent.start()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
self._setup_api_connection()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
agent_config_id=self.agent_config_id,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1788, in register_agent
tenant_id=self.tenant_id,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 264, in tenant_id
self._tenant_id = self._get_default_server_tenant()
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 176, in _get_default_server_tenant
response = self.graphql({"query": {"tenant": {"id"}}})
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 449, in graphql
retry_on_api_error=retry_on_api_error,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 405, in post
retry_on_api_error=retry_on_api_error,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 646, in _request
raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.
Here is the .yml:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: prefect-agent
namespace: prefect
name: prefect-agent
spec:
replicas: 1
selector:
matchLabels:
app: prefect-agent
template:
metadata:
labels:
app: prefect-agent
spec:
containers:
- args:
- prefect agent kubernetes start --job-template "<s3://xxx>"
command:
- /bin/bash
- -c
env:
- name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
value: ''
- name: PREFECT__CLOUD__API
value: <https://api.prefect.io>
- name: NAMESPACE
value: prefect
- name: IMAGE_PULL_SECRETS
- name: PREFECT__CLOUD__AGENT__LABELS
value: '[''xxx'', ''xxxxx'']'
- name: JOB_MEM_REQUEST
value: ''
- name: JOB_MEM_LIMIT
value: ''
- name: JOB_CPU_REQUEST
value: ''
- name: JOB_CPU_LIMIT
value: ''
- name: IMAGE_PULL_POLICY
value: Always
- name: SERVICE_ACCOUNT_NAME
value: ''
- name: PREFECT__BACKEND
value: cloud
- name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
value: http://:8080
- name: PREFECT__CLOUD__API_KEY
value: xxx
- name: PREFECT__CLOUD__TENANT_ID
value: xxx
- name: PREFECT__LOGGING__EXTRA_LOGGERS
value: "['xxxxx', 'xxxxxxxx']"
image: prefecthq/prefect:1.2.2-python3.7
imagePullPolicy: Always
livenessProbe:
failureThreshold: 2
httpGet:
path: /api/health
port: 8080
initialDelaySeconds: 40
periodSeconds: 40
name: agent
Jake
07/29/2022, 6:46 PMprefect.exceptions.ClientError: Malformed response received from API.
Here is the full stack trace:
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
agent.start()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
self._setup_api_connection()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
agent_config_id=self.agent_config_id,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1788, in register_agent
tenant_id=self.tenant_id,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 264, in tenant_id
self._tenant_id = self._get_default_server_tenant()
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 176, in _get_default_server_tenant
response = self.graphql({"query": {"tenant": {"id"}}})
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 449, in graphql
retry_on_api_error=retry_on_api_error,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 405, in post
retry_on_api_error=retry_on_api_error,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 646, in _request
raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.
Here is the .yml:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: prefect-agent
namespace: prefect
name: prefect-agent
spec:
replicas: 1
selector:
matchLabels:
app: prefect-agent
template:
metadata:
labels:
app: prefect-agent
spec:
containers:
- args:
- prefect agent kubernetes start --job-template "<s3://xxx>"
command:
- /bin/bash
- -c
env:
- name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
value: ''
- name: PREFECT__CLOUD__API
value: <https://api.prefect.io>
- name: NAMESPACE
value: prefect
- name: IMAGE_PULL_SECRETS
- name: PREFECT__CLOUD__AGENT__LABELS
value: '[''xxx'', ''xxxxx'']'
- name: JOB_MEM_REQUEST
value: ''
- name: JOB_MEM_LIMIT
value: ''
- name: JOB_CPU_REQUEST
value: ''
- name: JOB_CPU_LIMIT
value: ''
- name: IMAGE_PULL_POLICY
value: Always
- name: SERVICE_ACCOUNT_NAME
value: ''
- name: PREFECT__BACKEND
value: cloud
- name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
value: http://:8080
- name: PREFECT__CLOUD__API_KEY
value: xxx
- name: PREFECT__CLOUD__TENANT_ID
value: xxx
- name: PREFECT__LOGGING__EXTRA_LOGGERS
value: "['xxxxx', 'xxxxxxxx']"
image: prefecthq/prefect:1.2.2-python3.7
imagePullPolicy: Always
livenessProbe:
failureThreshold: 2
httpGet:
path: /api/health
port: 8080
initialDelaySeconds: 40
periodSeconds: 40
name: agent
Mason Menges
07/29/2022, 7:01 PM