Paul Lucas
07/28/2022, 3:47 PMJack Chang
07/28/2022, 4:00 PMprefect run --name
and it set the flow to be scheduled on the cloud UI. I clicked start now
and the time has passed the 'Schedule to start' but my tasks are still in the scheduled state. Any thoughts on what I could do to start the flow?Andrew Miller
07/28/2022, 4:10 PMUserWarning: Attempted import of 'prefect.Parameter', which is part of Prefect 1.x, while Prefect 2.0.0 is installed. If you're upgrading you'll need to update your code, see the Prefect 2.x migration guide:However, the migration link gives me a 404 - Not found message.. Otherwise ensure that your code is pinned to the expected version.<https://orion-docs.prefect.io/migration_guide/>
Matthias
07/28/2022, 4:36 PMDmitrii Egunov
07/28/2022, 4:49 PMJack Chang
07/28/2022, 5:11 PMSana Shaikh
07/28/2022, 5:23 PMAlex Tam
07/28/2022, 6:02 PMFilip Floegel
07/28/2022, 6:13 PMKelvin
07/28/2022, 6:21 PMAndrew Miller
07/28/2022, 6:22 PMTasks cannot be run outside of a flow. To call the underlying task function outside of a flow useI triedtask.fn()
with prefect_test_harness():
but no luck...
Ah. Never mind, I should have been reading the error more. Calling fn
instead of run
fixes old test cases.Andrew Pruchinski
07/28/2022, 6:41 PMSlackbot
07/28/2022, 6:57 PMAndrew Miller
07/28/2022, 7:24 PM.map
method on my tasks. I wonder if Prefect 2.0 does not support thatChris Hansen
07/28/2022, 7:39 PMChu
07/28/2022, 8:06 PM@task('currency_orchestrator')
def call_currency_orchestrator():
return create_flow_run(
flow_name = "currency_orchestrator",
project_name = "xxxx",
)
why this throws me an error ??
aise TypeError("`fn` must be callable")
TypeError: `fn` must be callable
basically I wanna use that task to call a registered flow..Matt Delacour
07/28/2022, 8:12 PMMatt Delacour
07/28/2022, 8:17 PMSince API keys can be used across tenants if linked to a user account, you may also pass a tenant:
https://docs-v1.prefect.io/orchestration/concepts/api.html#authenticating-the-client-with-cloudWhat are the difference between "Tenants" and "Workspaces"? I might not fully understand those 2 keywords the right way
Andreas Nigg
07/28/2022, 8:22 PMAndrew Miller
07/28/2022, 8:38 PMprefect.config
in Prefect 2.0? I tried using Settings but I cant put custom key value pairs in it.John Mizerany
07/28/2022, 9:01 PMFlow __ not found
so was not sure if passing the type wrong?Chu
07/28/2022, 9:13 PMaise TypeError("`fn` must be callable")
TypeError: `fn` must be callable
Sam Dyson
07/28/2022, 10:10 PMDeployment
function to handle the deployment, as well as specifying FilePackager
and PickleSerializer
configurations so that we could create a single file from our (rather large) list of flow files and supporting library files. This file would then be pushed to an S3 bucket and pulled down during job execution on K8s using a standard docker image for the core dependencies.
For our use-case, it doesn't make sense to upload the entire directory of 200+ files for each deployment, so we really like the ability to package our flow into a single file. We also don't want to have to create a single storage block for each flow, which would multiply the number of files we're storing in s3 (and therefore have to pull down for each flow run). The documentation now states that the Deployment
function is no longer usable, but the YAML file and manifests provide no way to implement the FilePackager
and PickleSerializer
classes.
My question for the prefect team - are the FilePackager
and supporting serializers going to be available for deployments? Or is the new "upload all files" approach going to be the standard for Orion forever? If I were to hack the packager/serializer approach into our deployment methodology, will I run the risk of having the packager/serializer classes removed from the prefect package in the future? Anna has mentioned in previous comments that the prefect team is just getting started with deployments, but the lack of clarity and the fast breaking changes over the course of a week has made it quite difficult to plan our migration effectively - any clarity on this topic would be greatly appreciated!Austin Anderson
07/28/2022, 10:13 PMChris Hansen
07/28/2022, 10:52 PMGeoffrey Greenleaf
07/29/2022, 1:36 AMFaheem Khan
07/29/2022, 2:30 AMValueError: Unable to find block document named mini1 for block type RemoteFileSystem
Pedro Machado
07/29/2022, 2:49 AMSlackbot
07/29/2022, 4:42 AMViet Nguyen
07/29/2022, 6:25 AMxarray
dataset conversion, using DaskTaskRunner
, the created dask client (task_runner
) doesn't automatically pickup the xarray
functions such as xarray.dataset.concat()
or <http://xarray.dataset.to|xarray.dataset.to>_zarr
to fill up the works to all CPU threads like without Prefect. To bring the tasks to dask client with Prefect, I have to explicitly submit the task with @task
decorator and calling .submit()
This way, however, disallow xarray
functions to be executed in all CPU threads, only the wrapped task function to be executed in individual thread, which is not ideal. If anyone has come across this issue before and have a solution, much appreciated . Thanks a lot.