Andreas Nord
08/12/2022, 3:12 PMFlow could not be retrieved from deployment.
On Prefect 1 I would package all my python dependencies in an Docker image, as well as the flow code (using storage=Module()). What would be the equivalent for Prefect 2? I saw the option of putting the flow definitions on AWS but I don't really see the pointStéphan Taljaard
08/12/2022, 4:08 PMPhilip MacMenamin
08/12/2022, 4:13 PMtask_name
, which returns "some_value", and I do:
state = flow.run()
How to I get the return of task_name
? (ie "some_value")Jimmy Le
08/12/2022, 4:31 PMHeather DeHaven
08/12/2022, 4:42 PMwith Flow(name="my_parent_flow") as flow:
name = Parameter('name')
...
create_flow_run.map(
flow_name=unmapped('child_flow')
...
run_name=unmapped(f'<use Parameter 'name''s value> child run')
)
Alex Tam
08/12/2022, 5:02 PMSean Malone
08/12/2022, 5:05 PMDeepak Pilligundla
08/12/2022, 6:23 PMFailed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'redshift_connector\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
Isaac Kargar
08/12/2022, 6:38 PMPanos V.
08/12/2022, 6:41 PMMatt Delacour
08/12/2022, 6:43 PMAlex Shea
08/12/2022, 7:10 PMPREFECT__CLOUD__AGENT__ENV_VARS
agent variable?Chris Marchetti [Datateer]
08/12/2022, 7:46 PM[2022-08-12 19:42:34+0000] INFO - prefect.Docker | Pushing image to the registry...
The error is:
InterruptedError: EOF
Complete traceback in thread. Anyone seen this before?Tim Helfensdörfer
08/12/2022, 8:30 PMEmil Ordoñez
08/12/2022, 9:52 PMEdmondo Porcu
08/12/2022, 9:54 PMBlake Hamm
08/12/2022, 10:24 PMdict
, it still reads it in as a string in the UI and can't seem to parse it out. I'm pretty much stuck with the default dictionary parameter or I can trigger it from the CLI. Do I essentially have to use pydantic
and explicitly specify the model when working with dictionary parameters?
Some other funny parameter behavior, the UI displays the default parameter as [object Object]
when I'm creating a custom flow run. But on the flow run (with defaults), it looks like more like a dictionary.Brad
08/12/2022, 10:59 PMRohan Chutke
08/13/2022, 12:25 AMMasatoShima
08/13/2022, 1:38 PMArtur Spatari
08/13/2022, 2:44 PMMoises Vera
08/13/2022, 6:10 PMMarwan Sarieddine
08/13/2022, 6:40 PMRio McMahon
08/13/2022, 7:11 PMhttps://
prefix but s3://
is hardcoded in the S3 filesystem object). I did this by copying the S3 storage object code and removing the hardcoded prefix but leaving everything else the same (except name and credential arguments). The block successfully registers to prefect cloud. However when I try to attach it as the storage block I get the error in the comments.
From what I can tell this is happening during the validation step. At some point prefect is looking at the registry for the Block
type via:
from prefect.utilities.dispatch import lookup_type, get_registry_for_type
from prefect.blocks.core import Block
registry = get_registry_for_type(Block)
print(registry)
which returns all the prefect defined blocks. If I try to build a deployment it will fail because the custom filesystem class I created does not show up in that registry. Currently the remote file system object stores all ‘settings’ as plain text which isn’t ideal because I’m storing access credentials. Thoughts on the best way to proceed?Dev Dabke
08/13/2022, 7:43 PMMarwan Sarieddine
08/13/2022, 8:32 PMOscar Björhn
08/14/2022, 11:22 AMJames Brady
08/14/2022, 1:40 PMdask_kubernetes.KubeCluster
class – trying to get a flow to run on a GPU-enabled node. What's happening:
• The pod_template I'm using for the flow specifies <http://nvidia.com/gpu|nvidia.com/gpu>: 1
, per the docs
• karpenter starts a new node, which has a GPU (🙌)
• However, the new node can't accommodate the dask client because it doesn't have the right resource annotation ("0/3 nodes are available: … , 3 Insufficient nvidia.com/gpu")
I realise this might be a question better-suited to the dask community, but would appreciate any stories of people successfully running Prefect 2 workloads on GPU-accelerated nodes and/or help figuring out the above specific issue.James Phoenix
08/14/2022, 9:02 PMJames Phoenix
08/14/2022, 9:02 PMFile system created with scheme 'gcs' from base path '<gcs://prefect_source_data>' could not be created. You are likely missing a Python module required to use the given storage protocol.
James Phoenix
08/14/2022, 9:02 PMFile system created with scheme 'gcs' from base path '<gcs://prefect_source_data>' could not be created. You are likely missing a Python module required to use the given storage protocol.
Anna Geller
08/14/2022, 9:47 PM