Madison Schott
07/07/2021, 6:09 PMwiretrack
07/07/2021, 8:40 PMflow
), I was wondering if the large amount of rows on other tables will start to get in the way of the frontend performance (and hasura’s, and apollo’s) . Putting state
in mongodb or something should completely solve the challenge (not really sure if it’s really a challenge), but it seems that this would be a huge change, since the code is really well tied together. I was wondering how do you guys see scalability on the server, and curious on what approaches the cloud version uses to overcome potencial scalability issues in the long term.Aric Huang
07/07/2021, 9:55 PMFailed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')
error running the flow, when using the job_template_path
option for KubernetesRun. I can successfully register the flow, and when running the flow it seems to be respecting the template.yml
I passed in (I see my Kubernetes cluster running an appropriate pod based on my template) - but after pulling the image I get the FileNotFoundError
. Thoughts on what is going on? My flow basically looks like this:
with Flow("Test") as test_flow:
...
test_flow.run_config = KubernetesRun(
job_template_path="template.yml"
)
test_flow.storage = GitHub(
repo="<path>",
path="flows/test_flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN"
)
Joseph Loss
07/07/2021, 10:02 PMFlow run 478adfa1-5f4f-4dec-a121-a2443bc0a253 has a `run_config` of type `LocalRun`, only `DockerRun` is supported
I had registered the flow and previously used the flow on LocalRun, now all of a sudden it's failing but it was working a few hours ago?Maria
07/08/2021, 12:05 AMFailed to load and execute Flow's environment: FlowStorageError("An error occurred while unpickling the flow: ModuleNotFoundError("No module named 'transform'")")
If I comment out transform, it will complain about next module. Project setup example is in the threadMike Wochner
07/08/2021, 6:24 AMDavid Elliott
07/08/2021, 8:48 AMChristian Michelsen
07/08/2021, 9:25 AMMichael Hadorn
07/08/2021, 2:44 PMAlain Prasquier
07/08/2021, 3:37 PMFailed to load and execute Flow's environment: UnpicklingError("invalid load key, '{'."
I’ve read the Slack thread that referred this error: https://prefect-community.slack.com/archives/CL09KU1K7/p1623777537484700?thread_ts=1623704787.418000&cid=CL09KU1K7
which seems to point to version compatibility issues.
My setup :
• Agent Prefect version 0.15.0
• Running on Prefect Cloud (“core_version”: “0.14.22+9.g61192a3ee”)
• My task is a simple hello-world log, deployed with with Flow("s3_flow", storage=S3_STORAGE, run_config=ECSRun_CONFIG) as flow:
• is_serializable(flow) = True
Should I be downgrading my agent to match the server version ?
Any help will be very welcome !Nicholas Chammas
07/08/2021, 5:44 PMprefect register --force
to make sure the new executor config is being sent up to Prefect Cloud. But I don’t know if that’s necessary.Jeff Baatz
07/08/2021, 6:17 PMStartFlowRun
and the parent flow is throwing ValueError: Failed to find current tenant None in result {'data': {'tenant'...
The subflows are triggering and running correctly, but it looks as though the parent flow can't view their status? Does the agent running the parent flow need to have a user API key or something attached to it in order to query flow status, but not to submit a flow for execution?Matthias Roels
07/08/2021, 6:52 PMprefect build
command, which seems like an interesting command for my use-case. However, I cannot find any documentation on this command. Is there any, and if so, where can I find it? Thanks!Ben Muller
07/08/2021, 8:12 PMCharles Liu
07/08/2021, 9:42 PMitay livni
07/08/2021, 10:40 PMTom Blake
07/09/2021, 8:11 AMException raised while calling state handlers: ValueError("Failed to find current tenant None in result {'data': {'tenant': [{'slug': 'myTenant', 'id': myTenantId}]}}")
Any ideas to what could be causing this?Hugo Shi
07/09/2021, 11:30 AMBruno Murino
07/09/2021, 12:45 PMPaolo
07/09/2021, 3:27 PMNicholas Chammas
07/09/2021, 4:22 PMParameter
directly to a DatabricksRunNow
task, Prefect detects the dependency and passes the information from the parameter to the task correctly. e.g.
path = Parameter("path")
DatabricksRunNow(...)(
databricks_conn_secret=SECRET,
notebook_params: {
"path": path,
}.
)
However, if I plug the parameter into a formatted string, for example, the information is no longer passed from the parameter to the task correctly.:
path = Parameter("path")
DatabricksRunNow(...)(
databricks_conn_secret=SECRET,
notebook_params: {
"path": f"{path}",
}.
)
In this case, the parameter class instance is plugged into the string vs. the actual parameter value that we want. So the notebook is given a repr()
of a Parameter
class instance — which is unusable, of course — instead of the string value of the parameter that we actually want.
Why is that, and is there a way around this?Jacob Goldberg
07/09/2021, 4:58 PMVincent Chéry
07/09/2021, 6:45 PM..autofunction:: my_beautiful_task
directive, sphinx will document it according to its docstring without any issue, but if I rely on ..automodule::
to automatically document all the members (functions, classes...) of my module, it only finds regular functions and classes, not prefect tasks and flows.
pdoc : I just gave it a quick shot, the same happends, it documents functions and classes, but not prefect tasks and flows :(
Any idea? Thx !Jan Vlčinský
07/09/2021, 10:01 PMprefect.run_configs.ECSRun
. Our flow keeps starting and never completes - not even printing single line of text.
Is there any working example of AWS ECS based prefect runner storing data on mounted AWS EFS volume?Son Mai
07/10/2021, 3:55 AMSayandip Sarkar
07/10/2021, 4:32 PMdex
07/10/2021, 6:36 PMdex
07/11/2021, 8:26 AMflows.py
has my flow definition, and utils.py
hosts some number of helper functions. And I'm using Github storage for the flow. Since Github storage only specify the path of the flows.py
, it got module not found during execution. I wonder if Github storage does not support module? I can't seem to find a good descrption in the documentation. Thanks in advance if anyone can give me a pointer.Mexson Fernandes
07/11/2021, 2:57 PMScott Vermillion
07/11/2021, 10:15 PMfrom prefect import Client
client = Client()
client.create_flow_run(flow_id="<some flow_id>")
But lo and behold, someone comes along an re-registers the flow with Cloud. The previous flow_id gets archived and a new one is generated.
Now I have to go and update my Python app with the new flow_id? Can this be done by name or something? Or can I pull the flow_id as a variable? Or?
Thank you.Scott Vermillion
07/11/2021, 10:15 PMfrom prefect import Client
client = Client()
client.create_flow_run(flow_id="<some flow_id>")
But lo and behold, someone comes along an re-registers the flow with Cloud. The previous flow_id gets archived and a new one is generated.
Now I have to go and update my Python app with the new flow_id? Can this be done by name or something? Or can I pull the flow_id as a variable? Or?
Thank you.Kevin Kho
07/11/2021, 10:21 PMStartFlowRun(…).run()
to run it.Scott Vermillion
07/11/2021, 10:24 PMKevin Kho
07/12/2021, 4:54 PMprefect auth login
.Scott Vermillion
07/12/2021, 5:27 PM