Kevin Kho
04/26/2022, 6:15 PMAnna Geller
04/26/2022, 6:31 PMAnna Geller
04/26/2022, 11:26 PMone of the best data talk▾
Mansoor Hassan
05/06/2022, 11:31 AMMarcin Grzybowski
05/30/2022, 8:35 AMNameError: name 'err' is not defined
when I run this line:
from prefect.tasks.snowflake import SnowflakeQuery
It leads me to this file
/src/prefect/tasks/snowflake/__init__.py
and this is the code from init.py that fails for me:
try:
from prefect.tasks.snowflake.snowflake import (
SnowflakeQuery,
SnowflakeQueriesFromFile,
)
except ImportError:
raise ImportError(
'Using `prefect.tasks.snowflake` requires Prefect to be installed with the "snowflake" extra.'
) from err
what am I doing wrong? Thank youYossi
06/29/2022, 10:08 AMGuilherme Bordignon
06/29/2022, 10:26 PM@async_task()
async def get_all_objects_from_models(
models: List
):
"""
Get all objects from each model from the @param models.
"""
return await asyncio.gather(*[model.get_all_objects() for model in models])
to be used like this
all_objects_model_1, all_objects_model_2, all_objects_model_3 = get_all_objects_from_models([Model1, Model2, Model3])
the problem is that I have to pass it a nout
in the decorator or a Tuple
in the return type annotation with the number of items inside (Tuple[List, List, List]
) for this to work, otherwhise a got this error `'Task is not iterable. If your task returns multiple results, pass nout
to the task decorator/constructor, or provide a Tuple
return-type annotation to your task.',` but I would like to make this function generic and pass and return as many models as I want. Is there a way to make it work?Kevin Focke
07/02/2022, 3:13 PMDavid Bender
07/04/2022, 2:46 PMJarrod Clark
07/08/2022, 3:04 PMJaco
07/13/2022, 4:01 PM@flow def main(): train,test,unique_movie_titles,unique_user_ids = getData().result()
i keep getting type errors like: tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot convert a Tensor of dtype variant to a NumPy array
, am I limited in what type of return I need to have in my functions to use the .result()?J
07/14/2022, 6:53 PMFalk
07/15/2022, 10:04 AMwith open(f"{os.getcwd()}{file_name}.yaml", "r") as file:
return yaml.safe_load(file)
Both flow.py
and file.yaml
are in the same directory.
I also tried setting the path without cwd and like this:
with open(f"{file_name}.yaml", "r") as file:
return yaml.safe_load(file)
but also no luck.
I always get
FileNotFoundError: [Errno 2] No such file or directory: '/file.yaml'
Any ideas what is causing this? Do I have to specify files other than .py
files for the agent somehow?rami ben shalom
07/20/2022, 12:00 PMishan
07/22/2022, 9:49 AMA -> C, B -> D, (C, D) -> E
• When the dependency graph is a DAG, I am not sure how to approach it. E will try to evaluate B and C, but both could concurrently try to evaluate A, while we only want A to be executed once.
◦ A -> B, A -> C, (B, C) -> E
A solution to this is topological sorting the full graph, and executing views as soon as their parent dependencies have already been run. I wrote the code manually but curious if the prefect API can handle it for me.Alex
07/25/2022, 9:58 PMJohn Kang
07/26/2022, 3:40 PMStefan
07/27/2022, 10:37 PMMarty Ko
07/29/2022, 10:18 AMMarty Ko
07/29/2022, 10:18 AMMarty Ko
07/29/2022, 10:18 AMAndreas Nigg
08/02/2022, 7:37 AMeddy davies
08/02/2022, 10:54 AMNikita Kodenko
08/04/2022, 6:19 AMMarty Ko
08/08/2022, 8:27 AMBigya Man Pradhan
08/08/2022, 2:46 PMprefect deployment build --help
Which has a line --manifest-only Generate the manifest file only.
But the yaml file is also being generated when the build command is run with manifest-only flag.
Additionallly, adding the --output
flag and setting an output name results in only the YAML file being renamed and not the manifest file.
Is this an intended behaviour?Adam Eury
08/10/2022, 1:01 PM--manifest-only
flag. My understanding is that it skips the step of uploading the flow file to the configured storage location. Is it intended to support the use case of having the flow file added to a Docker image rather than remote storage like GCS or S3?Chris L.
08/10/2022, 1:24 PMworker_client
context manager doesn't seem to work either and raises a "no workers found" error, despite the fact that I can access the Dask dashboard in localhost showing Prefect 2.0 tasks.
Below is a small reproducible example. You can see that in the screenshot that none of the inc
double
add
delayed function calls are picked up in the dashboard.
Wondering if there is some generic way for me to access the Dask task runner's cluster within tasks in Prefect 2.0?
import dask
import time
from prefect import flow, task
from prefect_dask import DaskTaskRunner
@task
def parallel_sum(data):
def inc(x):
return x + 1
def double(x):
return x * 2
def add(x, y):
time.sleep(5)
return x + y
output = []
for x in data:
a = dask.delayed(inc)(x)
b = dask.delayed(double)(x)
c = dask.delayed(add)(a, b)
output.append(c)
total = dask.delayed(sum)(output).compute()
time.sleep(30)
return total
@flow(task_runner=DaskTaskRunner())
def run_parallel_sum(data):
parallel_sum.submit(data)
if __name__ == "__main__":
run_parallel_sum(list(range(1000)))
Angel Acosta
08/11/2022, 3:41 PMYusuf
08/15/2022, 7:46 PMYusuf
08/15/2022, 7:46 PMAnna Geller
08/15/2022, 8:08 PMYusuf
08/15/2022, 8:14 PMAnna Geller
08/15/2022, 10:41 PMYusuf
08/15/2022, 11:10 PMAnna Geller
08/15/2022, 11:18 PM