CA Lee
08/03/2022, 2:14 AMHafsa Junaid
08/03/2022, 2:45 AMHafsa Junaid
08/03/2022, 2:48 AMwonsun
08/03/2022, 3:18 AMRajvir Jhawar
08/03/2022, 7:07 AMVersion: 2.0.2
API version: 0.8.0
Python version: 3.9.13
I having problems with both the read_flow_by_name
and read_deployment_by_name
. Neither of these method is working in when i try to use them:
Sample code:
from prefect.client import get_client
import asyncio
import json
async def main():
async with get_client() as client:
data = await client.read_flow_by_name(flow_name='test_flow')
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Error:
Traceback (most recent call last):
File
data = await client.read_flow_by_name(flow_name='test_flow')
/.venv/lib/python3.9/site-packages/prefect/client.py", line 508, in read_flow_by_name
return schemas.core.Deployment.parse_obj(response.json())
File "pydantic/main.py", line 521, in pydantic.main.BaseModel.parse_obj
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for Deployment
flow_id
field required (type=value_error.missing)
manifest_path
field required (type=value_error.missing)
Viet Nguyen
08/03/2022, 7:12 AMOscar Björhn
08/03/2022, 8:15 AMT B
08/03/2022, 10:04 AMAlvaro Durán Tovar
08/03/2022, 10:14 AMharis khan
08/03/2022, 10:17 AMShrikkanth
08/03/2022, 11:34 AMAmol Shirke
08/03/2022, 12:43 PMAmol Shirke
08/03/2022, 12:44 PMPaul Lucas
08/03/2022, 12:45 PMdbt_shell_task = DbtShellTask(
profiles_dir=str(DBT_PROFILES_DIR),
environment=str(DBT_TARGET),
overwrite_profiles=False,
helper_script=f"cd {DBT_ROOT}",
return_all=True,
stream_output=True,
state_handlers=[post_to_slack],
)
If I run dbt outside Prefect, i.e. dbt build --target=$DBT_TARGET
it works as expected. Am I using the wrong variable for target
?Scarlett King
08/03/2022, 1:18 PMBenjamin.bgx
08/03/2022, 1:44 PMMatt Delacour
08/03/2022, 2:30 PMRio McMahon
08/03/2022, 2:41 PMessential container in task exited
with no generated logs. The entrance command ["/bin/sh", "-c", "prefect execute flow-run"]
was failing because (I think) prefect 2.0 doesn’t have an execute
command. I resolved this by pinning my prefect dependency to pip install "prefect[aws, gitlab]==1.3.0"
.Rajvir Jhawar
08/03/2022, 2:55 PMfrom prefect import flow, get_run_logger
@flow(name="test", version="1.0beta")
def main():
logger = get_run_logger()
<http://logger.info|logger.info>(f"Hello World")
if __name__ == "__main__":
main()
Flow model details:
id=UUID('120dea46-79e8-4f42-99f0-7e300a2b3cb9') name='test' tags=[]
This is big blocker for me as I want to the create_flow_run
method, but it errors out after you give it a flow model without a version in it.
Any suggestion on what i am doing wrong here?Chu
08/03/2022, 3:13 PMMyParams = Parameter('My_Param', default="my_input")
, in Prefect UI we can input some parameters to overwrite the default at run time, my question is: is there a way to get the run time parameter input (not the MyParams object, but the actual input)?
In our case, the default
is a list (where we pass a list of ids), we need to customize the flow run name by extracting each id from MyParams
and formalizing a new list (child flow run name) for map
function to work.
From what I understand and tested, MyParams
is a serialized object, to access its input first needs to deserialize it or using another way, we cannot just loop over it using [i+'_customzied_string' for i in MyParams]
Tim Enders
08/03/2022, 3:26 PMTim Enders
08/03/2022, 3:29 PMMatt Delacour
08/03/2022, 3:32 PMresult
is about ...
ThanksAmol Shirke
08/03/2022, 3:40 PMHarald Kirkerød
08/03/2022, 3:51 PMben
08/03/2022, 4:04 PMKha Nguyen
08/03/2022, 5:18 PMfrom prefect import flow, task, get_run_logger
from prefect_ray import RayTaskRunner
@task
def generate_numbers(n: int) -> list[int]:
return list(range(n))
@task
def double_the_numbers(d: int) -> int:
return d * 2
@flow(
task_runner=RayTaskRunner(
address="<anyscale://singularity/demo1>",
init_kwargs={"runtime_env": {"working_dir": "."}},
)
)
def run_workflow():
logger = get_run_logger()
<http://logger.info|logger.info>("Hello World!")
numbers = generate_numbers.submit(100)
<http://logger.info|logger.info>(numbers.result())
doubled = double_the_numbers.map(numbers)
<http://logger.info|logger.info>([d.wait().result() for d in doubled])
run_workflow()
ray::begin_task_run() (pid=20006, ip=<http://xxx.xx.xx.xxx|xxx.xx.xx.xxx>)
File "/opt/homebrew/Caskroom/miniforge/base/envs/singularity2/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
File "/home/ray/anaconda3/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/home/ray/anaconda3/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/home/ray/anaconda3/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/home/ray/anaconda3/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/ray/anaconda3/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/home/ray/anaconda3/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/home/ray/anaconda3/lib/python3.10/site-packages/prefect/engine.py", line 946, in begin_task_run
return await orchestrate_task_run(
File "/home/ray/anaconda3/lib/python3.10/site-packages/prefect/engine.py", line 1083, in orchestrate_task_run
await _persist_serialized_result(
File "/home/ray/anaconda3/lib/python3.10/site-packages/prefect/results.py", line 15, in _persist_serialized_result
await filesystem.write_path(key, content)
File "/home/ray/anaconda3/lib/python3.10/site-packages/prefect/filesystems.py", line 162, in write_path
path.parent.mkdir(exist_ok=True, parents=True)
File "/home/ray/anaconda3/lib/python3.10/pathlib.py", line 1177, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/home/ray/anaconda3/lib/python3.10/pathlib.py", line 1177, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/home/ray/anaconda3/lib/python3.10/pathlib.py", line 1177, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/home/ray/anaconda3/lib/python3.10/pathlib.py", line 1173, in mkdir
self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/Users'
@Taylor Curran @Kalise RichmondStefano De Feo
08/03/2022, 5:44 PMKhuyen Tran
08/03/2022, 6:29 PMStephen Herron
08/03/2022, 6:53 PM