Rajvir Jhawar
08/08/2022, 5:17 AMDanny Zhang
08/08/2022, 6:07 AMDanny Zhang
08/08/2022, 6:07 AMHa Pham
08/08/2022, 10:03 AMAnton L.
08/08/2022, 9:06 PMJeffery Newburn
08/08/2022, 10:41 PMEmil Ordoñez
08/11/2022, 5:27 AMfrom prefect import task, flow, get_run_logger
@task
def get_x(x: int=1):
return x
@task
def sum_task(x: int=1, y: int=2):
return x + y
@flow
def flow_exec(init: int=1, end: int=4):
i=1
x1 = get_x(i)
sum1 = sum_task(x1)
i = i+1
x2=get_x(i)
sum2 = sum_task(x2)
i = i+1
x3=get_x(i)
sum3 = sum_task(x3)
i = i+1
x4=get_x(i)
sum4 = sum_task(x4)
flow_exec()
I've coded in this ways (Xn variables) instead of appending variables to a list for simplifying the code.
This is just a simple equivalent recreation of a real use case that I have that consists of iterating through a list of integrator.io flows and running them from Prefect instead of using the integrated scheduler on integrator.io
I'm having this same issue on my real case. I'm putting this here to see if you can help me understand this Radar UI, maybe I'm just not getting the whole idea, or maybe I can help troubleshooting or providing ideas.
Sorry for the long text, I just wanted to be very explicit.Thomas Agung Santoso
08/12/2022, 6:31 AMDarren
08/12/2022, 7:43 PMMarwan Sarieddine
08/13/2022, 6:28 PMkwmiebach
08/15/2022, 2:32 PMkwmiebach
08/15/2022, 2:33 PMPREFECT_ORION_API_HOST=0.0.0.0 prefect orion start
Jaime Raldua Veuthey
08/16/2022, 6:49 AMIf no storage is explicitly configured, Prefect will useIf I addstorage by default. Local storage works fine for many local flow run scenarios, especially when testing and getting started. However, due to the inherit lack of portability, many use cases are better served by using remote storage such as S3 or Google Cloud Storage.LocalFileSystem
--storage-block s3/example-block
when doing prefect deployment build
I get a copy of all my files on my S3 bucket. But how can I ensure that the Deployment in Prefect Cloud always takes the flow codes from the S3 bucket?Dennis Hinnenkamp
08/16/2022, 11:05 AMAniruddha Sengupta
08/16/2022, 4:05 PMcompetitor
, lang_code
, model
and work_dir
@task(name="test_eval", log_stdout=True)
def run_test(
competitor: str,
lang_code: str,
testset_config: dict,
model: str,
work_dir: str
):
testset_path = testset_config[lang_code]
savepath = f"/exp/aniruddhas/comp_eval/test_outputs/2022-08-16/test_dynamic_outputs/{competitor}_{testset_path}_{lang_code}_{model}.txt"
return f'aladdin/evaluation/test.sh --competitor {competitor} --testset_path "{testset_path}" --lang_code {lang_code} --model "{model}" --work_dir "{work_dir}" | tee {savepath}'
From here, I have a list of competitors and a list of languages. I want to be able to create a Shell Prefect task but mapped over a list of competitors and languages. So far I have this:
shell_result = shell_task(
command=run_test.map(
competitors,
languages,
testset_config=unmapped(testset_config),
model=unmapped(model),
work_dir=unmapped(work_dir)
)
)
But this doesnt seem to work. I want a ShellTask to be able to iterate through each language for each competitor. How can I write this?kwmiebach
08/17/2022, 8:11 AMkwmiebach
08/17/2022, 8:11 AMkwmiebach
08/17/2022, 8:12 AMkwmiebach
08/17/2022, 8:15 AMkwmiebach
08/17/2022, 8:17 AMkwmiebach
08/17/2022, 8:18 AMfrom prefect import flow
@flow(name="Prefect 2.0 Flow")
def flow_prefect():
print("Hello Prefect 2.0!")
import time
time.sleep(5)
print("Good Bye Prefect 2.0!")
flow_prefect()
kwmiebach
08/17/2022, 8:20 AMkwmiebach
08/17/2022, 8:32 AMPREFECT_ORION_API_HOST="127.0.0.1"
and start the server with prefect orion start --host 0.0.0.0
kwmiebach
08/17/2022, 8:48 AMkwmiebach
08/17/2022, 9:12 AMMohammad Tami
08/17/2022, 2:43 PMprefect deployment build ./flows/my_awesome_flow.py:my_stars -n flow1 -t dev -sb s3/s3test
I got the following error:
RuntimeError: File system created with scheme 's3' from base path '<s3://my_s3_bucket>' could not be created. You are likely missing a Python module required to use the given storage protocol.
the error does not have enough info what module I miss.
any idea is appreciated ?Jaime Raldua Veuthey
08/17/2022, 4:24 PMManh-Ha
08/18/2022, 12:12 PM$ prefect deployment build ./expenses.py:export_raw_expenses -n export-raw-expenses -t kubernetes -ib kubernetes-job/job --storage-block azure/azure-storage
Found flow 'export-raw-expenses'
Successfully uploaded 2 files to <az://prefect/htoh-prefect>
Traceback (most recent call last):
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/prefect/cli/deployment.py", line 477, in build
infrastructure = await Block.load(infra_block)
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/prefect/blocks/core.py", line 514, in load
return cls._from_block_document(block_document)
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/prefect/blocks/core.py", line 415, in _from_block_document
block = block_cls.parse_obj(block_document.data)
File "pydantic/main.py", line 521, in pydantic.main.BaseModel.parse_obj
File "/Users/manhha/Library/Caches/pypoetry/virtualenvs/htoh-prefect-8QVbx3Ri-py3.10/lib/python3.10/site-packages/prefect/blocks/core.py", line 103, in __init__
super().__init__(*args, **kwargs)
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for KubernetesJob
customizations
string indices must be integers (type=type_error)
It looks like the string that the system doesn't like the string I input to the Customizations field:
{
"op": "add",
"path": "/spec/template/spec/containers/0/imagePullPolicy",
"value": "Always"
}
I tried to consult the doc but it only mention the JSON 6902 patch, but nothing more. There is however an example in the Github repo of Prefect.
Do you know if I'm missing something?Saman
08/19/2022, 7:44 AMbitsofinfo
08/22/2022, 9:14 PMui.apolloApiUrl
my UI is reporting it fails to connect Couldn't connect to *Prefect Server* at *<http://localhost:4200/graphql>*
if i describe the deployment I see PREFECT_SERVER__APOLLO_URL is set = what I specified in ui.apolloApiUrl