Jacob Goldberg
08/02/2022, 5:44 PM404 Client Error for <http+docker://localhost/v1.41/containers/create?name=elastic-salamander>: Not Found ("No such image: <http://26123096XXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest-165946XXXX%22|26123096XXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest-165946XXXX">)
I can go on AWS ECR, see the container, and see that the URI matches with the one from the error so it does exist…I have changed some settings recently regarding the python version and docker build, but cannot understand why i would get a container not found error
edit: formattingharis khan
08/02/2022, 5:51 PMChu
08/02/2022, 6:36 PMMansour Zayer
08/02/2022, 8:50 PMfrom underlying_flows.flow_a import flow_a
flow_a.register(project...)
With Flow(...):
...
Because when we do that, prefect throws a warning UserWarning: Attempting to call "flow.register" during execution of flow file will lead to unexpected results.
John Mizerany
08/02/2022, 9:19 PM--cli-input-json
tag for cli, should there be an extra argument in the prefect-agent-td.json
file called agentRoleArn
? There are two for executionRoleArn
and taskRoleArn
but if the agent itself needs a Role in AWS should that be specified or can it be included in the other variables?Billy McMonagle
08/02/2022, 9:25 PMHafsa Junaid
08/02/2022, 9:48 PMprefect deployment build ./sample.py:samplefunc --name "ALS Deployment" --storage-block recommenderprefectserver
Error: Deployment.yaml is not created with following exceptionTom Kaszemacher
08/02/2022, 10:02 PM@task()
def get_inputs() -> List:
# This task retrieves a list of inputs to process from a DB and returns it
pass
with Flow(name='dispatcher') as dispatcher:
inputs = get_inputs()
for i in inputs:
create_flow_run('worker', parameters={'input': i})
It doesn’t work as I need to provide either nouts or return a Tuple in order to iterate over my inputs, however I don’t know the size of it.
What would be a correct approach here?
Thanks!André Bonatto
08/02/2022, 10:11 PMHafsa Junaid
08/02/2022, 10:54 PMCA Lee
08/03/2022, 2:14 AMHafsa Junaid
08/03/2022, 2:45 AMHafsa Junaid
08/03/2022, 2:48 AMwonsun
08/03/2022, 3:18 AMRajvir Jhawar
08/03/2022, 7:07 AMVersion: 2.0.2
API version: 0.8.0
Python version: 3.9.13
I having problems with both the read_flow_by_name
and read_deployment_by_name
. Neither of these method is working in when i try to use them:
Sample code:
from prefect.client import get_client
import asyncio
import json
async def main():
async with get_client() as client:
data = await client.read_flow_by_name(flow_name='test_flow')
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Error:
Traceback (most recent call last):
File
data = await client.read_flow_by_name(flow_name='test_flow')
/.venv/lib/python3.9/site-packages/prefect/client.py", line 508, in read_flow_by_name
return schemas.core.Deployment.parse_obj(response.json())
File "pydantic/main.py", line 521, in pydantic.main.BaseModel.parse_obj
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for Deployment
flow_id
field required (type=value_error.missing)
manifest_path
field required (type=value_error.missing)
Viet Nguyen
08/03/2022, 7:12 AMOscar Björhn
08/03/2022, 8:15 AMT B
08/03/2022, 10:04 AMAlvaro Durán Tovar
08/03/2022, 10:14 AMharis khan
08/03/2022, 10:17 AMShrikkanth
08/03/2022, 11:34 AMAmol Shirke
08/03/2022, 12:43 PMAmol Shirke
08/03/2022, 12:44 PMPaul Lucas
08/03/2022, 12:45 PMdbt_shell_task = DbtShellTask(
profiles_dir=str(DBT_PROFILES_DIR),
environment=str(DBT_TARGET),
overwrite_profiles=False,
helper_script=f"cd {DBT_ROOT}",
return_all=True,
stream_output=True,
state_handlers=[post_to_slack],
)
If I run dbt outside Prefect, i.e. dbt build --target=$DBT_TARGET
it works as expected. Am I using the wrong variable for target
?Scarlett King
08/03/2022, 1:18 PMBenjamin.bgx
08/03/2022, 1:44 PMMatt Delacour
08/03/2022, 2:30 PMRio McMahon
08/03/2022, 2:41 PMessential container in task exited
with no generated logs. The entrance command ["/bin/sh", "-c", "prefect execute flow-run"]
was failing because (I think) prefect 2.0 doesn’t have an execute
command. I resolved this by pinning my prefect dependency to pip install "prefect[aws, gitlab]==1.3.0"
.Rajvir Jhawar
08/03/2022, 2:55 PMfrom prefect import flow, get_run_logger
@flow(name="test", version="1.0beta")
def main():
logger = get_run_logger()
<http://logger.info|logger.info>(f"Hello World")
if __name__ == "__main__":
main()
Flow model details:
id=UUID('120dea46-79e8-4f42-99f0-7e300a2b3cb9') name='test' tags=[]
This is big blocker for me as I want to the create_flow_run
method, but it errors out after you give it a flow model without a version in it.
Any suggestion on what i am doing wrong here?Chu
08/03/2022, 3:13 PMMyParams = Parameter('My_Param', default="my_input")
, in Prefect UI we can input some parameters to overwrite the default at run time, my question is: is there a way to get the run time parameter input (not the MyParams object, but the actual input)?
In our case, the default
is a list (where we pass a list of ids), we need to customize the flow run name by extracting each id from MyParams
and formalizing a new list (child flow run name) for map
function to work.
From what I understand and tested, MyParams
is a serialized object, to access its input first needs to deserialize it or using another way, we cannot just loop over it using [i+'_customzied_string' for i in MyParams]
Chu
08/03/2022, 3:13 PMMyParams = Parameter('My_Param', default="my_input")
, in Prefect UI we can input some parameters to overwrite the default at run time, my question is: is there a way to get the run time parameter input (not the MyParams object, but the actual input)?
In our case, the default
is a list (where we pass a list of ids), we need to customize the flow run name by extracting each id from MyParams
and formalizing a new list (child flow run name) for map
function to work.
From what I understand and tested, MyParams
is a serialized object, to access its input first needs to deserialize it or using another way, we cannot just loop over it using [i+'_customzied_string' for i in MyParams]
Anna Geller
08/03/2022, 4:12 PMBlake Enyart
08/03/2022, 8:13 PMprefect.context.parameters.get("<parameter name>")
. However, when I try to use that syntax (in a flow which is triggered by the Prefect UI and run via a Universal Run agent) I am getting the error that the .parameters
attribute is not found as part of the .context
module.
I see the documentation which indicates it should be there on the context at runtime, but here is a screenshot attached to this message that shows the intellisense code completion options I have in VS Code with the parameters function missing.
Is there something I and @Chu might be missing in all this where the goal is to directly access the list used in the run in the rest of the code (i.e. if I run the flow with the defaults it grabs that and uses it in the rest of the code, but if I pass in a different value at runtime to the Prefect flow it will use the updated values in the rest of the code instead of the default values)Chu
08/03/2022, 8:18 PMAnna Geller
08/03/2022, 8:36 PMBlake Enyart
08/03/2022, 8:50 PMtableau_online_datasources
do I do tableau_online_datasources.result.value
?Anna Geller
08/04/2022, 9:35 AMChu
08/04/2022, 12:08 PMwith Flow("My flow") as flow:
urls = Parameter("urls", default=['url1', 'url2', 'url3'])
result = [url+'_string' for url in urls]
This behavior will give an error said, Task is not iterable.. I understand urls is a serialized Object, is there a way to diseralize it to retrieve the run time value (which maybe something different than default)Anna Geller
08/04/2022, 12:11 PMChu
08/04/2022, 12:26 PMBlake Enyart
08/04/2022, 5:25 PMAnna Geller
08/04/2022, 5:33 PMBlake Enyart
08/04/2022, 5:43 PM