https://prefect.io logo
Channels
Powered by
e

E Li

03/16/2022, 8:13 PM
Hi, I am running on prefect UI and getting this attribute error: can't get attribute 'get_target_name" on <module prefect_scripts.tasks.trade_cache_tasks'. the trade_cache_task is like below, similar structure works fine on other scripts, anyone has any ideas?
Copy code
def get_target_name(x, **kwargs):
	…
	return target_name

def get_task_name(x, **kwargs):
	…
	return task_name

@task(task_run_name=get_task_name,target=get_target_name,checkpoint=True,result=LocalResult())
def task_a(x, y, z):
	…
	return …
k

Kevin Kho

03/16/2022, 8:23 PM
Looking into this
how do you import these in your flow file?
e

E Li

03/16/2022, 8:30 PM
they are on the same script
and it works well with flow.run() on my local win machine
k

Kevin Kho

03/16/2022, 8:34 PM
What storage are you using?
e

E Li

03/16/2022, 8:34 PM
local
k

Kevin Kho

03/16/2022, 8:34 PM
Do you other scripts use mapping too? Are you using the LocalDaskExecutor here?
e

E Li

03/16/2022, 8:35 PM
yes, other scripts use mapping too and daskExecutor
k

Kevin Kho

03/16/2022, 8:42 PM
I am pretty confused by this. I think i’d need a reproducible example to see because the same setup works for your other scripts right?
e

E Li

03/16/2022, 8:45 PM
correct. so if i have a dask scheduler and a few dask workers, and prefect server is running on the scheduler, do i need to have the same python scripts saved on each worker?
k

Kevin Kho

03/16/2022, 8:47 PM
No because the work gets sent to the Dask scheduler and sent to a worker but all the dependencies of the work are serialized together and sent together with cloudpickle so you just need them in memory
e

E Li

03/17/2022, 3:39 PM
so i ended up with redeploy my code, and it just works, not sure exactly what went wrong before
k

Kevin Kho

03/17/2022, 3:40 PM
Oh ok that’s good to know. I guess this may happen when the code is out of sync with the registered version
e

E Li

03/17/2022, 4:05 PM
on a different topic, is graphql-engine v1.3.3 compatible with other images at version of 0.15.5?
k

Kevin Kho

03/17/2022, 4:07 PM
I think so
e

E Li

03/17/2022, 5:17 PM
Copy code
i got this error when register flow with v1.3.3
File "/home/monjuu-g/dev/conda-pyFullReval/totoro_fv_env/lib/python3.6/site-packages/prefect/client/client.py", line 569, in graphql raise ClientError(result["errors"]) prefect.exceptions.ClientError: [{'message': '[{\'extensions\': {\'path\': \'$.selectionSet.flow_group.args.where._and[0].tenant_id._eq\', \'code\': \'validation-failed\'}, \'message\': \'unexpected null value for type "uuid"\'}]', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_flow_from_compressed_string'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '[{\'extensions\': {\'path\': \'$.selectionSet.flow_group.args.where._and[0].tenant_id._eq\', \'code\': \'validation-failed\'}, \'message\': \'unexpected null value for type "uuid"\'}]'}}}]
i have a tenant, do i also need to assign an uuid to it?
k

Kevin Kho

03/17/2022, 5:19 PM
I think this is something like the migrations did not run. Did you upgrade and then this happened?
e

E Li

03/17/2022, 5:21 PM
yeah, i took down the containers, updated docker-compose.yml and ran sudo dock-compose -f ... --env-file ... up -d
k

Kevin Kho

03/17/2022, 5:22 PM
Can you try running the migrations manually like mentioned here ?
e

E Li

03/17/2022, 5:53 PM
database updated successfully with docker exec graphqlContainerName prefect-server database upgrade -y, but still got the error
k

Kevin Kho

03/17/2022, 5:56 PM
Will have to ask the team aviyt tgus
e

E Li

03/17/2022, 6:05 PM
figured it out, so the tenant for that flow is null
that project*
k

Kevin Kho

03/17/2022, 6:07 PM
You have more than one tenant?
e

E Li

03/17/2022, 6:10 PM
i think the UI is not connected to hasura, i can register the flow but can only see it in hasura with queries not on UI, likewise, the project i created on UI has tenant as null
so as you said, it's like the db is not connected?
k

Kevin Kho

03/17/2022, 6:23 PM
UI might not be connected to the database? The interactive API is a good way to test
e

E Li

03/17/2022, 6:26 PM
ok i am confused, interactive api can query the flows and everything, but flows are not shown up on ui
k

Kevin Kho

03/17/2022, 6:28 PM
Maybe you can open the console of your browser to check for errors? How many tenants do you have?
e

E Li

03/17/2022, 6:32 PM
just 1 tenant
k

Kevin Kho

03/17/2022, 6:36 PM
Are there any errors when you open the browser console?
e

E Li

03/17/2022, 6:44 PM
exception: 'unexpected null value for type \"uuid\", code: "validation-failed"
k

Kevin Kho

03/17/2022, 7:10 PM
Asking the team
e

E Li

03/17/2022, 7:12 PM
thank you
a

Anna Geller

03/17/2022, 7:19 PM
Some questions for troubleshooting: 1. Did your agent and flow registration environment get upgraded too? 2. Doing DB migration was the right thing to do, but did you restart your Apollo container after the migration? You need to do it so that Apollo picks up the changes
Copy code
docker exec graphqlContainerName prefect-server database upgrade -y

docker restart apolloContainerName
e

E Li

03/17/2022, 7:25 PM
1. i only updated graphql-engine, the other images and environment are all in 0.15.5. I can see my agent, but just cannot see my flows
2. I just executed the 2 cmd, still not able to see flows
a

Anna Geller

03/17/2022, 7:30 PM
ideally, when running Prefect with docker-compose (the default setup), you should normally have all images running the same "Prefect Core" version. This is what it typically looks like:
any reason why you want to run mixed versions or why you can't start all services from scratch? can you explain your full setup - e.g. what is the full command you used to start your Server services? Did you attach a volume to your Postgres docker DB?
e

E Li

03/17/2022, 7:38 PM
I just the changed image version for graphql-engine as the old one is no longer available on our internal site. i brought down the containers by 
docker-compose -f docker-compose.yml --env-file env down
, modified the version in docker-compose.yml and start the containers by 
docker-compose -f docker-compose.yml --env-file env up -d
a

Anna Geller

03/17/2022, 7:38 PM
can you share that docker compose file?
e

E Li

03/17/2022, 7:43 PM
a

Anna Geller

03/17/2022, 7:51 PM
sorry, it's huge 🙈 can you perhaps say what did you change as compared to the default configuration? still don't entirely understand why you can't use the default command:
Copy code
prefect server start
It would be easier if you shared the config command you used to generate this file to understand the changes you made
Copy code
prefect server config --help
also - any reasons why you can't upgrade to the latest Prefect Core 1.1? I saw you are still using Python 3.6 - with that the highest version of Prefect you can get is the one you have right now. But if you upgrade Python to 3.7 or higher, you can get Prefect 1.1 which has many improvements
Copy code
/home/monjuu-g/dev/conda-pyFullReval/totoro_fv_env/lib/python3.6
so the best solution would probably be just start from scratch in a fresh environment with Python >= 3.7, reinstall Prefect and start all server components again and you can still mount the same Postgres volume so that you don't lose your data. Then run DB migrations and restart Apollo and it should work in theory:
Copy code
docker exec graphqlContainerName prefect-server database upgrade -y

docker restart apolloContainerName
e

E Li

03/17/2022, 8:03 PM
this yml has been working fine in the past, the only thing i changed today is from 
image: "${DOCKER-REGISTRY}/hasura/graphql-engine:v1.3.1
to 
mage: "${DOCKER-REGISTRY}/hasura/graphql-engine:latest
, basically just a different version of graphql image. it's probably then just a compatible issue with python?
a

Anna Geller

03/17/2022, 8:12 PM
what was the reason for that change?
e

E Li

03/17/2022, 8:16 PM
it's an internal update which removed the old version, i have asked to revert that back, the new one seems not working well
a

Anna Geller

03/17/2022, 8:20 PM
Gotcha. there must be some version incompatibility then. In general, we test the components so that the default versions picked up based on your Python and Prefect versions should work out of the box. And then it should work after running DB migrations. Keep us posted
e

E Li

03/17/2022, 8:27 PM
sure, is there a reference table to look up the default version based on my python version?
k

Kevin Kho

03/17/2022, 8:28 PM
Prefect 1.0 dropped support for Python 3.6 but is tested across 3.7-3.9. 0.15.0 and above should be compatible with 3.6-3.9 i think
a

Anna Geller

03/17/2022, 8:31 PM
☝️ is correct for Prefect core. If you're asking about Server components, each release has a docker-compose file as in here. E.g. the latest Server release uses GraphQL 2.1.1
k

Kevin Kho

03/17/2022, 8:35 PM
It looks like graphql engine 2.1.1 is tied to 0.15.13 based on this PR.
4 Views
Powered by