https://prefect.io
Join Slack
Hi! I am trying to do a PoC on prefect with backend = server. Running prefect-server using docker-co...
# prefect-ui
j
Hi! I am trying to do a PoC on prefect with backend = server. Running prefect-server using docker-compose on Mac M1. I was able to run a sample flow but the UI does not list my registered flow or the runs that happened. I am able to get the flows with graphql queries though. Attaching screenshots.
these are the images that Im running:
Copy code
image: prefecthq/apollo
image: prefecthq/server:core-0.15.10
image: hasura/graphql-engine:v2.1.0-beta.3.arm64
image: postgres:11
image: prefecthq/server
image: prefecthq/ui
(I was ablt to get the hasura engine run only using this arm64 image.)
a
1. How do you start your server? 2. Did you switch to Server backend? 3. What is your agent setup? 4. How did you configure flow storage?
j
docker-compose.yaml
docker-compose.yaml
1. prefect server started using the docker-compose file.
2, 3. yes. had run 
prefect server backend
and started local agent using 
prefect agent local start --name "Default Agent
after doing 
docker compose up
4. haven't configured flow storage
a
I see. I can try to reproduce if you share your flow.
j
thanks Anna. this is what im running
Copy code
import prefect
from prefect import task, Flow
from prefect.storage import Local

@task
def say_hello():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Cloud!")

with Flow("hello-flow", storage=Local()) as flow:
    say_hello()

storage=Local()

# Register the flow under the "tutorial" project
flow.register(project_name="tutorial")
flow.storage.build()

flow.run()
just tried this using local storage for flow, still the same result
also, im only seeing flow runs on the dashboard if I run it from the dashboard. if im doing it from my notebook, it succeeds, but doesnt come up on the dash.
a
if you run it locally in a notebook with flow.run(), it’s running only locally, so without backend. But if you register your flow and then run using CLI/UI/API, e.g.:
Copy code
prefect run --name yourflowname --project yourproject --watch
then you’ll see it in the UI because it runs with a backend
btw, your agent has no labels which may cause troubles down the road - it’s better to set labels explicitly on your agent and run config
@Jons Cyriac I could not reproduce your issue. The docker-compose file you shared didn’t work for me, the GraphQL container throws:
Copy code
Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc34448fa10>: Failed to establish a new connection: [Errno 111] Connection refused'))

Could not upgrade the database!

Running Alembic migrations...
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.

Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f89b51e86d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
I would recommend using the approach from the docs:
Copy code
prefect backend server
prefect server start
# in a new terminal window
prefect agent local start --label yourlabel
This should work well. If you need any custom addition to it, have a look at all the override options provided by the CLI:
Copy code
prefect server config --help
another things to watch out for: #1 You don’t need to build storage explicitly because the prefect register does this by default - 
flow.register(build=True)
is the default #2 Ideally, you should not keep 
flow.register(project_name="tutorial")
in your flow file. It’s easier to use CLI for this:
Copy code
prefect register --project xyz -p yourflow.py
📝 1
j
@Anna Geller i was getting this same error when I tried to follow the docs 😕
hasura just wasnt starting up
a
did you allocate enough memory to your Docker Desktop on Mac? I heard some people had issues with Hasura container due to not enough memory on the Docker Desktop with M1
j
no I havent tried that. is there any recommendation on the amount of memory?
a
based on this Hasura issue, they recommend 8 GB
j
thanks, checking this
a
@Jons Cyriac apparently, they released a docker image for M1 today just 2 hours ago 😄 https://github.com/hasura/graphql-engine/issues/6337#issuecomment-990896581
j
yup, saw this yesterday in docker hub. I was using this beta version.
allocating 8gb worked for me.
🙌 1
is this a work around? or its a requirement?
a
I don’t know. Is your memory limited?
in the end, running it on Mac is probably a temporary solution anyways, right? when you want to deploy Server to be used in production, you would probably use some Linux VM and you wouldn’t have this problem any more, correct?
j
thats right. thank you so much!
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by