Prefect server stopped working and not responding tried `prefect server stop` and then `prefect ser...

09/15/2021, 9:08 PM

Prefect server stopped working and not responding tried

prefect server stop

and then

prefect server start --detach --use-volume

but it get stuck

Copy code

$ prefect server start --detach --use-volume
Pulling postgres ... done
Pulling hasura   ... done
Pulling graphql  ... done
Pulling towel    ... done
Pulling apollo   ... done
Pulling ui       ... done
Creating network "prefect-server" with the default driver
temp_data_processing_postgres_1 is up-to-date
Creating temp_data_processing_hasura_1 ... done
temp_data_processing_graphql_1 is up-to-date
Creating temp_data_processing_towel_1  ... done
Creating temp_data_processing_apollo_1 ... done
Creating temp_data_processing_ui_1     ... done

does not detach on on the UI I get

Kevin Kho

09/15/2021, 9:10 PM

Did you upgrade Prefect by chance? Asking because 0.15.5 has different behavior

09/15/2021, 9:12 PM

does it do it automatically in some way ?

Kevin Kho

09/15/2021, 9:13 PM

No the opposite. There were breaking changes there for some users in terms of accessing the API. I think your API may have failed to boot up? Also check the memory of the machine?

09/15/2021, 9:13 PM

stopping and trying to restart again I get

Copy code

$ prefect server start --detach --use-volume
Pulling postgres ... done
Pulling hasura   ... done
Pulling graphql  ... done
Pulling towel    ... done
Pulling apollo   ... done
Pulling ui       ... done
Creating network "prefect-server" with the default driver
Starting temp_data_processing_postgres_1 ... error

ERROR: for temp_data_processing_postgres_1  Cannot start service postgres: network <some code> not found

ERROR: for postgres  Cannot start service postgres: network <some code> not found
ERROR: Encountered errors while bringing up the project.

Kevin Kho

09/15/2021, 9:14 PM

Will ask the team for ideas

09/15/2021, 9:15 PM

this is the production server BTW…

09/15/2021, 9:18 PM

I also have this in the crontab

Copy code

0 8 * * * docker exec -it temp_data_processing_postgres_1 sh -c "psql PGPASSWORD=<PW> -U prefect -d prefect_server -c \"DELETE FROM log WHERE timestamp < NOW() - INTERVAL '14 days';\" "

Kevin Kho

09/15/2021, 9:27 PM

would you be able to check the postgres container for logs?

Zanie

09/15/2021, 9:28 PM

<some code>

the literal output from docker-compose?

upvote 1

09/15/2021, 9:53 PM

I stopped then restarted without

--use-volume

is started fine then stopped again Cleaned some data from the computer and started with

--use-volume

now it appears to be OK

👍 1

09/15/2021, 10:22 PM

when doing

df -h

I see

09/15/2021, 10:23 PM

looks like it is using 34GB is this a Prefect process ? does 34GB make sense ?

09/15/2021, 10:27 PM

I reduced the days in the log to only 5

09/15/2021, 11:25 PM

still not work well the agent having problem to refresh connection

09/15/2021, 11:25 PM

the

Last queried

does not appear to be resetting

09/15/2021, 11:28 PM

the agent

Last queried

keep getting larger and then the flows get heartbeat error

Kevin Kho

09/16/2021, 12:49 AM

are you using a DaskExecutor? This is when you run a flow right? Is the flow memory intensive? Do you have a very large map operation (10000) tasks?

09/16/2021, 4:18 PM

task running on the same time are in the order of several 10th

09/16/2021, 4:21 PM

maybe even less

09/16/2021, 4:41 PM

is it expected to take 34GB ?

Kevin Kho

09/16/2021, 4:49 PM

34GB for a flow run? It’s hard to say without knowing the the flow is doing. Are you working with DataFrames? Or do you have a large mapped task?

09/16/2021, 6:16 PM

not on the VM that runs prefect

09/16/2021, 6:16 PM

all the jobs are running on a different machine

09/16/2021, 6:17 PM

Prefect imports the functions and runs them or runs

os.system(cmd)

Zach Angell

09/16/2021, 9:27 PM

Hi @YD -

overlay

is not a Prefect process. I'm not sure how much space it should be using. There is a long thread here about cleaning up / identifying contents of

/var/lib/docker/overlay

that might be helpful https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604. Happy to help debug a bit, but it's hard without more detail

3 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.