https://prefect.io logo
Channels
Powered by
y

YD

09/15/2021, 9:08 PM
Prefect server stopped working and not responding tried 
prefect server stop
and then 
prefect server start --detach --use-volume
but it get stuck
Copy code
$ prefect server start --detach --use-volume
Pulling postgres ... done
Pulling hasura   ... done
Pulling graphql  ... done
Pulling towel    ... done
Pulling apollo   ... done
Pulling ui       ... done
Creating network "prefect-server" with the default driver
temp_data_processing_postgres_1 is up-to-date
Creating temp_data_processing_hasura_1 ... done
temp_data_processing_graphql_1 is up-to-date
Creating temp_data_processing_towel_1  ... done
Creating temp_data_processing_apollo_1 ... done
Creating temp_data_processing_ui_1     ... done
does not detach on on the UI I get
k

Kevin Kho

09/15/2021, 9:10 PM
Did you upgrade Prefect by chance? Asking because 0.15.5 has different behavior
y

YD

09/15/2021, 9:12 PM
no
does it do it automatically in some way ?
k

Kevin Kho

09/15/2021, 9:13 PM
No the opposite. There were breaking changes there for some users in terms of accessing the API. I think your API may have failed to boot up? Also check the memory of the machine?
y

YD

09/15/2021, 9:13 PM
stopping and trying to restart again I get
Copy code
$ prefect server start --detach --use-volume
Pulling postgres ... done
Pulling hasura   ... done
Pulling graphql  ... done
Pulling towel    ... done
Pulling apollo   ... done
Pulling ui       ... done
Creating network "prefect-server" with the default driver
Starting temp_data_processing_postgres_1 ... error

ERROR: for temp_data_processing_postgres_1  Cannot start service postgres: network <some code> not found

ERROR: for postgres  Cannot start service postgres: network <some code> not found
ERROR: Encountered errors while bringing up the project.
k

Kevin Kho

09/15/2021, 9:14 PM
Will ask the team for ideas
y

YD

09/15/2021, 9:15 PM
this is the production server BTW…
I also have this in the crontab
Copy code
0 8 * * * docker exec -it temp_data_processing_postgres_1 sh -c "psql PGPASSWORD=<PW> -U prefect -d prefect_server -c \"DELETE FROM log WHERE timestamp < NOW() - INTERVAL '14 days';\" "
k

Kevin Kho

09/15/2021, 9:27 PM
would you be able to check the postgres container for logs?
z

Zanie

09/15/2021, 9:28 PM
Is 
<some code>
the literal output from docker-compose?
upvote 1
y

YD

09/15/2021, 9:53 PM
I stopped then restarted without 
--use-volume
is started fine then stopped again Cleaned some data from the computer and started with 
--use-volume
now it appears to be OK
👍 1
when doing 
df -h
I see
looks like it is using 34GB is this a Prefect process ? does 34GB make sense ?
I reduced the days in the log to only 5
still not work well the agent having problem to refresh connection
the 
Last queried
does not appear to be resetting
the agent 
Last queried
keep getting larger and then the flows get heartbeat error
k

Kevin Kho

09/16/2021, 12:49 AM
are you using a DaskExecutor? This is when you run a flow right? Is the flow memory intensive? Do you have a very large map operation (10000) tasks?
y

YD

09/16/2021, 4:18 PM
task running on the same time are in the order of several 10th
maybe even less
is it expected to take 34GB ?
k

Kevin Kho

09/16/2021, 4:49 PM
34GB for a flow run? It’s hard to say without knowing the the flow is doing. Are you working with DataFrames? Or do you have a large mapped task?
y

YD

09/16/2021, 6:16 PM
not on the VM that runs prefect
all the jobs are running on a different machine
Prefect imports the functions and runs them or runs 
os.system(cmd)
z

Zach Angell

09/16/2021, 9:27 PM
Hi @YD - 
overlay
is not a Prefect process. I'm not sure how much space it should be using. There is a long thread here about cleaning up / identifying contents of 
/var/lib/docker/overlay
that might be helpful https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604. Happy to help debug a bit, but it's hard without more detail
2 Views
Powered by