Alexis Lucido
02/14/2022, 1:37 PMKevin Kho
02/14/2022, 3:02 PMAlexis Lucido
02/14/2022, 5:43 PMKevin Kho
02/14/2022, 5:51 PMAlexis Lucido
02/15/2022, 8:18 AMKevin Kho
02/15/2022, 3:10 PMAlexis Lucido
02/16/2022, 8:07 AMKevin Kho
02/16/2022, 2:33 PMAlexis Lucido
02/16/2022, 4:58 PMKevin Kho
02/16/2022, 5:31 PMAlexis Lucido
02/21/2022, 9:47 AMKevin Kho
02/21/2022, 2:08 PMAlexis Lucido
02/21/2022, 2:47 PMKevin Kho
02/21/2022, 2:52 PMprefect server start
do deploy server?Alexis Lucido
02/21/2022, 4:08 PM# To launch script printing logs to the console and saving them to a deploy_app_logs.txt file as well:
# source deploy_app.sh 2>&1 | ts | tee -a logs/deploy_app.$(date +'%Y-%m-%dT%H:%M:%S').log
# Automatic deployment
# Ignore logging of sudo commands
export HISTIGNORE='*sudo -S*'
### BACKEND ###
conda activate blx-vpp_backend
# Stop prefect then kill processes
prefect server stop
echo "y" | docker system prune
# Kill potential processes in the background
conda deactivate
echo <password> | sudo -S pkill python
sudo pkill -9 supervisord
sudo kill -9 $(pgrep geckodriver)
sudo kill -9 $(pgrep python)
sudo kill -9 $(pgrep prefect)
sudo killall -9 firefox-esr
sudo unlink /tmp/supervisor.sock
conda deactivate
# Delete old github repo
echo <password> | sudo -S rm -r ~/blx-vpp_backend
# Clone git repo after ssh key creation
cd ~
git clone --branch master <git-link>
# Reinstall venv
conda activate blx-vpp_backend
~/miniconda3/envs/blx-vpp_backend/bin/pip3 install -r blx-vpp_backend/requirements.txt --use-deprecated=legacy-resolver
conda deactivate
# Install geckodriver
cd ~/blx-vpp_backend/drivers
wget <https://github.com/mozilla/geckodriver/releases/download/v0.29.0/geckodriver-v0.29.0-linux64.tar.gz>
tar -xvzf geckodriver-v0.29.0-linux64.tar.gz
sudo chmod +x geckodriver
rm geckodriver-v0.29.0-linux64.tar.gz
sudo chmod -R 777 ~/blx-vpp_backend/drivers
# Launch backend
cd ~
conda activate blx-vpp_backend
prefect server start --expose --use-volume >> logs/prefect_server.$(date +'%Y-%m-%dT%H:%M:%S').log 2>&1 &
sleep 20 # To let the server start
cd blx-vpp_backend
supervisord
python main.py &
conda deactivate
Kevin Kho
02/21/2022, 4:11 PMsupervisord
creating these multiple processes? What is in the main.yp
?Alexis Lucido
02/21/2022, 4:13 PMsupervisord
is super basic, I almost did not modify the basic one generated by the Prefect command:; Supervisor config file for running Prefect Agents.
;
; To run for the first time:
; Save this file with the name `supervisord.conf` and run:
; supervisord # will start all programs
; supervisorctl # will open an interactive prompt for inspection
;
; If you need to edit this file and restart:
; supervisorctl reload # will reload config and restart programs
;
; For more information on configuring supervisor, please see:
; <http://supervisord.org/configuration.html>
;
; To configure supervisor logging, please see:
; <http://supervisord.org/logging.html>
;
; To inspect the status of a given program,
; use the interactive supervisorctl command:
; supervisorctl
; fg prefect-agent
;
; To stop all running programs, run:
; supervisorctl stop all
;
[unix_http_server]
file=/tmp/supervisor.sock ; the path to the socket file
[supervisord]
loglevel=debug ; log level; default info; others: debug,warn,trace
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[program:prefect-agent]
command=prefect agent local start
redirect_stderr=true
stdout_logfile=%(ENV_HOME)s/logs/supervisord.log
; you can add as many programs as you'd like
; including multiple Prefect Agents
; [program:foo-agent]
; command=prefect agent start local -l foo
Kevin Kho
02/21/2022, 4:15 PMAlexis Lucido
02/21/2022, 4:17 PMfrom flows import etl_inputs
etl_inputs
looks like this:
with Flow('da_prices', schedule=schedules_prefect['da_prices']) as da_prices:
sql_da_prices = workflows_wrappers.da_prices_(
ftp_1_credentials, ftp_2_credentials)
workflows_load.load_batch(
dict_tn['table_name_da_prices'], sql_da_prices, 'Day-Ahead prices')
with Flow('id_prices', schedule=schedules_prefect['id_prices']) as id_prices:
sql_id_prices = workflows_wrappers.id_prices_(
ftp_1_credentials, ftp_2_credentials)
workflows_load.load_batch(
dict_tn['table_name_id_prices'], sql_id_prices, 'Intraday prices')
flows_dict = {'da_prices': da_prices, 'id_prices': id_prices}
utilities.register_flows(flows_dict, 'etl_inputs')
And utilities.register_flows
(may be the issue) is as such:
from prefect import Client
def register_flows(flows_dict, project_name):
"""
Registers each flow in flows_dict (key: name of the flow, value:
prefect.Flow) in a Prefect project project_name.
Parameters
----------
flows_dict : dict
dict of Prefect Flows.
project_name : str
Name of the project to register Flows into.
Returns
-------
None.
"""
client = Client()
client.create_project(project_name=project_name)
for k, flow in flows_dict.items():
flow.register(project_name=project_name)
Kevin Kho
02/21/2022, 8:51 PMprefect server start --expose
and agent as a separate process or yes on another VM and actually that would let us determine which one is causing the extra processes right?Anna Geller
02/21/2022, 9:07 PMprefect register --project XXX -p path/to/flows/
prefect server config --expose > docker-compose.yml
docker-compose up -d
You can customize the docker-compose file as you wish to have everything in a single script. You can also check out this user contributed docker-compose setup that may make it easier to run your Server on a VM and have it all run as a script:
https://github.com/flavienbwk/prefect-docker-composeAlexis Lucido
02/22/2022, 9:25 AMAnna Geller
03/04/2022, 10:16 AMAlexis Lucido
03/04/2022, 11:26 AMdef register_flows(flows_dict, project_name):
"""
Registers each flow in flows_dict (key: name of the flow, value:
prefect.Flow) in a Prefect project project_name.
Parameters
----------
flows_dict : dict
dict of Prefect Flows.
project_name : str
Name of the project to register Flows into.
Returns
-------
None.
"""
client = Client()
try:
client.create_project(project_name=project_name)
except:
logger.warning(
"Error during project creation. It may already have been created.")
for k, flow in flows_dict.items():
flow.register(project_name=project_name, add_default_labels=False)
So I believe the flows should not be associated with any label. However, it seems a random label is given to the Flow at registration (see the attached picture).
Here is how I define my flow before registering it:
with Flow('id_trades_live', schedule=schedules_prefect[
'id_trades_live']) as id_trades_live:
pass
Here is the docker-compose for launching the agent:
version: "3.7"
services:
agent:
build:
dockerfile: docker/agent/Dockerfile
context: ./
command: bash -c "prefect agent local start --name $$(uuid) --no-hostname-label --label app"
volumes:
- /srv/docker/prefect/flows:/root/.prefect/flows
- type: bind
source: ./docker/agent/config.toml
target: /root/.prefect/config.toml
read_only: true
So there is a label for the agent, and I understand a Flow with no label can only be run by an agent with no label. But the problem lies in the default label attributed to the Flow.
To recap I have server in one container, agent in another container and a client registering the flows in one final container.
Any idea how to solve this? Thank you very much!Kevin Kho
03/10/2022, 3:37 PMflow.storage.add_default_labels=False
or Flow(…,storage=SomeStorage(…, add_default_labels=False)
Alexis Lucido
03/10/2022, 3:42 PMKevin Kho
03/10/2022, 3:46 PMAlexis Lucido
03/10/2022, 4:08 PMwith Flow('ms_det_fc', schedule=schedules_prefect['ms_fc']) as ms_det_fc:
ms_det_fc.storage = Local(add_default_labels=False)
ftd_ms_det_fc = <http://workflows_wrappers.ms|workflows_wrappers.ms>_fc(
'foo', False, ms_ftp_credentials, credentials)
workflows_load.load_batch(dict_tn['table_name_ms'], ftd_ms_det_fc,
'Deterministic forecasts')
Kevin Kho
03/10/2022, 4:12 PMwith Flow(...) as ms_det_fc:
...
ms_det_fc.storage = Local(add_default_labels=False)
Alexis Lucido
03/10/2022, 4:17 PMKevin Kho
03/10/2022, 4:21 PMAlexis Lucido
03/18/2022, 5:07 PMKevin Kho
03/18/2022, 5:09 PM