Adam Brusselback
03/09/2021, 4:35 PMCharles Leung
03/10/2021, 6:45 PMJosh Greenhalgh
03/11/2021, 4:43 PMprefect server start
- traceback in threadale
03/11/2021, 6:57 PMChohang Ng
03/11/2021, 8:20 PMChohang Ng
03/11/2021, 9:55 PMLevi Leal
03/12/2021, 2:29 PMJosh Greenhalgh
03/12/2021, 2:55 PMdef register_deploy():
# pylint: disable=missing-function-docstring
with prefect.context(
config={
"cloud": {"graphql": os.environ["GRPAHQL_ENDPOINT"]},
"server": {"ui": {"endpoint": os.environ["UI_ENDPOINT"]}},
}
):
for flow_name, flow in all_flows.items():
<http://logging.info|logging.info>(f"Registering flow: {flow_name}")
flow.register(project_name=PROD_PROJECT, labels=[SHARED_LABEL])
This works (in terms of registering) but the stdout is the following;
Result check: OK
Flow URL: <http://localhost:8080/default/flow/a6013647-a31e-4d3c-9e7e-82e69c02f71e>
└── ID: d78a0fc5-3990-461d-91d7-593a4278b6b2
└── Project: prod
└── Labels: ['prefect-agent']
I thought setting config.server.ui.endpoint
would lead to that endpoint being printed but it seems to not (localhost:8080
as opposed to UI_ENDPOINT
) - any ideas?Josh Greenhalgh
03/12/2021, 6:16 PMJosh Greenhalgh
03/15/2021, 3:28 PMResult
classes to my flows.
If I specify the result class in the same file that the flow is defined via flow.result = ...
everything works as expected.
However if I import the flow def and then attach the result class (same way just outside the flow def) I do not get any stored results.
I have put together a minimal repo here -> https://github.com/josh-gree2/minimal-repo/compare/result-in-cli...master (the two branches show what I mean)Gustavo Silva
03/15/2021, 3:52 PMGustavo Silva
03/15/2021, 3:53 PMProcess PID 266 returned non-zero exit code
Dana Merrick
03/15/2021, 5:22 PMprefect execute flow-run
but I want to add a command before thatChohang Ng
03/16/2021, 1:00 AMCharles Leung
03/16/2021, 3:02 PMValueError: Local Secret "VAULT_TOKEN" was not found.
Is this the expected behavior? should all secrets be registered wherever flows are created/registered?Josh Greenhalgh
03/16/2021, 7:01 PMINFO:root:Registering flow: weather_data_etl
Result check: OK
Flow URL: <http://localhost:8080/default/flow/b366cae3-8200-42d9-87b5-f621711e897a>
└── ID: 2570985c-efd0-4b29-88f1-96b86862d34a
└── Project: prod
└── Labels: ['prefect-agent']
INFO:root:Registering flow: test_flow
Result check: OK
Flow URL: <http://localhost:8080/default/flow/96b48708-c731-4dd7-9ba0-f354b8ef7028>
└── ID: 1307c478-cc0e-49ee-b4b8-b00a8834eed6
└── Project: prod
└── Labels: ['prefect-agent']
So all seems good...
However if I go to the UI there are no flows visible at all - that being said if I look into the postgres DB then the flows with the above IDs both exist...Josh Greenhalgh
03/16/2021, 11:10 PMAdam Lewis
03/17/2021, 1:52 PMCharles Liu
03/17/2021, 11:20 PMCharles Leung
03/18/2021, 5:00 PMCharles Liu
03/18/2021, 10:56 PMMichael Stolte
03/19/2021, 12:04 AMChristian
03/19/2021, 11:23 AMprefect server start --use-volume -d
is pulling the images, creating the network and creating the parts (tmp_postgres_1, tmp_hasura_1, …). However, it never finishes (no detach) and I do get errors in the logs of tmp_apollo_1:
Trying again in 3 seconds...
Building schema...
FetchError: request to <http://hasura:3001/v1alpha1/graphql> failed, reason: connect ECONNREFUSED 192.168.64.3:3001
at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1461:11)
at ClientRequest.emit (events.js:315:20)
at Socket.socketErrorListener (_http_client.js:469:9)
at Socket.emit (events.js:315:20)
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21) {
type: 'system',
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED'
}
Trying again in 3 seconds...
Not sure what’s going on?
My config is pretty simple, too:
[server]
[server.database]
host = "localhost"
port = "5432"
host_port = "5434"
[server.hasura]
host = "localhost"
port = "3000"
host_port = "3001"
The micro-services seem to be up and running:
(prefect) [root@myserver:~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
aec1a48767a3 prefecthq/ui:core-0.14.12 "/docker-entrypoint.…" 17 minutes ago Up 17 minutes (healthy) 80/tcp, 0.0.0.0:8080->8080/tcp tmp_ui_1
2996c1840fe1 prefecthq/apollo:core-0.14.12 "tini -g -- bash -c …" 17 minutes ago Up 17 minutes (healthy) 0.0.0.0:4200->4200/tcp tmp_apollo_1
2263ca7701e6 prefecthq/server:core-0.14.12 "tini -g -- python s…" 17 minutes ago Up 17 minutes tmp_towel_1
ed29fdb1b3b0 prefecthq/server:core-0.14.12 "tini -g -- bash -c …" 17 minutes ago Up 17 minutes (healthy) 0.0.0.0:4201->4201/tcp tmp_graphql_1
2a3dffb7d641 hasura/graphql-engine:v1.3.3 "graphql-engine serve" 17 minutes ago Up 17 minutes (healthy) 0.0.0.0:3001->3000/tcp tmp_hasura_1
cafbde90dc48 postgres:11 "docker-entrypoint.s…" 17 minutes ago Up 17 minutes (healthy) 0.0.0.0:5434->5432/tcp tmp_postgres_1
Christian
03/19/2021, 11:23 AMMassoud Mazar
03/19/2021, 3:08 PM0.14.6
to 0.14.12
and since then I see even when idle, CPU hovers between 10%-50%.
docker stats --all --format "table {{.ID}}\t{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"
shows the following:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT
c6eda45c6f59 tmp_ui_1 0.00% 4.953MiB / 7.432GiB
2466edd82357 tmp_towel_1 0.00% 52.57MiB / 7.432GiB
e9d0643d134e tmp_apollo_1 2.69% 62.68MiB / 7.432GiB
ad41dcf8646a tmp_graphql_1 3.24% 69.5MiB / 7.432GiB
a8f65bcb2efa tmp_hasura_1 4.62% 153.8MiB / 7.432GiB
bc9742482bd8 tmp_postgres_1 4.64% 27.86MiB / 7.432GiB
Dana Merrick
03/19/2021, 3:38 PMhttps://i.imgur.com/RhqSS8A.png▾
Dana Merrick
03/19/2021, 3:39 PMPaul Prescod
03/21/2021, 3:30 PMLukas N.
03/23/2021, 9:42 AMDana Merrick
03/23/2021, 4:15 PMRunNamespacedJob
? do I add it inside a task, so a task can create a Job during the course of a Flow?