Tom Shaffner
11/17/2021, 4:53 PMprefect server start --expose --ui-port=80 --server-port=1720
), I can at least get to the UI then, but the UI gives me a failure message about not connecting to graphql and a link to this slack (second picture below). I've tried setting the graphql port flag 5060 or 443 or even also to 1720 or using the no port mapping flag (https://docs.prefect.io/api/latest/cli/server.html), all to no avail. I'm new to this so I'm probably doing something stupid but I can't figure out what; any thoughts?
In case it's related, I'm also unable to start a user agent. When I try prefect agent local start
I get requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=1720): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d60a34e20>: Failed to establish a new connection: [Errno 111] Connection refused'))
. I'm unclear on whether the server is automatically starting agents of its own or if I'd have to do this myself in addition to the server.
I could potentially have the option to open some additional ports, but it takes several days, a long approval process that might get rejected, and I'm not even clear on which I'd need to open or why. After all, these are mostly internal ports to the same machine, right?
I know that's a lot but I'm a bit at a loss here; any help?Kevin Kho
11/17/2021, 5:00 PMconfig.toml
that points them where to connect to. It looks like you were on the right track there.
In general, I think the approach here would be to make sure the UI is working first and correcting to graphQL. I am not sure if setting it to port 1720 will work because that might be a reserved port. I think in general you want to set your ports to ports not in use.
It says it’s trying to connect to 4200 in your second image. Could you show me what your config.toml
looks like?Tom Shaffner
11/17/2021, 5:08 PMbackend="server"
in it. And agreed on getting the UI working first; that was my plan too. Once it's up and working I'll focus on the agents.
Setting it to ports not in use would mean I should actually stop using these flags, right? If I try only prefect server start --expose --ui-port=80
I get, among many other things, these results:
Pulling postgres ... done
Pulling hasura ... done
Pulling graphql ... done
Pulling apollo ... done
Pulling towel ... done
Pulling ui ... done
Creating network "prefect-server" with the default driver
Creating tmp_postgres_1 ... done
Creating tmp_hasura_1 ... done
Creating tmp_graphql_1 ... done
Creating tmp_towel_1 ... done
Creating tmp_apollo_1 ... done
Creating tmp_ui_1 ... done
Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_apollo_1, tmp_towel_1, tmp_ui_1
apollo_1 | Checking GraphQL service at <http://graphql:4201/health> ...
apollo_1 | Checking GraphQL service at <http://graphql:4201/health> ...
apollo_1 | Checking GraphQL service at <http://graphql:4201/health> ...
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
towel_1 | {"severity": "INFO", "name": "prefect-server.Scheduler", "message": "Scheduled 0 flow runs."}
graphql_1 |
graphql_1 | Running Alembic migrations...
graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
graphql_1 | INFO [alembic.runtime.migration] Running upgrade -> 27811b58307b, Create extensions and initial settings
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 27811b58307b -> 72e2cd3e0469, Initial database tables migration
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 72e2cd3e0469 -> c4d792bdd05e, Add flow run idempotency key
graphql_1 | INFO [alembic.runtime.migration] Running upgrade c4d792bdd05e -> 3398e4807bfb, Add traversal functions
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 3398e4807bfb -> b9086bd4b962, Create message table
graphql_1 | INFO [alembic.runtime.migration] Running upgrade b9086bd4b962 -> c1f317aa658c, Remove state_id foreign keys
graphql_1 | INFO [alembic.runtime.migration] Running upgrade c1f317aa658c -> 6611fd0ccc73, Simplify run state update triggers
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 6611fd0ccc73 -> 70528cee0d2b, Add agent persistence
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 70528cee0d2b -> 9cb7539b7363, Add index on agent_id
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 9cb7539b7363 -> e148cf9f1e5b, Add task run name
graphql_1 | INFO [alembic.runtime.migration] Running upgrade e148cf9f1e5b -> 850b76d44332, Add flow run config
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 850b76d44332 -> 24f10aeee83e, Add label column to flow runs
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 24f10aeee83e -> 3c87ad7e0b71, Add artifact api
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 3c87ad7e0b71 -> 57ac2cb01ac1, Add index for task run names
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 57ac2cb01ac1 -> 7ca57ea2fdff, Add run_config to flow runs and flow groups
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 7ca57ea2fdff -> 9116e81c6dc2, Add description to flow group table
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 9116e81c6dc2 -> 459a61bedc9e, Improve run triggers to handle same-version states
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 459a61bedc9e -> a666a3f4e422, Add unique index for idempotency key
graphql_1 | INFO [alembic.runtime.migration] Running upgrade a666a3f4e422 -> ac5747fb571c, Add unique constraint on edge table for task IDs
graphql_1 | Applied Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
graphql_1 |
graphql_1 | Database upgraded!
hasura_1 | {"type":"pg-client","timestamp":"2021-11-17T17:05:00.057+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
hasura_1 | {"type":"pg-client","timestamp":"2021-11-17T17:05:00.057+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1 | {"type":"startup","timestamp":"2021-11-17T17:05:00.057+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.25.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.25.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default timezone ... Etc/UTC
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | creating configuration files ... ok
postgres_1 | running bootstrap script ... ok
postgres_1 | performing post-bootstrap initialization ... ok
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....2021-11-17 17:04:58.984 UTC [49] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2021-11-17 17:04:59.089 UTC [50] LOG: database system was shut down at 2021-11-17 17:04:58 UTC
postgres_1 | 2021-11-17 17:04:59.107 UTC [49] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 | CREATE DATABASE
postgres_1 |
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | 2021-11-17 17:05:00.291 UTC [49] LOG: received fast shutdown request
postgres_1 | waiting for server to shut down....2021-11-17 17:05:00.303 UTC [49] LOG: aborting any active transactions
postgres_1 | 2021-11-17 17:05:00.305 UTC [49] LOG: background worker "logical replication launcher" (PID 56) exited with exit code 1
postgres_1 | 2021-11-17 17:05:00.307 UTC [51] LOG: shutting down
postgres_1 | 2021-11-17 17:05:00.609 UTC [49] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2021-11-17 17:05:00.638 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2021-11-17 17:05:00.638 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2021-11-17 17:05:00.659 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2021-11-17 17:05:00.696 UTC [77] LOG: database system was shut down at 2021-11-17 17:05:00 UTC
postgres_1 | 2021-11-17 17:05:00.709 UTC [1] LOG: database system is ready to accept connections
ui_1 | Missing the PREFECT_SERVER__BASE_URL environment variable. Using default
ui_1 | :space_invader::space_invader::space_invader: UI running at localhost:8080 :space_invader::space_invader::space_invader:
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: using the "epoll" event method
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: nginx/1.20.1
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: built by gcc 8.3.0 (Debian 8.3.0-6)
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: OS: Linux 5.11.0-1021-azure
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: getrlimit(RLIMIT_NOFILE): 1048576:1048576
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: start worker processes
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: start worker process 12
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: start worker process 13
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: start worker process 14
ui_1 | 2021/11/17 17:05:09 [notice] 11#11: start worker process 15
graphql_1 | {"severity": "INFO", "name": "prefect-server.GraphQL Server", "message": "Using uvicorn log level = 'debug'"}
graphql_1 | INFO: Started server process [8]
graphql_1 | INFO: Waiting for application startup.
graphql_1 | INFO: Application startup complete.
graphql_1 | INFO: Uvicorn running on <http://0.0.0.0:4201> (Press CTRL+C to quit)
apollo_1 | Checking GraphQL service at <http://graphql:4201/health> ...
graphql_1 | INFO: 172.25.0.5:52090 - "GET /health HTTP/1.1" 200 OK
apollo_1 | {"status":"ok","version":"2021.11.09"}
apollo_1 | GraphQL service healthy!
apollo_1 |
apollo_1 | > @ serve /apollo
apollo_1 | > node dist/index.js
apollo_1 |
apollo_1 | Building schema...
graphql_1 | INFO: 172.25.0.5:52118 - "POST /graphql/ HTTP/1.1" 200 OK
apollo_1 | Building schema complete!
apollo_1 | Server ready at <http://0.0.0.0:4200> :rocket: (version: 2021.11.09)
apollo_1 | Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"startup","payload":{"id":"d0f3dca5-469c-4e76-9ad7-ba50f304d71c","prefect_server_version":"2021.11.09","api_version":"0.2.0"}}
graphql_1 | INFO: 172.25.0.5:52136 - "POST /graphql/ HTTP/1.1" 200 OK
graphql_1 | INFO: 172.25.0.5:52148 - "POST /graphql/ HTTP/1.1" 200 OK
WELCOME TO
_____ _____ ______ ______ ______ _____ _______ _____ ______ _______ ________ _____
| __ \| __ \| ____| ____| ____/ ____|__ __| / ____| ____| __ \ \ / / ____| __ \
| |__) | |__) | |__ | |__ | |__ | | | | | (___ | |__ | |__) \ \ / /| |__ | |__) |
| ___/| _ /| __| | __| | __|| | | | \___ \| __| | _ / \ \/ / | __| | _ /
| | | | \ \| |____| | | |___| |____ | | ____) | |____| | \ \ \ / | |____| | \ \
|_| |_| \_\______|_| |______\_____| |_| |_____/|______|_| \_\ \/ |______|_| \_\
Visit <http://localhost:80> to get started, or check out the docs at <https://docs.prefect.io>
Kevin Kho
11/17/2021, 5:13 PMTom Shaffner
11/17/2021, 5:15 PMKevin Kho
11/17/2021, 5:20 PMTom Shaffner
11/17/2021, 5:29 PMsudo ufw allow 4200
), but if I start the server after I still get the same error. If this is internal, and this is a standard ubuntu VM, what might cause the port to be blocked like this?Kevin Kho
11/17/2021, 5:49 PM--expose
flag. Sorry I think I was wrong the first time around. I re-read the logs and it looks like it was able to start. Just to make sure, the error you see if that the UI can’t connect to the API?Anna Geller
11/17/2021, 5:58 PMTom Shaffner
11/17/2021, 6:59 PMpostgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2021-11-17 19:53:52.121 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2021-11-17 19:53:52.121 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2021-11-17 19:53:52.132 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2021-11-17 19:53:52.172 UTC [76] LOG: database system was shut down at 2021-11-17 19:53:52 UTC
postgres_1 | 2021-11-17 19:53:52.187 UTC [1] LOG: database system is ready to accept connections
postgres_1 | 2021-11-17 19:53:56.594 UTC [84] ERROR: duplicate key value violates unique constraint "pg_extension_name_index"
postgres_1 | 2021-11-17 19:53:56.594 UTC [84] DETAIL: Key (extname)=(pgcrypto) already exists.
postgres_1 | 2021-11-17 19:53:56.594 UTC [84] STATEMENT: CREATE EXTENSION IF NOT EXISTS pgcrypto SCHEMA public
ui_1 | Missing the PREFECT_SERVER__BASE_URL environment variable. Using default
ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: using the "epoll" event method
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: nginx/1.20.1
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: built by gcc 8.3.0 (Debian 8.3.0-6)
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: OS: Linux 5.11.0-1021-azure
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: getrlimit(RLIMIT_NOFILE): 1048576:1048576
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: start worker processes
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: start worker process 19
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: start worker process 20
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: start worker process 21
ui_1 | 2021/11/17 19:54:01 [notice] 18#18: start worker process 22
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: [Errno 113] Connect call failed ('192.168.16.3', 3000)", "exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n request, auth=auth, timeout=timeout, history=history\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n response = await self.send_single_request(request, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n timeout=timeout.as_dict(),\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n method, url, headers=headers, stream=stream, timeout=timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n self.socket = await self._open_socket(timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n hostname, port, ssl_context, timeout\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n stream_reader=stream_reader, stream_writer=stream_writer\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 113] Connect call failed ('192.168.16.3', 3000)"}
To ensure there weren't any simple corruption issues, I deleted and recreated the virtual environment and tried to initialize the server again. The error messages I'm getting changed as a result. The full list was too long to paste here, but I pasted the above log which shows the last bit that was working and then the first error message (it's long). That same error then appears to repeat, with minor variations, five more times.Kevin Kho
11/17/2021, 8:02 PMTom Shaffner
11/17/2021, 8:21 PMKevin Kho
11/17/2021, 8:23 PMprefect server start --expose
is going to the GraphQL Playground. The default is localhost:4200
.Tom Shaffner
11/17/2021, 8:26 PMKevin Kho
11/17/2021, 8:27 PMTom Shaffner
11/17/2021, 8:27 PMKevin Kho
11/17/2021, 8:28 PMconfig.toml
because it’s getting the default localhost. One sec will get thatTom Shaffner
11/17/2021, 8:28 PMKevin Kho
11/17/2021, 8:29 PMTom Shaffner
11/17/2021, 8:36 PMKevin Kho
11/17/2021, 8:41 PM[server]
endpoint = "YOUR_MACHINES_PUBLIC_IP:4200/graphql"
in the config.toml of the machine you are connecting from. This is to register flows and communicate with the server API. You can test if this connection is good with prefect agent local start
and it should be able to connect.
The second is the UI. The UI needs to be configured to point the API of your Server (port 4200 by default). The default is the localhost:4200/graphql
. If this is not changed, your local machine goes to the Prefect UI and then gets directed to localhost:4200/graphql
, which is wrong. You want to direct it to the server API. To change this, you can do:
[server]
[server.ui]
apollo_url = "<http://YOUR_MACHINES_PUBLIC_IP:4200/graphql>"
in the config.toml
file that lives on the server. Do this before you start the server so that it points to the API. When you connect to the UI from your local machine, it will talk to the right API endpoint also.Tom Shaffner
11/17/2021, 8:43 PMKevin Kho
11/17/2021, 8:44 PMTom Shaffner
11/17/2021, 8:45 PMKevin Kho
11/17/2021, 8:46 PMTom Shaffner
11/17/2021, 9:32 PMKevin Kho
11/29/2021, 3:57 PMconfig.toml
on the VM when you started Prefect Server? Let me re-read some stuffhttp://*YOUR_MACHINES_PUBLIC_IP*:8080
works right?Tom Shaffner
11/29/2021, 4:03 PMKevin Kho
11/29/2021, 4:06 PMlocalhost
instead of your IP. Just clarifying, do you have two separate config.toml
files? One on the VM and one on local?Tom Shaffner
11/29/2021, 4:09 PMls
of the folder and contents attached, in order.
When you say local, what do you mean? I'm trying to run this all from the VM, and the only other connection to it is via the web browser; I wouldn't need a config.toml on the machine with the web browser, would I?Kevin Kho
11/29/2021, 4:11 PMlocalhost:4200/graphql
.Tom Shaffner
11/29/2021, 4:11 PMKevin Kho
11/29/2021, 4:14 PM~/.prefect
folder of the machine you are connecting from. I would say yes in general you need the config.toml
file for every computer that will connect to server because that the very least, you’ll see to point it to the server API to register flows.Tom Shaffner
11/29/2021, 4:20 PM~
folder in Windows is the user account folder in C:\Users
, correct? So C:\Users\USERNAME
? I just tried it, restarted the server and reloaded the UI in a fresh browser window with the .prefect
folder added to my local user account and a config.toml
in it; same error.Kevin Kho
11/29/2021, 4:24 PMprefect backend server
, this should create the home directory I think. Also if you do this in Python,:
from os.path import expanduser
expanduser("~")
the output should be the home directory where your .prefect
folder should live,Anna Geller
11/29/2021, 4:26 PMTom Shaffner
11/29/2021, 4:27 PMAnna Geller
11/29/2021, 4:31 PMTom Shaffner
11/29/2021, 4:32 PMKevin Kho
11/29/2021, 4:32 PMTom Shaffner
11/29/2021, 4:35 PMAnna Geller
11/29/2021, 4:38 PMprefect auth login --key XXX
Kevin Kho
11/29/2021, 4:38 PMTom Shaffner
11/29/2021, 4:43 PMif
command can break the flow structure itself too (separate thread); that seems a far cry from the advertised simplicity of being able to add some decorators to my existing python code. The purpose here is a test of this system to see if it works for us, and if the setup is so complex that we'd have to buy a solution to get it working...well, that's not a dynamic I'd feel comfortable recommending.
I still like the prospect here, I'm still hopeful I can get set up and start transitioning some of our current ETL processes over, but unless/until I can do that and verify they work reliably for some time I'm not going to push the company to go through the time and effort to review this from a security and data privacy perspective.Anna Geller
11/29/2021, 4:55 PMpip install prefect
and start building and executing flows locally, without the backend API configured. You can then run: flow.run()
or with the CLI “prefect run -p flow.py” to run the flow locally without having to talk to the backend API.Tom Shaffner
11/29/2021, 4:56 PMAnna Geller
11/29/2021, 4:57 PMTom Shaffner
11/29/2021, 4:59 PMAnna Geller
11/29/2021, 5:00 PMTom Shaffner
11/29/2021, 5:12 PMKevin Kho
11/29/2021, 5:13 PMTom Shaffner
11/29/2021, 5:18 PMKevin Kho
11/29/2021, 5:20 PM