Rajan Subramanian
03/16/2022, 6:36 PMKevin Kho
03/16/2022, 6:37 PMpip install --no-deps
to not install dependencies and then do them manuallyRajan Subramanian
03/16/2022, 6:38 PMKevin Kho
03/16/2022, 6:42 PMRajan Subramanian
03/16/2022, 6:44 PMZanie
03/16/2022, 6:44 PMRajan Subramanian
03/16/2022, 6:45 PMZanie
03/16/2022, 6:46 PMRajan Subramanian
03/16/2022, 8:08 PMCan't locate revision identified by '71a57ec351d1'
Zanie
03/16/2022, 8:10 PMprefect orion database stamp HEAD
then prefect orion database reset
should fix it; if not you can delete the database file manually from ~/.prefect/orion.db
Rajan Subramanian
03/16/2022, 8:10 PMZanie
03/16/2022, 9:32 PM--host 0.0.0.0
Rajan Subramanian
03/16/2022, 9:48 PMZanie
03/16/2022, 9:49 PMRajan Subramanian
03/16/2022, 9:50 PMZanie
03/16/2022, 9:50 PMRajan Subramanian
03/16/2022, 9:50 PMZanie
03/16/2022, 9:52 PMRajan Subramanian
03/16/2022, 9:52 PMZanie
03/16/2022, 9:54 PMRajan Subramanian
03/16/2022, 9:54 PMZanie
03/16/2022, 9:55 PMRajan Subramanian
03/16/2022, 9:56 PMKevin Kho
03/16/2022, 11:47 PMRajan Subramanian
03/17/2022, 1:30 PMexport PREFECT_ORION_DATABASE_CONNECTION_URL="<postgresql+asyncpg://ubuntu:mypassword@ec2-34>-????.<http://compute-1.amazonaws.com:5432/orion|compute-1.amazonaws.com:5432/orion>"
?Kevin Kho
03/17/2022, 2:29 PMRajan Subramanian
03/17/2022, 2:31 PMKevin Kho
03/17/2022, 3:33 PMRajan Subramanian
03/17/2022, 3:33 PMKevin Kho
03/17/2022, 3:34 PMRajan Subramanian
03/17/2022, 3:37 PMKevin Kho
03/17/2022, 3:41 PMRajan Subramanian
03/17/2022, 3:46 PMKevin Kho
03/17/2022, 4:09 PMRajan Subramanian
03/17/2022, 4:19 PMSuccessfully logged in and set workspace to my_company_workspace in profile: default
But then i see an error at the end:
Traceback (most recent call last):
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/cli/base.py", line 58, in wrapper
return fn(*args, **kwargs)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 120, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 67, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 56, in run
return asynclib.run(func, *args, **backend_options)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 233, in run
return native_run(wrapper(), debug=debug)
File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
return await func(*args)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/cli/cloud.py", line 216, in login
exit_with_success(
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/cli/base.py", line 193, in exit_with_success
raise typer.Exit(0)
click.exceptions.Exit: 0
An exception occurred.
Kevin Kho
03/17/2022, 4:20 PMZanie
03/17/2022, 4:22 PMRajan Subramanian
03/17/2022, 4:22 PMengine$ prefect storage ls
Traceback (most recent call last):
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/cli/base.py", line 58, in wrapper
return fn(*args, **kwargs)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 120, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 67, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 56, in run
return asynclib.run(func, *args, **backend_options)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 233, in run
return native_run(wrapper(), debug=debug)
File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
return await func(*args)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/cli/storage.py", line 178, in ls
json_blocks = await client.read_blocks(block_spec_type="STORAGE", as_json=True)
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/client.py", line 937, in read_blocks
response = await <http://self._client.post|self._client.post>(
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/utilities/httpx.py", line 137, in post
return await self.request(
File "/home/ubuntu/phobos/env/lib/python3.9/site-packages/prefect/utilities/httpx.py", line 47, in request
request = self.build_request(
TypeError: build_request() got an unexpected keyword argument 'extensions'
An exception occurred.
Zanie
03/17/2022, 4:24 PMpip uninstall httpx
and pip install httpx
pip show httpx
as well)Rajan Subramanian
03/17/2022, 4:29 PMKevin Kho
03/17/2022, 4:29 PMRajan Subramanian
03/17/2022, 4:30 PMprefect deployment create binance/orderbook_deployment.py
prefect work-queue create test_queue
prefect agent start 'uiud'
from previous step work_queue.
How do i get these to execute on the cloud? ```Kevin Kho
03/19/2022, 3:39 PMRajan Subramanian
03/19/2022, 3:39 PMKevin Kho
03/19/2022, 3:39 PMRajan Subramanian
03/19/2022, 3:39 PMKevin Kho
03/19/2022, 3:40 PMRajan Subramanian
03/19/2022, 3:40 PMKevin Kho
03/19/2022, 3:44 PMRajan Subramanian
03/19/2022, 3:44 PMKevin Kho
03/19/2022, 3:47 PMprefect work-queue preview --hours 12 'acffbcc8-ae65-4c83-a38a-96e2e5e5b441'
Rajan Subramanian
03/19/2022, 3:54 PMKevin Kho
03/19/2022, 5:07 PMRajan Subramanian
03/21/2022, 3:58 PMKevin Kho
03/21/2022, 4:00 PMRajan Subramanian
03/21/2022, 4:00 PMKevin Kho
03/21/2022, 4:02 PMRajan Subramanian
03/21/2022, 4:03 PMprefect agent start 'unique_uiud'
where unique_uiud is the unique uiud thats generated when i do
prefect work-queue create live_feeds_to_postgres_agent
Kevin Kho
03/21/2022, 4:04 PMRajan Subramanian
03/21/2022, 4:05 PMAnna Geller
03/21/2022, 4:06 PMRajan Subramanian
03/21/2022, 4:07 PMAnna Geller
03/21/2022, 4:12 PMRajan Subramanian
03/21/2022, 7:38 PMprefect deployment create ftx/order_book_deployments.py and i see 10 deployments created on the UI.
When i go to the UI and click run, suddenly the ec2 instance freezes up.
I then rebooted and repeated the steps above but instead of going to the UI to hit run, i did following:
prefect deployment execute live_feeds_to_redis_pipeline/ftx_L1_avaxusd and then i see the following message:
Its completely stuck, i cant hit cancel or anything, and it shows up late on the UI. Same sequence of steps works completely fine on my local machine.
The pipeline makes a shell call to a python file. When i go to the ssh terminal to the local of the python file and do a python3 file.py it streams perfectly on the shell,
But the deployment is creating a problem.Kevin Kho
03/21/2022, 7:42 PMRajan Subramanian
03/21/2022, 7:44 PMKevin Kho
03/21/2022, 7:45 PMRajan Subramanian
03/21/2022, 7:46 PMKevin Kho
03/21/2022, 8:54 PMRajan Subramanian
03/21/2022, 8:54 PMKevin Kho
03/21/2022, 10:36 PMRajan Subramanian
03/21/2022, 10:37 PMKevin Kho
03/21/2022, 10:37 PMRajan Subramanian
03/21/2022, 10:37 PMKevin Kho
03/21/2022, 10:39 PMRajan Subramanian
03/21/2022, 10:39 PMKevin Kho
03/21/2022, 11:14 PMRajan Subramanian
03/22/2022, 1:56 PMKevin Kho
03/22/2022, 2:25 PMRajan Subramanian
03/22/2022, 2:26 PMKevin Kho
03/22/2022, 2:30 PMRajan Subramanian
03/22/2022, 2:31 PMKevin Kho
03/22/2022, 2:33 PMRajan Subramanian
03/22/2022, 2:34 PMKevin Kho
03/22/2022, 2:41 PMRajan Subramanian
03/22/2022, 3:09 PMKevin Kho
03/22/2022, 3:10 PMRajan Subramanian
03/22/2022, 3:24 PMKevin Kho
03/22/2022, 3:28 PMasync
so it can change task in between while waiting for other stuff. But I wouldn’t know what the ratio of cpu and concurrent tasks needs to be for your use caseRajan Subramanian
03/22/2022, 3:29 PMKevin Kho
03/22/2022, 3:30 PMRajan Subramanian
03/22/2022, 3:33 PM