Surawut Jirasaktavee
07/30/2022, 7:21 PM(base) ubuntu@ip-172-31-26-134:~/course-mlops-zoomcamp/03-orchestration$ prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
warnings.warn(
Found flow 'xgboost_optimization'
Manifest created at '/home/ubuntu/course-mlops-zoomcamp/03-orchestration/main-manifest.json'.
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/cli/deployment.py", line 502, in build
file_count = await storage.put_directory()
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/filesystems.py", line 350, in put_directory
return await self.filesystem.put_directory(
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/filesystems.py", line 250, in put_directory
self.filesystem.put_file(f, fpath)
File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 111, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 96, in sync
raise return_result
File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
result[0] = await coro
File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 1015, in _put_file
await self._call_s3(
File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 332, in _call_s3
return await _error_wrapper(
File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 137, in _error_wrapper
raise err
File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 110, in _error_wrapper
return await func(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/client.py", line 235, in _make_api_call
request_dict = await self._convert_to_request_dict(
File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/client.py", line 283, in _convert_to_request_dict
api_params = await self._emit_api_params(
File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/client.py", line 312, in _emit_api_params
await self.meta.events.emit(
File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
response = await resolve_awaitable(handler(**kwargs))
File "/home/ubuntu/.local/lib/python3.10/site-packages/botocore/handlers.py", line 243, in validate_bucket_name
raise ParamValidationError(report=error_msg)
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "s3:": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
An exception occurred.
bucket_name = "mlflow-artifacts-prem/"
os.environ["AWS_PROFILE"] = "MLOps-dev"
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
def get_storage(bucket_path: str,
AWS_ACCESS_KEY_ID: str,
AWS_SECRET_ACCESS_KEY: str):
block = S3(bucket_path=bucket_name,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
PREFECT_URL_UI_API_URL
to set the config of the remote server and PREFECT_API_URL
from my local to set the IP address.
After updating this version I try to set the same but got some errors with the same variables.
And after reading the docs I found this.
I am not sure how to set the other server apart from the Prefect Cloud and the Local.
# use a local Orion API server
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
# use Prefect Cloud
prefect config set PREFECT_API_URL=<http://app.prefect.cloud/api>
Taylor Curran
07/30/2022, 7:33 PM--storage-block S3/mlflow-artifacts-prem
Surawut Jirasaktavee
07/30/2022, 7:50 PMraise ValueError(
ValueError: Unable to find block document named mlflow-artifacts-prem for block type S3
Taylor Curran
07/30/2022, 7:53 PMmlflow-artifacts-prem
?Surawut Jirasaktavee
07/30/2022, 7:53 PMTaylor Curran
07/30/2022, 7:57 PMprefect orion start
?Surawut Jirasaktavee
07/30/2022, 8:00 PMprefect orion start
but using S3 for storage.Taylor Curran
07/30/2022, 8:02 PMprefect cloud login
at all?Surawut Jirasaktavee
07/30/2022, 8:05 PMPREFECT_ORION_URL_UI_URL
(base) ubuntu@ip-172-31-26-134:~/course-mlops-zoomcamp/03-orchestration$ prefect config set PREFECT_ORION_URL_UI_URL="<http://54.193.87.108:4200/api>"
Unknown setting name 'PREFECT_ORION_URL_UI_URL'.
Taylor Curran
07/30/2022, 8:11 PMSurawut Jirasaktavee
07/30/2022, 8:14 PM# use a local Orion API server
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
# use Prefect Cloud
prefect config set PREFECT_API_URL=<http://app.prefect.cloud/api>
(base) ubuntu@ip-172-31-26-134:~/course-mlops-zoomcamp/03-orchestration$ prefect deployment build ./prefect_deploy.py:main --name xgboost_optimization --tag dev --infra process --storage-block s3/mlflow-artifacts-prem
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
warnings.warn(
Found flow 'xgboost_optimization'
Manifest created at '/home/ubuntu/course-mlops-zoomcamp/03-orchestration/main-manifest.json'.
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
await get_running_loop().create_connection(
File "/usr/lib/python3.10/asyncio/base_events.py", line 1064, in create_connection
raise exceptions[0]
File "/usr/lib/python3.10/asyncio/base_events.py", line 1049, in create_connection
sock = await self._connect_sock(
File "/usr/lib/python3.10/asyncio/base_events.py", line 960, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 499, in sock_connect
return await fut
File "/usr/lib/python3.10/asyncio/selector_events.py", line 534, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('54.193.87.108', 4200)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
yield
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
raise exc
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
stream = await self._connect(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 111, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
return await self._backend.connect_tcp(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 107, in connect_tcp
with map_exceptions(exc_map):
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/cli/deployment.py", line 496, in build
template = await Block.load(storage_block)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/blocks/core.py", line 507, in load
block_document = await client.read_block_document_by_name(
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client.py", line 1209, in read_block_document_by_name
response = await self._client.get(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
return await self.request(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client.py", line 256, in send
await super().send(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
response = await self._send_handling_auth(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
with map_httpcore_exceptions():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
Jenny
07/30/2022, 8:46 PMSurawut Jirasaktavee
07/31/2022, 4:21 AM(course-mlops-zoomcamp) 🚦 ~/Github/MLOps/course-mlops-zoomcamp/03-orchestration main prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
Found flow 'xgboost_optimization'
Manifest created at '/Users/premsurawut/Github/MLOps/course-mlops-zoomcamp/03-orchestration/main-manifest.json'.
Traceback (most recent call last):
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
await get_running_loop().create_connection(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/base_events.py", line 1065, in create_connection
raise exceptions[0]
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection
sock = await self._connect_sock(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock
await self.sock_connect(sock, address)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/selector_events.py", line 499, in sock_connect
return await fut
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/selector_events.py", line 534, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 61] Connect call failed ('3.101.74.249', 4200)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
yield
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
raise exc
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
stream = await self._connect(request)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection.py", line 111, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
return await self._backend.connect_tcp(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/cli/deployment.py", line 496, in build
template = await Block.load(storage_block)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/blocks/core.py", line 507, in load
block_document = await client.read_block_document_by_name(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/client.py", line 1209, in read_block_document_by_name
response = await self._client.get(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
return await self.request(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/client.py", line 256, in send
await super().send(*args, **kwargs)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
response = await self._send_handling_auth(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
S3
:
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "s3:": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
An exception occurred.
my command to run:
(course-mlops-zoomcamp) ✘ 🇹🇭 ~/Github/MLOps/course-mlops-zoomcamp/03-orchestration main ± prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
I used the above command to run and check the status of the Prefect server in the terminal as the image below shows the status seems to work well.
but when I go to Prefect UI. It shows nothing from today’s command that I have run. I have filtered only for today’s date.
I have tried to change a lot for the s3 but the result is still the same.
even for the path from the Prefect UI.Khuyen Tran
07/31/2022, 4:18 PMprefect deployment apply
after the build command?Surawut Jirasaktavee
08/01/2022, 2:00 AMdeployment.yaml
in order to run the command prefect deployment apply
right? but because of the issue above from the command:
prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
the command above only generates a JSON
file for me. I think it is because the command did not completed yet.
For now, I can prove that my flow can run and the Prefect & MLflow remote server works well. my s3 bucket store the artifacts as they should be.
but only the deployment build command that is not complete according to the issue from S3
name. So the command did not generate deployment.yaml
for me.
Any ideas about this issue? I took a few days to work with it but still got nothing.Khuyen Tran
08/01/2022, 2:20 AMSurawut Jirasaktavee
08/01/2022, 4:23 PMbucket_name = "mlflow-artifacts-prem"
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
def get_storage(s3_bucket: str,
AWS_ACCESS_KEY_ID: str,
AWS_SECRET_ACCESS_KEY: str):
block = S3(bucket_path=s3_bucket,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
return block
block = get_storage(bucket_name, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
Khuyen Tran
08/02/2022, 3:10 PMSurawut Jirasaktavee
08/02/2022, 3:29 PMKhuyen Tran
08/02/2022, 3:58 PMSurawut Jirasaktavee
08/02/2022, 4:00 PMKhuyen Tran
08/02/2022, 5:01 PMSurawut Jirasaktavee
08/03/2022, 1:52 AMKhuyen Tran
08/03/2022, 3:07 PMmlflow-artifacts-prem
, not <s3://mlflow-artifacts-prem>
Surawut Jirasaktavee
08/03/2022, 4:10 PM```bucket_name = "mlflow-artifacts-prem"
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
def get_storage(s3_bucket: str,
AWS_ACCESS_KEY_ID: str,
AWS_SECRET_ACCESS_KEY: str):
block = S3(bucket_path=s3_bucket,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
return block
block = get_storage(bucket_name, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)```Actually, my bucket path doesn’t include the
s3://
. like the first line from the code snippet above. I think the s3://
return from the Class S3 from Prefect docs. prefect.filesystems - Prefect 2.0Khuyen Tran
08/03/2022, 6:27 PMs3://
when creating an S3 block thoughSurawut Jirasaktavee
08/04/2022, 3:42 PMKhuyen Tran
08/04/2022, 5:02 PM