https://prefect.io logo
Join Slack
Powered by
Hi. Question about the S3 storage. I got this erro...
# prefect-community
s
Hi. Question about the S3 storage. I got this error in order to run the command to build the deployment. [Thread]
Copy code
(base) ubuntu@ip-172-31-26-134:~/course-mlops-zoomcamp/03-orchestration$ prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
  warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
Found flow 'xgboost_optimization'
Manifest created at '/home/ubuntu/course-mlops-zoomcamp/03-orchestration/main-manifest.json'.
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/cli/deployment.py", line 502, in build
    file_count = await storage.put_directory()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/filesystems.py", line 350, in put_directory
    return await self.filesystem.put_directory(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/filesystems.py", line 250, in put_directory
    self.filesystem.put_file(f, fpath)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 1015, in _put_file
    await self._call_s3(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 332, in _call_s3
    return await _error_wrapper(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 137, in _error_wrapper
    raise err
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 110, in _error_wrapper
    return await func(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/client.py", line 235, in _make_api_call
    request_dict = await self._convert_to_request_dict(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/client.py", line 283, in _convert_to_request_dict
    api_params = await self._emit_api_params(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/client.py", line 312, in _emit_api_params
    await self.meta.events.emit(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/home/ubuntu/.local/lib/python3.10/site-packages/botocore/handlers.py", line 243, in validate_bucket_name
    raise ParamValidationError(report=error_msg)
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "s3:": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
An exception occurred.
my code about s3
Copy code
bucket_name = "mlflow-artifacts-prem/"
os.environ["AWS_PROFILE"] = "MLOps-dev"
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY") 

def get_storage(bucket_path: str, 
                AWS_ACCESS_KEY_ID: str,
                AWS_SECRET_ACCESS_KEY: str):

    block = S3(bucket_path=bucket_name, 
            aws_access_key_id=AWS_ACCESS_KEY_ID, 
            aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
my prefect command:
Copy code
prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
Another question is in the previous version. I have used the variables 
PREFECT_URL_UI_API_URL
to set the config of the remote server and 
PREFECT_API_URL
from my local to set the IP address. After updating this version I try to set the same but got some errors with the same variables. And after reading the docs I found this. I am not sure how to set the other server apart from the Prefect Cloud and the Local.
Copy code
# use a local Orion API server
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>

# use Prefect Cloud
prefect config set PREFECT_API_URL=<http://app.prefect.cloud/api>
t
Hi Surawut, did you try specifying the block type with a capital S? Like S3: 
--storage-block S3/mlflow-artifacts-prem
s
Hi. I try it and got this new error:
Copy code
raise ValueError(
ValueError: Unable to find block document named mlflow-artifacts-prem for block type S3
t
In the UI is it showing an S3 block with the exact name 
mlflow-artifacts-prem
?
s
let me check the UI a bit.
Yeah. The UI already showing the S3 block.
t
Are you using cloud? Or did you run 
prefect orion start
?
s
just 
prefect orion start
but using S3 for storage.
t
hmm, I wonder if you may have logged into cloud in the terminal and now can not access this local block?
I would also try creating a new block in the UI and trying again.
I am not sure for now, but let me know if you find anything out!
Have you run 
prefect cloud login
at all?
s
No. I haven’t used Prefect cloud.
I want to try with my remote server first. but I don’t know how to set the config variable for 
PREFECT_ORION_URL_UI_URL
that one is the variable that I used before in Prefect 2.0b5 or 6.
the error I got after assigning to the old variable.
Copy code
(base) ubuntu@ip-172-31-26-134:~/course-mlops-zoomcamp/03-orchestration$ prefect config set PREFECT_ORION_URL_UI_URL="<http://54.193.87.108:4200/api>"
Unknown setting name 'PREFECT_ORION_URL_UI_URL'.
Trying to find the new one to set it.
t
Here is some documentation that you may find helpful. Regarding the S3 block as I said, it wouldn’t hurt to recreate the S3 block in the UI and try again with that. Overall just verify that your cli commands (flows you run, workqueues listed) match what you see in the UI. Regarding the config variable, My other colleagues may be around later who know more about this, but I would ask it as a separate question in the channel.
s
Yeah, I read from this one. found some commands that I think should be the ones that I looking for.
Copy code
# use a local Orion API server
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>

# use Prefect Cloud
prefect config set PREFECT_API_URL=<http://app.prefect.cloud/api>
Yeah, It works well. I can use Prefect with my remote server now. but it seems like the new one.
No block and no run.
I got connect error for now. after using my remote server and run the command to build deployment
Copy code
(base) ubuntu@ip-172-31-26-134:~/course-mlops-zoomcamp/03-orchestration$ prefect deployment build ./prefect_deploy.py:main --name xgboost_optimization --tag dev --infra process --storage-block s3/mlflow-artifacts-prem
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
  warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
Found flow 'xgboost_optimization'
Manifest created at '/home/ubuntu/course-mlops-zoomcamp/03-orchestration/main-manifest.json'.
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1064, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1049, in create_connection
    sock = await self._connect_sock(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 960, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 499, in sock_connect
    return await fut
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 534, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('54.193.87.108', 4200)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 107, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/cli/deployment.py", line 496, in build
    template = await Block.load(storage_block)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/blocks/core.py", line 507, in load
    block_document = await client.read_block_document_by_name(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client.py", line 1209, in read_block_document_by_name
    response = await self._client.get(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client.py", line 256, in send
    await super().send(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
j
Hi @Surawut Jirasaktavee - you should be able to see what settings options there are here: https://github.com/PrefectHQ/prefect/blob/ea6f062d5005096ff890aab9595cecc1e4bde9ed/src/prefect/settings.py
gratitude thank you 1
s
Hi, Jenny. Thank you. I will check this.
Now I can use my remote server to open the Prefect UI. but when I run the command to build the deployment. I get the connection error.
Copy code
(course-mlops-zoomcamp)  🚦   ~/Github/MLOps/course-mlops-zoomcamp/03-orchestration   main  prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
Found flow 'xgboost_optimization'
Manifest created at '/Users/premsurawut/Github/MLOps/course-mlops-zoomcamp/03-orchestration/main-manifest.json'.
Traceback (most recent call last):
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/base_events.py", line 1065, in create_connection
    raise exceptions[0]
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection
    sock = await self._connect_sock(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock
    await self.sock_connect(sock, address)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/selector_events.py", line 499, in sock_connect
    return await fut
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/selector_events.py", line 534, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 61] Connect call failed ('3.101.74.249', 4200)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/cli/deployment.py", line 496, in build
    template = await Block.load(storage_block)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/blocks/core.py", line 507, in load
    block_document = await client.read_block_document_by_name(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/client.py", line 1209, in read_block_document_by_name
    response = await self._client.get(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/prefect/client.py", line 256, in send
    await super().send(*args, **kwargs)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/premsurawut/anaconda3/envs/course-mlops-zoomcamp/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
For now, I think I can use my local connect with Prefect remote server perfectly but I am stuck with the error about 
S3
:
Copy code
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "s3:": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
An exception occurred.
my command to run:
Copy code
(course-mlops-zoomcamp)  ✘ 🇹🇭   ~/Github/MLOps/course-mlops-zoomcamp/03-orchestration   main ±  prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
I used the above command to run and check the status of the Prefect server in the terminal as the image below shows the status seems to work well. but when I go to Prefect UI. It shows nothing from today’s command that I have run. I have filtered only for today’s date. I have tried to change a lot for the s3 but the result is still the same. even for the path from the Prefect UI.
k
Did you run 
prefect deployment apply
after the build command?
s
If I don’t misunderstand. I should have the 
deployment.yaml
in order to run the command 
prefect deployment apply
right? but because of the issue above from the command:
Copy code
prefect deployment build ./prefect_deploy.py:main \
--name xgboost_optimization \
--tag dev \
--infra process \
--storage-block s3/mlflow-artifacts-prem
the command above only generates a 
JSON
file for me. I think it is because the command did not completed yet. For now, I can prove that my flow can run and the Prefect & MLflow remote server works well. my s3 bucket store the artifacts as they should be. but only the deployment build command that is not complete according to the issue from 
S3
name. So the command did not generate 
deployment.yaml
for me. Any ideas about this issue? I took a few days to work with it but still got nothing.
k
Can you show the configuration of your S3 block? It could be that the path you set in the S3 block is the problem
s
This is my block configuration function.
Copy code
bucket_name = "mlflow-artifacts-prem"
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")

def get_storage(s3_bucket: str, 
                AWS_ACCESS_KEY_ID: str,
                AWS_SECRET_ACCESS_KEY: str):

    block = S3(bucket_path=s3_bucket, 
            aws_access_key_id=AWS_ACCESS_KEY_ID, 
            aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
    return block

block = get_storage(bucket_name, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
Anyway, can I solve this issue? or Am I missing something in my code block?
k
So you bucket path is mlflow-artifacts-prem? Do you have that bucket in your S3
s
yeah. I have already created the S3 before referring to it.
k
Have you tried to add block configuration on the UI?
s
It is already shown in the Prefect UI as well.
but it will show fail of invalid name s3 then the command cannot complete and not generate the deployment.yaml to me.
k
could you take a screenshot of your configuration in the S3 block? You can blur out your configurations if you want to. I just want to double check
s
Hi, Here is my block page in the Prefect UI.
k
ah I see the problem. Your bucket path should be 
mlflow-artifacts-prem
, not 
<s3://mlflow-artifacts-prem>
s
Hmm. I think I have tried this way before. but I will try again.
I’ll update you.
wait. I think my bucket path is as below:
```bucket_name = "mlflow-artifacts-prem"
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
def get_storage(s3_bucket: str,
AWS_ACCESS_KEY_ID: str,
AWS_SECRET_ACCESS_KEY: str):
block = S3(bucket_path=s3_bucket,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
return block
block = get_storage(bucket_name, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)```
Actually, my bucket path doesn’t include the 
s3://
. like the first line from the code snippet above. I think the 
s3://
return from the Class S3 from Prefect docs. prefect.filesystems - Prefect 2.0
k
Hmm. That’s weird. I don’t have 
s3://
when creating an S3 block though
Can you create one directly from the UI?
s
Hi, @Khuyen Tran It’s work now. I think this is my misunderstanding from the first time I created the bucket. Thank you for supporting me. really appreciated. after applying the deployment I should create the work queue and agent to run the pipeline, right?
k
I’m glad that you figured it out. I’m always happy to help. Yes, that’s would be the next step
🙌 1
11 Views
Open in Slack
PreviousNext
Powered by