Luuk
02/25/2022, 12:22 PMFailed to load and execute flow run: Exception('Azure connection string not provided. Set `AZURE_STORAGE_CONNECTION_STRING` environment variable or save connection string as Prefect secret.')
First I was working on the Local storage, but that gave me ModuleErrors and an old prefect thread explained that it should be solved through the use of Azure/Github/etc storages.
with Flow(
"extract",
storage=Azure(container="prefect-flows"),
run_config=DockerRun(image="<http://xxx.azurecr.io/extract:latest|xxx.azurecr.io/extract:latest>", labels=["extract"], env=env_config.dict()),
) as flow:
print(env_config.dict(), file=stderr)
# Prints: {'AZURE_STORAGE_CONNECTION_STRING': 'DefaultEndpointsProtocol=https;AccountName=xxxxxx==;EndpointSuffix=<http://core.windows.net|core.windows.net>'}
Anna Geller
02/25/2022, 12:48 PMexport PREFECT__CLOUD__USE_LOCAL_SECRETS=false
The reason for it is: perhaps your flow run tries to grab the Secrets from Prefect cloud and when you're using Server you need to use local Secrets.
You can also check out this similar thread that explains it a bit moreLuuk
02/25/2022, 12:57 PM/app/extract_flows.py
/app/classes:
envs.py
metadata.py
But extract_flows.py
can't import the classes by using: from classes.envs import Class
as you normally would do, this gives me the following error:
Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'classes\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')How should I import other files within my docker image? Does the pickle influence this?
Anna Geller
02/25/2022, 1:09 PM/opt/prefect
. Check this sample Dockerfile that you can use as a templateLuuk
02/25/2022, 1:13 PMflow_utils.db
if __name__ == "__main__":
flow.register(project_name="default")
extract_1 | Traceback (most recent call last):
extract_1 | File "/opt/prefect/flows/extract_flows.py", line 6, in <module>
extract_1 | from classes.envs import EnvConfig
extract_1 | ModuleNotFoundError: No module named 'classes
Folder structure:
/opt/prefect/classes:
envs.py
metadata.py
/opt/prefect/config:
example.yml
/opt/prefect/flows:
extract_flows.py
FROM prefecthq/prefect:latest
RUN /usr/local/bin/python -m pip install --upgrade pip
WORKDIR /opt/prefect
COPY ./requirements.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt
COPY ./src/classes/ /opt/prefect/classes/
COPY ./src/config/ /opt/prefect/config/
COPY ./src/utils/ /opt/prefect/utils/
COPY ./src/extract_flows.py /opt/prefect/flows/
Kevin Kho
02/25/2022, 2:33 PMLuuk
02/25/2022, 2:36 PMKevin Kho
02/25/2022, 2:40 PMAnna Geller
02/25/2022, 2:56 PMfrom src.classes import XYZ
you would need to have:
from flow_utils.classes import XYZ
or whatever you named the package in setup.py and the folder name (doesn't have to be named flow_utils
) - hope this is clearer now but if I confused you even further LMK 🙂Luuk
02/25/2022, 3:07 PMException: Azure connection string not provided. SetEven though I do give both options to the flow.environment variable or save connection string as Prefect secret.AZURE_STORAGE_CONNECTION_STRING
run_config=DockerRun(image="<http://xxxx.azurecr.io/extract:latest|xxxx.azurecr.io/extract:latest>", labels=["extract"], env=env_config.dict()),
env_config.dict()
{'AZURE_STORAGE_CONNECTION_STRING': 'DefaultEndpointsProtocol=https;AccountName=xxxx==;EndpointSuffix=<http://core.windows.net|core.windows.net>', 'PREFECT__CLOUD__USE_LOCAL_SECRETS': False}
Sometimes code has their own life.Kevin Kho
02/25/2022, 3:09 PMLuuk
02/25/2022, 3:13 PMrun_config=DockerRun(image="<http://xxx.azurecr.io/extract:latest|xxx.azurecr.io/extract:latest>", labels=["extract"], env={"PREFECT__CLOUD__USE_LOCAL_SECRETS": False}),
The secret is inside my cloud as key.Kevin Kho
02/25/2022, 3:15 PMstorage=Azure(container="prefect-flows", connection_string_secret="AZURE_STORAGE_CONNECTION_STRING")
?Secret("AZURE….").get()
Luuk
02/25/2022, 3:20 PMAnna Geller
02/25/2022, 3:20 PMprefect agent local start --env PREFECT__CONTEXT__SECRETS__AZURE_STORAGE_CONNECTION_STRING=xxx
Luuk
02/25/2022, 3:23 PMValueError: Local Secret "AZURE_STORAGE_CONNECTION_STRING" was not found.
Kevin Kho
02/25/2022, 3:25 PMAZURE_STORAGE_CONNECTION_STRING
secret stored. The only thing you need it to point it to Cloud so it doesn’t pull a local secret so the Agent needs an env var PREFECT___CLOUD___USE_LOCAL_SECRETS=false_
Luuk
02/25/2022, 3:33 PMflow.register()
should try to find for AZURE_STORAGE_CONNECTION_STRING
no? Instead of the value DefaultEndpointsProtocol=https;AccountName=gensonadlsgen2;AccountKey=xxx==;EndpointSuffix=<http://core.windows.net|core.windows.net>
Traceback (most recent call last):
File "/opt/prefect/flows/extract_flows.py", line 45, in <module>
flow.register(project_name="default")
File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1715, in register
idempotency_key=idempotency_key,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 848, in register
serialized_flow = flow.serialize(build=build) # type: Any
File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1497, in serialize
storage = self.storage.build() # type: Optional[Storage]
File "/usr/local/lib/python3.7/site-packages/prefect/storage/azure.py", line 147, in build
client = self._azure_block_blob_service.get_blob_client(
File "/usr/local/lib/python3.7/site-packages/prefect/storage/azure.py", line 178, in _azure_block_blob_service
conn_str=self.connection_string
File "/usr/local/lib/python3.7/site-packages/prefect/storage/azure.py", line 162, in connection_string
return Secret(self.connection_string_secret).get()
File "/usr/local/lib/python3.7/site-packages/prefect/client/secrets.py", line 159, in get
) from exc
KeyError: 'The secret DefaultEndpointsProtocol=https;AccountName=xxxx==;EndpointSuffix=<http://core.windows.net|core.windows.net> was not found. Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>'
Unsure where it gets that value or key from...Kevin Kho
02/25/2022, 3:40 PMLuuk
02/25/2022, 7:14 PMwith Flow(
"extract",
storage=Azure(container="prefect-flows", connection_string_secret="xxxxx=="),
run_config=DockerRun(image="<http://xxx.azurecr.io/extract:latest|xxx.azurecr.io/extract:latest>", labels=["extract"]),
) as flow: