Alexey Gorchakov
09/22/2021, 9:44 AMKevin Kho
09/22/2021, 2:46 PMAlexey Gorchakov
09/22/2021, 2:57 PMfrom prefect.agent.local import LocalAgent
if __name__ == "__main__":
agent = LocalAgent(name="the_agent", labels=[...])
agent.start()
At the file system of the container our code is placed.
The way we are using to register our flows at the server is following:
we run the python script at the agent container, where the following env variables are set:
- PREFECT__BACKEND=server
- PREFECT__SERVER__HOST=${PREFECT__SERVER__HOST}
- PREFECT__SERVER__PORT=${PREFECT__SERVER__PORT}
/<home dir of a container>/reg/register.py
import contextlib
from prefect import Client
from prefect.utilities.exceptions import ClientError
from . common
def main():
project = "dixy_reg"
client = Client()
with contextlib.suppress(ClientError):
client.create_tenant(name="default", slug="default")
client.create_project(project_name=project)
client.register(common.init_flow(), project_name=project)
if __name__ == "__main__":
main()
We just activate the conda env and run python -m reg.register.
And it works.
Now we want to create a Gitlab CI stage that will test our new code.
We share the upcommig code from Gitlab with an Agent container, and we want to re-register flows the way that we used before, but at this time we are at a different container.
We use almost the same code, except that we explicitly specify the api_server param of prefect.Client.
After the registration at the server we have got a new version of the flow.
But when we are trying to run it,
we have faced with the following error
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/pipeline/'")We can meet this prefix at the flow serialize method result:
('storage',
{'path': None,
'secrets': [],
'directory': '/home/pipeline/.prefect/flows',
'stored_as_script': False,
'flows': {'check_version': '/home/pipeline/.prefect/flows/check-version/2021-09-21t17-43-15-413079-00-00'},
'__version__': '0.15.2',
'type': 'Local'})
But when I print
print(common.init_flow().serialize(build=True))
from /reg/register.py that listed above, I see the same thing:
('storage', {'stored_as_script': False, 'directory': '/home/pipeline/.prefect/flows', 'path': None, 'flows': {'check_version': '/home/pipeline/.prefect/flows/check-version/2021-09-22t09-37-37-140541-00-00'}, 'secrets': [], '__version__': '0.15.2', 'type': 'Local'})
So it seems that the issue should be at the different place.
Does anybody have an idea what is going wrong at our scheme of remote flow registration?Kevin Kho
09/22/2021, 3:00 PM.prefect
folder under your home directory.
So if you register a Flow from Computer A, it lives on the harddrive of Computer A, only an agent on Computer A will run it. You will see the error you get if an agent on Computer B tried to run the flow.
Going to your setup, I think the container is registering it and saving it locally inside the container but you might be trying to run the Flow from a machine or container that does not hold the Flow locallyAlexey Gorchakov
09/22/2021, 3:33 PMKevin Kho
09/22/2021, 5:18 PM