https://prefect.io logo
Title
v

Valantis Hatzimagkas

03/03/2022, 10:08 AM
Hello! I have noticed the following behavior and I cannot figure out what is going on. (I am performing the following action on my local machine) I have done he following: 1. I have created a flow which has a task that gets some data 2. I am using Docker storage 3. I have done the required steps(building base image, passing dependencies, running server with --expose, adding a docker agent that communicates in the same network with my server) My Issues: I have added a for loop in my task, this is causing the task to be stuck in the running state I have attempted a dictionary comprehension inside my task, it caused a ι SystemError('Objects/dictobject.c:1555: bad argument to internal function') I tried the same piece of code in local storage and it works fine, so I think there is something with Docker storage that I am missing.
a

Abhishek

03/03/2022, 10:11 AM
@Valantis Hatzimagkas can you share the code or a gist?
:upvote: 1
v

Valantis Hatzimagkas

03/03/2022, 10:19 AM
Yes of course!
from prefect import task, Flow, prefect
from prefect.storage import Docker
import requests

lat = 40
lon = 45
API_key = 'my_key'
url = f'<https://api.openweathermap.org/data/2.5/weather?lat={lat}&lon={lon}&appid={API_key}>'

@task
def get_weather():
    return requests.get(url).json()


@task
def logger(res):
    for k, v in res['main'].items():
        print(f'{k}: {v}')


with Flow('spray_service') as flow:
    data = get_weather()
    logger(data)

if __name__ == "__main__":
    flow.storage = Docker(image_name="my_image", image_tag="latest", local_image=True,
                          dockerfile='Dockerfile')
    #flow.register('test')
    flow.run()
a

Anna Geller

03/03/2022, 10:57 AM
Running docker agent with Server deployed in docker compose is a bit hard because you may end up running docker in docker. I think you would need to modify your docker compose setup so that the docker agent and your Server components are in the same Docker network. This thread seems similar and I added a couple of comments there and some links you may try, LMK if this is not helpful and you need more info https://prefect-community.slack.com/archives/C014Z8DPDSR/p1646249456379019?thread_ts=1646239884.507719&amp;cid=C014Z8DPDSR
Your flow is perfectly fine and I'm pretty sure if you switch to local storage and local agent this should work, but docker agent is a bit more involved to set up with Server
v

Valantis Hatzimagkas

03/03/2022, 11:31 AM
Thanks Anna, yes I tried it also on local storage and it worked fine, both in the ui and executing it via flow.run() in my code. I will give it a shot with the thread you provided me. Thanks a lot!
👍 1
Thanks again Anna, I found my actual problem. I am going to leave this here just in case: My Docker file had:
FROM prefecthq/prefect:0.15.13-python3.9
But! my flow was serialized in *python 3.10(During registration prefect complained about this,* but silly me I did not notice it*)* This caused this strange issue in my case
a

Anna Geller

03/03/2022, 3:50 PM
Thanks for sharing and nice work figuring this out! 🙌
🙌 1