https://prefect.io logo
Channels
Powered by
m

Madhup Sukoon

02/01/2022, 6:16 PM
I am using Prefect Cloud. My task Goes to 
Scheduled
> 
Submitted
and then nothing happens. The only entry in logs is:
Copy code
Submitted for execution: Container ID: 143cabe06c721.....
My flow uses GitHub Storage and DockerRun, and uses some secrets. Any way I could get more verbose logs on what might be happening? This is what my 
flow.py
looks like:
Copy code
from main import planner, creator, publisher

planner = task(planner)
creator = task(creator)
publisher = task(publisher)

foo = Secret('foo')
bar = Secret('bar')

with Flow(
    "myflow",
    run_config = DockerRun(
        env={"EXTRA_PIP_PACKAGES": "requests pyyaml"}
    ),
    storage = GitHub(
        repo="me/myrepo",
        path="flow.py",
        access_token_secret="GITHUB_ACCESS_TOKEN"
    )
    
) as flow:
    plan = planner()
    media = creator(foo, bar, plan)
    post = publisher(foo, bar, media)
main
is an adjacent python file (
main.py
)
k

Kevin Kho

02/01/2022, 6:21 PM
A couple of things to try: 1. Use debug level logs by adding 
PREFECT___LOGGING___LEVEL
to 
DEBUG
in the env of DockerRun and then re-register 2. If it is submitted, you can check the container logs on the machine with the agent Can I see the docs you used to define your Secrets like that?
a

Anna Geller

02/01/2022, 6:30 PM
Kevin is right that Secrets is likely the issue here. I saw a very similar issue yesterday with Gitlab. This submitted for execution log with container ID means that your flow run was successfully started but it probably doesn’t move forward because the container crashes since it cannot pull the flow from your Github Storage. Check if your Github access token is working correctly and whether it has proper permissions. You can do a curl command to Github to check if the token is working. Also you can use this syntax to ensure you use Cloud Secrets.
m

Madhup Sukoon

02/01/2022, 6:50 PM
I don't have access to the machine logs. I modified the code (added 
.get()
to secrets), here it is now:
Copy code
from prefect import Flow, task
from prefect.client import Secret
from prefect.storage import Docker, GitHub
from prefect.run_configs import DockerRun

from main import planner, creator, publisher

planner = task(planner)
creator = task(creator)
publisher = task(publisher)

foo = Secret('foo').get()
bar = Secret('bar').get()

with Flow(
    "myflow",
    run_config = DockerRun(
        env={
            "EXTRA_PIP_PACKAGES": "requests pyyaml",
            "PREFECT__LOGGING__LEVEL": "DEBUG"
        }
    ),
    storage = GitHub(
        repo="me/myrepo",
        path="flow.py",
        access_token_secret="GITHUB_ACCESS_TOKEN"
    )
    
) as flow:
    plan = planner()
    media = creator(foo, bar, plan)
    post = publisher(foo, bar, media)
k

Kevin Kho

02/01/2022, 6:52 PM
That worked for you? I’m not sure that was the issue
m

Madhup Sukoon

02/01/2022, 6:54 PM
Nope, still not working
k

Kevin Kho

02/01/2022, 6:55 PM
Do you have access to the agent? You can try debug level logs on the agent 
prefect agent docker start --log-level=DEBUG
m

Madhup Sukoon

02/01/2022, 6:56 PM
Do these logs get sent to the Prefect UI?
k

Kevin Kho

02/01/2022, 6:56 PM
Agent do not. Just Flow logs
Did you check Anna’s post?
m

Madhup Sukoon

02/01/2022, 6:56 PM
ah, but I do not have access to the Agent logs - it's running in a superuser process.
k

Kevin Kho

02/01/2022, 6:57 PM
Anna’s post may solve your issue
m

Madhup Sukoon

02/01/2022, 7:02 PM
I checked Anna's post. 1. The GitHub token is working properly
2. not sure if 
export PREFECT__CLOUD__USE_LOCAL_SECRETS=false
setting has to be done on the host running the Agent - from my (very limited) understanding, this should not be required, right?
k

Kevin Kho

02/01/2022, 7:07 PM
You shouldn’t have to. How do you know the token is working properly? A local run succeeded or you used it for another Flow?
a

Anna Geller

02/01/2022, 7:09 PM
correct, this would need to be set on the agent to prevent that agent tries to use local secrets. Since the container successfully starts, it would be great if you could share the container logs for this flow run to see what is happening within this flow run container. You can run:
Copy code
docker logs 143cabe06c721
m

Madhup Sukoon

02/01/2022, 7:09 PM
I did a local git clone using the token. Works as usual
a

Anna Geller

02/01/2022, 7:10 PM
locally you are authenticated with git. To check if the token is working, you would need to make an API request and set the token in the request header - you can do that e.g. with curl or using postman
m

Madhup Sukoon

02/01/2022, 7:11 PM
I tried the Github token on an ephemeral docker container I just spun up. The token seems fine.
Wait a minute - I am using
Copy code
DockerRun(
        env={"EXTRA_PIP_PACKAGES": "requests pyyaml"}
    )
This would be using the default prefect docker image. Does it run on ARM?
The agent is running on ARM hardware
k

Kevin Kho

02/01/2022, 7:14 PM
this PR is adding support for ARM
a

Anna Geller

02/01/2022, 7:15 PM
Can you try this? replace by your token
Copy code
curl -i -H "Authorization: token ghp_16C7e42F292c6912E7710c838347Ae178B4a" \
    <https://api.github.com/user/repos>
m

Madhup Sukoon

02/01/2022, 7:17 PM
Just did, works perfectly.
a

Anna Geller

02/01/2022, 7:18 PM
awesome, great to confirm that. In that case, your token seems correct and seems to be well scoped. Did you set it in the UI as a Secret? If so, then we can exclude the Secret from being the issue here
m

Madhup Sukoon

02/01/2022, 7:20 PM
Yes, it's set in the UI. Earlier it was not, so the 
register
step was failing. It is now set and the 
register
step works.
a

Anna Geller

02/01/2022, 7:22 PM
nice, can you now share the output of the flow run logs?
Copy code
docker logs 143cabe06c721
m

Madhup Sukoon

02/01/2022, 7:25 PM
Error: No such container
docker ps -a shows no entries. I'm wondering if it's because of the ARM issue - I don't see any arm images here https://hub.docker.com/r/prefecthq/prefect/tags
a

Anna Geller

02/01/2022, 7:25 PM
the way I would test it: I would try first a simpler flow with no extra dependencies and plain hello world, but also using the same Github repo and the same Docker agent:
Copy code
from prefect import task, Flow


@task(log_stdout=True)
def hello_world():
    print("hello world")


with Flow("hello", run_config = DockerRun(
        env={
           
            "PREFECT__LOGGING__LEVEL": "DEBUG"
        }
    ),
    storage = GitHub(
        repo="me/myrepo",
        path="flow.py",
        access_token_secret="GITHUB_ACCESS_TOKEN"
    )) as flow:
    hw = hello_world()
m

Madhup Sukoon

02/01/2022, 7:27 PM
Okay, I'll try that. Also, would you happen to know if there are any arm based prefect docker images?
a

Anna Geller

02/01/2022, 7:27 PM
yeah, I think for ARM you can’t use Prefect’s default image because it’s amd-based. You’re right that that’s likely the issue now. You would need to build a custom image or change to use Docker storage instead with:
Copy code
flow.storage = Docker(..., build_kwargs={"platform": "linux/arm64/v8"})  # or: "linux/amd64"})
nope, there are not atm. We don’t have sepaate images for different architectures atm. You can check that on Dockerhub https://hub.docker.com/r/prefecthq/prefect
but even if you build your own image or change to Docker Storage, I would recommend still starting this a simple hello world flow before you use it with custom dependencies - easier to do it step by step
they are all amd64
m

Madhup Sukoon

02/01/2022, 7:33 PM
Yup, I'll try starting from a simple flow and checking if things work. Can you tell me if there's a way for me to build the Prefect image from scratch? (I can build it for arm and test)
Or if it would be possible for you guys to build an arm version?
a

Anna Geller

02/01/2022, 7:41 PM
let me help you. It’s quite simple. You need a Dockerfile. Given your packages you may use this:
Copy code
FROM prefecthq/prefect:0.15.13-python3.8 # choose version
RUN pip install requests pyyaml
Then to build this image, run this command from the same directory where your dockerfile is:
Copy code
docker build --platform linux/amd64 -t your_image .
Then, once your image is built, in your DockerRun pass the image name:
Copy code
flow.run_config = DockerRun(image="your_image:latest")
This flag does all you need to build arm image: --platform linux/amd64
m

Madhup Sukoon

02/01/2022, 7:46 PM
Thanks, I'll try this.
Also many thanks for your patience!
👍 1
Changed to a very simple Flow (the hello world example by @Anna Geller) Still no luck, here are the logs:
Copy code
[2022-02-01 20:24:41,939] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:24:41,939] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:24:51,940] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:24:52,389] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:24:52,391] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:25:02,391] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:02,792] DEBUG - agent | Found 1 ready flow run(s): {'11fe4354-db57-4058-ba51-1e3225f9da2b'}
[2022-02-01 20:25:02,800] DEBUG - agent | Retrieving metadata for 1 flow run(s)...
[2022-02-01 20:25:03,177] DEBUG - agent | Submitting flow run 11fe4354-db57-4058-ba51-1e3225f9da2b for deployment...
[2022-02-01 20:25:03,186] INFO - agent | Deploying flow run 11fe4354-db57-4058-ba51-1e3225f9da2b to execution environment...
[2022-02-01 20:25:03,187] DEBUG - agent | Sleeping flow run poller for 0.25 seconds...
[2022-02-01 20:25:03,189] DEBUG - agent | Updating flow run 11fe4354-db57-4058-ba51-1e3225f9da2b state from Scheduled -> Submitted...
[2022-02-01 20:25:03,439] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:03,638] INFO - agent | Pulling image me/prefect:latest-python3.9-arm...
[2022-02-01 20:25:03,646] DEBUG - agent | {'status': 'Trying to pull repository <http://docker.io/me/prefect|docker.io/me/prefect> ... '}
[2022-02-01 20:25:03,867] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:03,867] DEBUG - agent | Sleeping flow run poller for 0.5 seconds...
[2022-02-01 20:25:04,368] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:04,758] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:04,761] DEBUG - agent | Sleeping flow run poller for 1.0 seconds...
[2022-02-01 20:25:05,761] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:06,162] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:06,165] DEBUG - agent | Sleeping flow run poller for 2.0 seconds...
[2022-02-01 20:25:06,519] DEBUG - agent | Sending agent heartbeat...
[2022-02-01 20:25:06,521] DEBUG - agent | Heartbeat succesful! Sleeping for 60.0 seconds...
[2022-02-01 20:25:07,342] DEBUG - agent | {'status': 'Pulling from <http://docker.io/me/prefect|docker.io/me/prefect>', 'id': 'latest-python3.9-arm'}
[2022-02-01 20:25:07,343] DEBUG - agent | {'status': 'Digest: sha256:aa9d2263f25beafa320633949fcba81b5d51e4cb2b0a0b78e67d5d1af2a272e0'}
[2022-02-01 20:25:07,343] DEBUG - agent | {'status': 'Status: Image is up to date for me/prefect:latest-python3.9-arm'}
[2022-02-01 20:25:07,343] INFO - agent | Successfully pulled image me/prefect:latest-python3.9-arm
[2022-02-01 20:25:07,343] DEBUG - agent | Creating Docker container me/prefect:latest-python3.9-arm
[2022-02-01 20:25:07,377] DEBUG - agent | Starting Docker container with ID e795629d74dc5cdda5b697789183e151188c000056732eedacc329cc87fd2629 and name 'olive-koel'
[2022-02-01 20:25:07,641] DEBUG - agent | Docker container e795629d74dc5cdda5b697789183e151188c000056732eedacc329cc87fd2629 started
[2022-02-01 20:25:07,642] INFO - agent | Completed deployment of flow run 11fe4354-db57-4058-ba51-1e3225f9da2b
[2022-02-01 20:25:08,165] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:08,525] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:08,525] DEBUG - agent | Sleeping flow run poller for 4.0 seconds...
[2022-02-01 20:25:12,526] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:12,945] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:12,946] DEBUG - agent | Sleeping flow run poller for 8.0 seconds...
[2022-02-01 20:25:20,947] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:21,293] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:21,296] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:25:31,297] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:31,656] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:31,656] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:25:41,656] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:42,002] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:42,002] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:25:52,002] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:52,438] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:52,441] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:26:02,441] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:26:02,818] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:26:02,822] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:26:06,544] DEBUG - agent | Sending agent heartbeat...
[2022-02-01 20:26:06,547] DEBUG - agent | Heartbeat succesful! Sleeping for 60.0 seconds...
[2022-02-01 20:26:12,822] DEBUG - agent | Querying for ready flow runs...
a

Anna Geller

02/01/2022, 8:32 PM
this looks good to me though! your flow run was successfully deployed. Can you check the flow run container?
m

Madhup Sukoon

02/01/2022, 8:33 PM
docker ps -a still shows no containers
Nothing here either:
This is what my flow looks like:
Copy code
@task(log_stdout=True)
def hello_world():
    print("hello world")

with Flow(
    "nomodosc",
    run_config = DockerRun(
        image="me/prefect:latest-python3.9-arm",
        labels=["personal-worker-arm-docker-test"],
        env={
            "EXTRA_PIP_PACKAGES": "requests pyyaml",
            "PREFECT__LOGGING__LEVEL": "DEBUG"
        }
    ),
    storage = GitHub(
        repo="me/myrepo",
        path="flow.py",
        access_token_secret="GITHUB_ACCESS_TOKEN"
    )
    
) as flow:
    hw = hello_world()
running 
flow.run()
locally works well.
k

Kevin Kho

02/01/2022, 8:48 PM
Catching up here. Flow.run will be fine because it doesn’t use the RunConfiguration. Where did you push your image to? Is it just a local image or you used some repository?
m

Madhup Sukoon

02/01/2022, 9:05 PM
Pushed it to a Private Github Repo
k

Kevin Kho

02/01/2022, 9:05 PM
Wait, not the code, the Docker image is on Github?
m

Madhup Sukoon

02/01/2022, 9:09 PM
Sorry, the image is on dockerhub
Just added 
--show-flow-logs
and noticed this 
standard_init_linux.go:228: exec user process caused: exec format error
k

Kevin Kho

02/01/2022, 9:11 PM
That is most likely related to the image incompatibility I think
m

Madhup Sukoon

02/01/2022, 9:11 PM
Copy code
[2022-02-01 21:06:19,944] DEBUG - agent | Sleeping flow run poller for 1.0 seconds...
[2022-02-01 21:06:20,944] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 21:06:21,290] DEBUG - agent | No ready flow runs found.
[2022-02-01 21:06:21,290] DEBUG - agent | Sleeping flow run poller for 2.0 seconds...
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Pulling from <http://docker.io/me/prefect|docker.io/me/prefect>', 'id': 'latest-python3.9-arm'}
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Digest: sha256:aa9d2263f25beafa320633949fcba81b5d51e4cb2b0a0b78e67d5d1af2a272e0'}
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Status: Image is up to date for me/prefect:latest-python3.9-arm'}
[2022-02-01 21:06:22,535] INFO - agent | Successfully pulled image me/prefect:latest-python3.9-arm
[2022-02-01 21:06:22,535] DEBUG - agent | Creating Docker container me/prefect:latest-python3.9-arm
/usr/local/lib/python3.6/site-packages/prefect/agent/docker/agent.py:423: UserWarning: `host.docker.internal` could not be automatically resolved to your local host. This feature is not supported on Docker Engine v19.03.11-ol, upgrade to v20.10.0+ if you encounter issues.
  "`host.docker.internal` could not be automatically resolved to your "
[2022-02-01 21:06:22,565] DEBUG - agent | Starting Docker container with ID f193912f48643eed4c3437f6cf2f0ca29a8cf50ad4600741878edb869c0d0754 and name 'cuddly-waxbill'
[2022-02-01 21:06:22,853] DEBUG - agent | Docker container f193912f48643eed4c3437f6cf2f0ca29a8cf50ad4600741878edb869c0d0754 started
[2022-02-01 21:06:22,854] INFO - agent | Completed deployment of flow run 9badb74e-4c9f-4f20-902c-ea32969a57de
standard_init_linux.go:228: exec user process caused: exec format error
[2022-02-01 21:06:23,290] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 21:06:23,644] DEBUG - agent | No ready flow runs found.
[2022-02-01 21:06:23,645] DEBUG - agent | Sleeping flow run poller for 4.0 seconds...
k

Kevin Kho

02/01/2022, 9:13 PM
Did you build with the 
--platform
like Anna suggested?
m

Madhup Sukoon

02/01/2022, 9:16 PM
yes
k

Kevin Kho

02/01/2022, 9:20 PM
Let me look at some stuff
Ok so it looks like even if you build it on the ARM platform, you need to specify it on the container creation side (when the Flow is spun up). The problem is that the 
docker
BuildKit CLI lets you do this, but 
docker-py
, which Prefect uses under the hood does not. You can read the details here This means you need some kind of hack like installing buildkit on the agent and loading the image already so that Prefect doesn’t try to download it. Ultimately, I don’t the the pieces from the underlying tools are there to support this yet (specifically docker-py)
🙂 1
🥲 1
5 Views
Powered by