Madhup Sukoon
02/01/2022, 6:16 PMScheduled
> Submitted
and then nothing happens. The only entry in logs is:
Submitted for execution: Container ID: 143cabe06c721.....
My flow uses GitHub Storage and DockerRun, and uses some secrets. Any way I could get more verbose logs on what might be happening?
This is what my flow.py
looks like:
from main import planner, creator, publisher
planner = task(planner)
creator = task(creator)
publisher = task(publisher)
foo = Secret('foo')
bar = Secret('bar')
with Flow(
"myflow",
run_config = DockerRun(
env={"EXTRA_PIP_PACKAGES": "requests pyyaml"}
),
storage = GitHub(
repo="me/myrepo",
path="flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN"
)
) as flow:
plan = planner()
media = creator(foo, bar, plan)
post = publisher(foo, bar, media)
main
is an adjacent python file (main.py
)Kevin Kho
PREFECT___LOGGING___LEVEL
to DEBUG
in the env of DockerRun and then re-register
2. If it is submitted, you can check the container logs on the machine with the agent
Can I see the docs you used to define your Secrets like that?Anna Geller
Madhup Sukoon
02/01/2022, 6:50 PM.get()
to secrets), here it is now:
from prefect import Flow, task
from prefect.client import Secret
from prefect.storage import Docker, GitHub
from prefect.run_configs import DockerRun
from main import planner, creator, publisher
planner = task(planner)
creator = task(creator)
publisher = task(publisher)
foo = Secret('foo').get()
bar = Secret('bar').get()
with Flow(
"myflow",
run_config = DockerRun(
env={
"EXTRA_PIP_PACKAGES": "requests pyyaml",
"PREFECT__LOGGING__LEVEL": "DEBUG"
}
),
storage = GitHub(
repo="me/myrepo",
path="flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN"
)
) as flow:
plan = planner()
media = creator(foo, bar, plan)
post = publisher(foo, bar, media)
Kevin Kho
Madhup Sukoon
02/01/2022, 6:54 PMKevin Kho
prefect agent docker start --log-level=DEBUG
Madhup Sukoon
02/01/2022, 6:56 PMKevin Kho
Kevin Kho
Madhup Sukoon
02/01/2022, 6:56 PMKevin Kho
Madhup Sukoon
02/01/2022, 7:02 PMMadhup Sukoon
02/01/2022, 7:03 PMexport PREFECT__CLOUD__USE_LOCAL_SECRETS=false
setting has to be done on the host running the Agent - from my (very limited) understanding, this should not be required, right?Kevin Kho
Anna Geller
docker logs 143cabe06c721
Madhup Sukoon
02/01/2022, 7:09 PMAnna Geller
Madhup Sukoon
02/01/2022, 7:11 PMMadhup Sukoon
02/01/2022, 7:13 PMDockerRun(
env={"EXTRA_PIP_PACKAGES": "requests pyyaml"}
)
This would be using the default prefect docker image. Does it run on ARM?Madhup Sukoon
02/01/2022, 7:13 PMKevin Kho
Anna Geller
curl -i -H "Authorization: token ghp_16C7e42F292c6912E7710c838347Ae178B4a" \
<https://api.github.com/user/repos>
Madhup Sukoon
02/01/2022, 7:17 PMAnna Geller
Madhup Sukoon
02/01/2022, 7:20 PMregister
step was failing. It is now set and the register
step works.Anna Geller
docker logs 143cabe06c721
Madhup Sukoon
02/01/2022, 7:25 PMError: No such container
docker ps -a shows no entries.
I'm wondering if it's because of the ARM issue - I don't see any arm images here https://hub.docker.com/r/prefecthq/prefect/tagsAnna Geller
from prefect import task, Flow
@task(log_stdout=True)
def hello_world():
print("hello world")
with Flow("hello", run_config = DockerRun(
env={
"PREFECT__LOGGING__LEVEL": "DEBUG"
}
),
storage = GitHub(
repo="me/myrepo",
path="flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN"
)) as flow:
hw = hello_world()
Madhup Sukoon
02/01/2022, 7:27 PMAnna Geller
flow.storage = Docker(..., build_kwargs={"platform": "linux/arm64/v8"}) # or: "linux/amd64"})
Anna Geller
Anna Geller
Anna Geller
Madhup Sukoon
02/01/2022, 7:33 PMMadhup Sukoon
02/01/2022, 7:33 PMAnna Geller
FROM prefecthq/prefect:0.15.13-python3.8 # choose version
RUN pip install requests pyyaml
Then to build this image, run this command from the same directory where your dockerfile is:
docker build --platform linux/amd64 -t your_image .
Then, once your image is built, in your DockerRun pass the image name:
flow.run_config = DockerRun(image="your_image:latest")
Anna Geller
Madhup Sukoon
02/01/2022, 7:46 PMMadhup Sukoon
02/01/2022, 7:46 PMMadhup Sukoon
02/01/2022, 8:29 PM[2022-02-01 20:24:41,939] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:24:41,939] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:24:51,940] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:24:52,389] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:24:52,391] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:25:02,391] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:02,792] DEBUG - agent | Found 1 ready flow run(s): {'11fe4354-db57-4058-ba51-1e3225f9da2b'}
[2022-02-01 20:25:02,800] DEBUG - agent | Retrieving metadata for 1 flow run(s)...
[2022-02-01 20:25:03,177] DEBUG - agent | Submitting flow run 11fe4354-db57-4058-ba51-1e3225f9da2b for deployment...
[2022-02-01 20:25:03,186] INFO - agent | Deploying flow run 11fe4354-db57-4058-ba51-1e3225f9da2b to execution environment...
[2022-02-01 20:25:03,187] DEBUG - agent | Sleeping flow run poller for 0.25 seconds...
[2022-02-01 20:25:03,189] DEBUG - agent | Updating flow run 11fe4354-db57-4058-ba51-1e3225f9da2b state from Scheduled -> Submitted...
[2022-02-01 20:25:03,439] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:03,638] INFO - agent | Pulling image me/prefect:latest-python3.9-arm...
[2022-02-01 20:25:03,646] DEBUG - agent | {'status': 'Trying to pull repository <http://docker.io/me/prefect|docker.io/me/prefect> ... '}
[2022-02-01 20:25:03,867] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:03,867] DEBUG - agent | Sleeping flow run poller for 0.5 seconds...
[2022-02-01 20:25:04,368] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:04,758] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:04,761] DEBUG - agent | Sleeping flow run poller for 1.0 seconds...
[2022-02-01 20:25:05,761] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:06,162] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:06,165] DEBUG - agent | Sleeping flow run poller for 2.0 seconds...
[2022-02-01 20:25:06,519] DEBUG - agent | Sending agent heartbeat...
[2022-02-01 20:25:06,521] DEBUG - agent | Heartbeat succesful! Sleeping for 60.0 seconds...
[2022-02-01 20:25:07,342] DEBUG - agent | {'status': 'Pulling from <http://docker.io/me/prefect|docker.io/me/prefect>', 'id': 'latest-python3.9-arm'}
[2022-02-01 20:25:07,343] DEBUG - agent | {'status': 'Digest: sha256:aa9d2263f25beafa320633949fcba81b5d51e4cb2b0a0b78e67d5d1af2a272e0'}
[2022-02-01 20:25:07,343] DEBUG - agent | {'status': 'Status: Image is up to date for me/prefect:latest-python3.9-arm'}
[2022-02-01 20:25:07,343] INFO - agent | Successfully pulled image me/prefect:latest-python3.9-arm
[2022-02-01 20:25:07,343] DEBUG - agent | Creating Docker container me/prefect:latest-python3.9-arm
[2022-02-01 20:25:07,377] DEBUG - agent | Starting Docker container with ID e795629d74dc5cdda5b697789183e151188c000056732eedacc329cc87fd2629 and name 'olive-koel'
[2022-02-01 20:25:07,641] DEBUG - agent | Docker container e795629d74dc5cdda5b697789183e151188c000056732eedacc329cc87fd2629 started
[2022-02-01 20:25:07,642] INFO - agent | Completed deployment of flow run 11fe4354-db57-4058-ba51-1e3225f9da2b
[2022-02-01 20:25:08,165] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:08,525] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:08,525] DEBUG - agent | Sleeping flow run poller for 4.0 seconds...
[2022-02-01 20:25:12,526] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:12,945] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:12,946] DEBUG - agent | Sleeping flow run poller for 8.0 seconds...
[2022-02-01 20:25:20,947] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:21,293] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:21,296] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:25:31,297] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:31,656] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:31,656] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:25:41,656] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:42,002] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:42,002] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:25:52,002] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:25:52,438] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:25:52,441] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:26:02,441] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 20:26:02,818] DEBUG - agent | No ready flow runs found.
[2022-02-01 20:26:02,822] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-02-01 20:26:06,544] DEBUG - agent | Sending agent heartbeat...
[2022-02-01 20:26:06,547] DEBUG - agent | Heartbeat succesful! Sleeping for 60.0 seconds...
[2022-02-01 20:26:12,822] DEBUG - agent | Querying for ready flow runs...
Anna Geller
Madhup Sukoon
02/01/2022, 8:33 PMMadhup Sukoon
02/01/2022, 8:34 PMMadhup Sukoon
02/01/2022, 8:39 PM@task(log_stdout=True)
def hello_world():
print("hello world")
with Flow(
"nomodosc",
run_config = DockerRun(
image="me/prefect:latest-python3.9-arm",
labels=["personal-worker-arm-docker-test"],
env={
"EXTRA_PIP_PACKAGES": "requests pyyaml",
"PREFECT__LOGGING__LEVEL": "DEBUG"
}
),
storage = GitHub(
repo="me/myrepo",
path="flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN"
)
) as flow:
hw = hello_world()
Madhup Sukoon
02/01/2022, 8:40 PMflow.run()
locally works well.Kevin Kho
Madhup Sukoon
02/01/2022, 9:05 PMKevin Kho
Madhup Sukoon
02/01/2022, 9:09 PMMadhup Sukoon
02/01/2022, 9:10 PM--show-flow-logs
and noticed this
standard_init_linux.go:228: exec user process caused: exec format error
Kevin Kho
Madhup Sukoon
02/01/2022, 9:11 PM[2022-02-01 21:06:19,944] DEBUG - agent | Sleeping flow run poller for 1.0 seconds...
[2022-02-01 21:06:20,944] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 21:06:21,290] DEBUG - agent | No ready flow runs found.
[2022-02-01 21:06:21,290] DEBUG - agent | Sleeping flow run poller for 2.0 seconds...
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Pulling from <http://docker.io/me/prefect|docker.io/me/prefect>', 'id': 'latest-python3.9-arm'}
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Digest: sha256:aa9d2263f25beafa320633949fcba81b5d51e4cb2b0a0b78e67d5d1af2a272e0'}
[2022-02-01 21:06:22,534] DEBUG - agent | {'status': 'Status: Image is up to date for me/prefect:latest-python3.9-arm'}
[2022-02-01 21:06:22,535] INFO - agent | Successfully pulled image me/prefect:latest-python3.9-arm
[2022-02-01 21:06:22,535] DEBUG - agent | Creating Docker container me/prefect:latest-python3.9-arm
/usr/local/lib/python3.6/site-packages/prefect/agent/docker/agent.py:423: UserWarning: `host.docker.internal` could not be automatically resolved to your local host. This feature is not supported on Docker Engine v19.03.11-ol, upgrade to v20.10.0+ if you encounter issues.
"`host.docker.internal` could not be automatically resolved to your "
[2022-02-01 21:06:22,565] DEBUG - agent | Starting Docker container with ID f193912f48643eed4c3437f6cf2f0ca29a8cf50ad4600741878edb869c0d0754 and name 'cuddly-waxbill'
[2022-02-01 21:06:22,853] DEBUG - agent | Docker container f193912f48643eed4c3437f6cf2f0ca29a8cf50ad4600741878edb869c0d0754 started
[2022-02-01 21:06:22,854] INFO - agent | Completed deployment of flow run 9badb74e-4c9f-4f20-902c-ea32969a57de
standard_init_linux.go:228: exec user process caused: exec format error
[2022-02-01 21:06:23,290] DEBUG - agent | Querying for ready flow runs...
[2022-02-01 21:06:23,644] DEBUG - agent | No ready flow runs found.
[2022-02-01 21:06:23,645] DEBUG - agent | Sleeping flow run poller for 4.0 seconds...
Kevin Kho
--platform
like Anna suggested?Madhup Sukoon
02/01/2022, 9:16 PMKevin Kho
Kevin Kho
docker
BuildKit CLI lets you do this, but docker-py
, which Prefect uses under the hood does not. You can read the details here
This means you need some kind of hack like installing buildkit on the agent and loading the image already so that Prefect doesn’t try to download it. Ultimately, I don’t the the pieces from the underlying tools are there to support this yet (specifically docker-py)