https://prefect.io
Join Slack
Hey all, I'm running into an issue with prefect cloud submitting jobs but not doing anything after t...
# ask-community
c
Hey all, I'm running into an issue with prefect cloud submitting jobs but not doing anything after that. The issue first presented with a new flow I was writing. I restarted my agent and I upgraded from 14.6 to 14.12. This only had the effect of spreading the issue to the rest of my flows. I was wondering if you could offer me some guidance in debugging this issue.
The place I'm running into an issue seems to be after a container is submitted - here's a flow from yesterday that succeeded vs where I'm hitting an issue now
I don't think the cloud flow runner is something I have control over. I just generated a new token and restarted my agent with that, it doesn't seem to have helped
m
Hello Chris! Looks like the issue is not with agent because your flow is submitted. Can you provide more information how you run your flow or redacted/simplified version of your flow?
c
I can do that, for sure
so here's the flow I was working on that started giving me issues
Copy code
from prefect import Flow
from prefect.engine.results import LocalResult
from ..app.data_object import DataObject
from prefect.schedules import CronSchedule
from .reference_handlers import cloud_only_slack_handler
from .reference_tasks import check_api, transform, import_to_database


# daily_schedule = CronSchedule("12 4 * * *")

with Flow(
        "draft_prospect_link_flow", 
        # schedule=daily_schedule, 
        result=LocalResult(), 
        state_handlers=[cloud_only_slack_handler]
        ) as flow:

    data_object = DataObject('draft_prospect_link', 'prospects')
    check_api_result = check_api(data_object=data_object)
the task that's imported looks like this
Copy code
@task(result=LocalResult(), tags=['api'])
def check_api(data_object, save_to_s3=False, result=LocalResult()):
    handler = ApiHandler(
            data_object,
            context=prefect.context,
            save_to_s3=save_to_s3)
    handler.get_file()
    return handler.data_object
currently I'm running flows on prefect cloud. I've got an agent started with this
Copy code
prefect agent docker start \
	--log-level DEBUG \
	--volume "${WORKDIR}/.config.toml:/opt/prefect/config.toml" \
	-t "${token}"
I'm kicking this flow off with the "quick run" button on cloud.prefect.io
so I think I found the issue. I was adding a block ot the 
context.secrets
block in my .config.toml file, and either I started with an old version of the file or something's changed since I added a new flow
either way, this change in my config file got me to working flows again
Copy code
auth_token = abcdefghijklmnop
-> 
auth_token = 'abcdefghijklmnop'
m
I'm glad you found and solved the issue 😁
c
me too!
my current theory is that my old agent was started before some change happened and was grandfathered in. but when I wanted to add some secrets for a new API, I had to restart the agent. In between agent starts, the quoting necessary on the 
auth_token
changed, new agent couldn't submit containers, and that's why my flows stopped.
does that make sense? I don't follow the cloud changelog too closely
also it's entirely possible that the config file I update is from some olde tyme, and I overwrote one that I had updated with the quotes in production
since it's not under version control cause secrets
m
I had similar issue, and as I remember toml files require to wrap strings in double or single quotes
c
my main question for myself is, why did past me think it was a good idea to have that string not be quoted?
don't really have a good answer to that, but I'm glad it's resolved
👍 1
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by