https://prefect.io logo
Title
z

Zack

03/09/2023, 10:53 PM
I'm seeing odd behavior with the script below. It's producing many flow runs(I'm expecting one) and none of the tasks are being run(they're omitted from the code block example). fwiw, the script is being run inside a lambda container. This lambda is being triggered from an event produced by a different lambda. Any ideas?
def handler(event: dict, context: LambdaContext):
    """This function is the entry point for the Lambda function"""

    <http://log.info|log.info>("starting lambda function for artist ranking")

    <http://log.info|log.info>(f"event: {event},context: {context}")

    start = time.time()

    deployment = Deployment.build_from_flow(
    flow=artist_ranker_etl,
    name="artist-ranker-deployment"    
    )
    deployment.apply()
        

    run_deployment(name="artist_ranker_staging/artist-ranker-deployment",tags=["staging","ranker"])
    
    
    end = time.time()
    
    print(f"Time to execute: {end - start}")
👀 1
@Taylor Curran
t

Taylor Curran

03/10/2023, 9:37 PM
Is there a reason why you are building and applying the deployment from inside the lambda?
When you run from outside of the lambda:
run_deployment(name="artist_ranker_staging/artist-ranker-deployment",tags=["staging","ranker"])
The flow executes successfully?
z

Zack

03/10/2023, 10:17 PM
It seemed the appropriate place to put it, which is with the service logic. Is there an alternative best solution? The flow run seems to register with the cloud api server but doesn't seem to be finishing execution, none of the task details are showing up. I'll send you a screenshot when I get to a place, in transit right now.
Attached.
t

Taylor Curran

03/11/2023, 2:07 AM
oooooh
do you have an agent set up?
z

Zack

03/11/2023, 5:10 AM
hmm, I don't think so. Can I add a line to the script to set up the agent?
@Taylor Curran Following up on this.
b

Bianca Hoch

03/14/2023, 6:52 PM
Hey Zach, I'm not terribly familiar with Lambda, but I do have some thoughts on how to set up the agent. You could try creating a bash script that contains the
prefect agent start -p "work-pool-name"
command. Then, you can trigger the bash script to run from within the lambda function. The bash script would look something like:
#!/bin/sh
prefect agent start -p "work-pool-name"
My thought is that, to run a deployment as you are doing in the lambda function, you need an agent set up first. If there isn't an agent set up yet- that could explain why the flows are entering a late state.
I think an alternative to
run_deployment
would be to use the
create_flow_run
API endpoint (if you'd like to take a look at that as well). The latter is used for ad-hoc runs, which seems like what you're trying to accomplish here with the lambda function.
run_deployment
is good for creating flow runs to be executed by a remote agent.
For your reference, there's a few other examples of using AWS lambda functions here as well: https://medium.com/the-prefect-blog/event-driven-data-pipelines-with-aws-lambda-prefect-and-github-actions-b3d9f84b1309