https://prefect.io
Join Slack
Hello everyone, I ‘m trying to run my flow using AWS ECS Fargate (following the great step-by-step o...
# ask-community
a
Hello everyone, I ‘m trying to run my flow using AWS ECS Fargate (following the great step-by-step of Anna Geller: https://towardsdatascience.com/serverless-data-pipelines-made-easy-with-prefect-and-aws-ecs-fargate-7e25bacb450c#3e93). When running the flow, I’m getting an unplickling error 
Failed to load and execute Flow's environment: UnpicklingError("invalid load key, '{'."
I’ve read the Slack thread that referred this error: https://prefect-community.slack.com/archives/CL09KU1K7/p1623777537484700?thread_ts=1623704787.418000&cid=CL09KU1K7 which seems to point to version compatibility issues. My setup : • Agent Prefect version 0.15.0 • Running on Prefect Cloud (“core_version”: “0.14.22+9.g61192a3ee”) • My task is a simple hello-world log, deployed with 
with Flow("s3_flow", storage=S3_STORAGE, run_config=ECSRun_CONFIG) as flow:
• 
is_serializable(flow) = True
Should I be downgrading my agent to match the server version ? Any help will be very welcome !
k
Hey @Alain Prasquier, could you try bumping the core version up and re-registering?
a
Can I do this on Prefect Cloud ?
k
No this has to be done on the Flow side.
Actually before you do that, I remember for Verun it was that his agent was not the version he thought it was.
a
prefect version  I
0.15.0
k
Is this the command line 
prefect version
?
a
yes
k
Yeah I would suggest trying matching versions for Flow and Agent. Either upgrade the flow registration or downgrade the agent.
a
flow registration and agent are both running 0.15.0
k
Could you also post your flow here so I can give it a shot? What Python version do you have?
a
Python 3.8.5
Copy code
import prefect
from prefect.storage import S3
from prefect.run_configs import ECSRun
from prefect import task, Flow
from prefect.utilities.debug import is_serializable


TASK_ARN = "arn:aws:iam::1234567899:role/ECSTaskS3ECRRole"
RUN_CONFIG = ECSRun(labels=['s3-flow-storage','fargate-dev'], task_role_arn=TASK_ARN, image='anisienia/prefect-pydata', memory=512, cpu=256)
STORAGE = S3(bucket='etl-bucket')


@task
def say_hello():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("START SAY HELLO")
    
with Flow("s3_pandas", storage=STORAGE, run_config=RUN_CONFIG) as flow:
    say_hello()
    

print(is_serializable(flow))
flow.register(project_name="etl-test")
on agent, python 3.7.10
k
Oh I think that might be the issue. You probably need the same Python version for the agent and registration.
a
yes, this makes sense.... let me try this !
unpickling from different version of python might be problematic
both agent and flow registration are running python 3.8.5 and I got the same error again.
k
Could you try 
LocalRun
first instead of 
ECSRun
so we know it’s not ECS related? Storage can still be S3. Just do 
RUN_CONFIG = LocalRun()
first?
a
Hi @Kevin Kho, I’m pinpointing the problem: I tried unplickling manually the file that is sent to s3 (see image) and I get the same error :
Copy code
_pickle.UnpicklingError: invalid load key, '{'.
The issue seems to be in the pickling step.
I’ll try the localRun
k
I expect it to break. I’ll test with LocalRun with these versions on both flow and agent
Just making sure, you only have one agent that can pick up the flows right now right?
a
yes
Not sure how to execute “LocalRun” : the flow gets registered but execution remains “pending...”
@Kevin Kho is star 🌟 - thanks for the amazing support !
k
For posterity: something is wrong with the serialization with 0.15.0, S3 Storage, and ECSRun. Will look into it and open an issue.
👏 1
After digging in, the issue is that image is Prefect 0.14.1. Updating the image to something later with 
prefecthq/prefect:latest-python3.8
will fix this I think
It runs with the mismatched versions (which works sometimes), but it’s that serialization/deserialization of mismatched versions that didn’t work since this in run on the container. It still would be good practice to update the image.
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by