https://prefect.io logo
Join Slack
Powered by
Not sure if this should be considered a feature re...
# ask-community
j
Not sure if this should be considered a feature request or discussion idea, but I would hope for updates to patch versions not to introduce breaking changes. I understand we’re still in initial development (0.y.z), it is disheartening to login to monitor flows to see that everything has failed in the last 24 hours because I haven’t updated updated the flows to the newest release. Specifically 0.14.15 introduced things like • 
terminal_state_handlers
on Flow objects that will cause a flow to fail if they aren’t present even though the 0.14.15 version defaults to None. • 
log_output
on the 
ExecuteNotebook
class that defaults to False on the 0.14.15 version, but the execution fails if a flow on a previous version is being run with the attribute not specified
c
Hi Josh — can you describe the errors you are seeing? From what you’ve provided here, it sounds like you are running Flows built with lower Prefect versions in environments with higher versions, which is not a pattern we recommend. We recommend that the version of Prefect that you use to build your Flow is the same that you use to run the Flow
j
Chris, you’re exactly right. I’m running scheduled flows in prefect cloud that were not updated when the it seems prefect cloud updated to 0.14.15. The scheduled flows run and, because of the mismatch, result in a Failed state of the flow.
c
To be precise: this isn’t related to Prefect Cloud updating or changing versions, but rather loose dependency management in your agent / execution environments. You’ll want to make sure that the environment / docker image being used to run your flows matches the prefect version that was used to build your flows.
Alternatively, you can use script based storage instead of pickle based storage, which I also believe will resolve these sorts of issues (and is probably easier to migrate to)
j
So, my prefect docker agent is running 0.14.12 and my docker image is built with 0.14.15. According to your comment, this will result in a failure on execution because the flow being executed tries to perform actions the agent is unable to handle?
c
When you say “my docker image” can you be more precise? What type of storage / run configuration are you using for this flow?
j
I’m using Docker storage and DockerRun as the run_config
c
Interesting, what is the traceback of the error that you are seeing
j
Copy code
Unexpected error: AttributeError("'XXXFlow' object has no attribute 'terminal_state_handler'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/flow_runner.py", line 673, in get_flow_run_state
    terminal_states=terminal_states,
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/flow_runner.py", line 720, in determine_final_state
    if self.flow.terminal_state_handler:
AttributeError: 'XXXFlow' object has no attribute 'terminal_state_handler'
c
Ah, what base image are you using for docker storage? You are most likely running Prefect version A when you build the Flow and your Docker storage is using version B, where B > A. The simplest way to resolve this is to include 
stored_as_script=True
in your Docker storage initialization
j
When I inspect the docker image, I see that it is using
Copy code
"Labels": {
                "io.prefect.python-version": "3.7.10",
                "org.label-schema.build-date": "2021-03-10T23:07:06Z",                                                               "org.label-schema.name": "prefect",
                "org.label-schema.schema-version": "= 1.0",                                                                          "org.label-schema.url": "<https://www.prefect.io/>",
                "org.label-schema.vcs-ref": "dea6358fe1bf623965d9538372c1cb5a76fd19a7",
                "org.label-schema.version": "0.14.12"
            }
c
Your traceback proves that can’t be correct, as terminal state handlers were only added in 0.14.15 so that label is wrong for some reason
j
While building, I install the requirements for the project on top of the base image. When I run 
prefect version
after attaching to a container run of the image, I get 
0.14.15
c
yup, makes sense and tracks with my explanation
j
What does 
stored_as_script
option do? I see if True, it stores the script as 
.py
but what is the alternative if False?
one of my build steps is to copy over the flow files from the repo into the image, is that effectively the same as 
stored_as_script
?
c
if False, it stores the Flow as a pickle which means everything about the Python object is saved as-is. If you then unpickle this artifact in other versions of Prefect (or any other packages that your flow relies on), unexpected things could happen (this is a true statement independently of Prefect and is why systems like Dask require all workers to run with identical Python environments). If you store the Flow as script, it will get re-initialized with each run, meaning that new attributes, etc. will be appropriately handled
j
got it. So even though I’m copying the flow files manually, because I’m not specifying the 
path
attribute in my Docker Storage initialization, it’s still using the pickled Flow?
c
Yea, unless you set 
stored_as_script=True
it will always use the pickled flow artifact
I fully understand that this can be confusing, so I’ll archive this thread for others to find @Marvin archive “Issue with upgrading to 0.14.15 using Docker storage”
👍 1
m
j
So to ensure my execution agent and the flows being executed are always matching, I should either 1. Pin the prefect dependency in the repo so when it builds, it’s always the same version until I manually upgrade for new features. If I upgrade in the repo, new images of flows will be a new Prefect version, so I’ll need to upgrade the agent. 2. Add 
stored_as_script=True
so that the python script itself is executed on the agent? Will that mean the script is executed on the agent’s prefect version, or the built image’s prefect version?
If my Agent Version is 0.14.12 and. my Docker Storage Flow Version is 0.14.15, but I’ve used 
stored_as_script=True
, will it still lead to issues?
c
Correct in spirit but note that your agent’s environment isn’t really related to your flow’s runtime environment when using a Docker agent. The Docker agent only creates the Docker containers that your Flows run in, so as long as the agent is running the same minor version as your flow (e.g., they both are roughly on 0.14.*) you won’t experience any issues. In general, we recommend upgrading agents regularly, independently of your flows. 1. Yup, this is the most secure way to deploy because your build environment will always match your runtime environment (note that you may consider a full requirements.txt pin for other packages that you depend on too for extra assurance). 2. This is the easiest; note that the flow is still going to run within the Docker image that you built it with, not in the agent’s environment. The only difference is that the Flow will get re-initialized in the new version of Prefect within the Docker image (pickled objects are not re-initialized)
an agent running 0.14.12 can still run docker storage flows built with 0.14.15, for example, regardless of the value of stored_as_script
j
Thanks for explaining. Seems like doing 1 & 2 would provide even better peace of mind.
Thanks for your help!
😄 1
c
Anytime!!
Open in Slack
PreviousNext
Powered by