Shriram Holla

    Shriram Holla

    3 months ago
    Hi, When I run a task locally using
    flow.run()
    , it works just fine. However, I’m running a local backend server and when I try to register the task using
    flow.register()
    , I get this error:
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 476, in save
        f(self, obj) # Call unbound method with explicit self
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 736, in save_tuple
        save(element)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 476, in save
        f(self, obj) # Call unbound method with explicit self
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 821, in save_dict
        self._batch_setitems(obj.items())
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 847, in _batch_setitems
        save(v)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 476, in save
        f(self, obj) # Call unbound method with explicit self
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 821, in save_dict
        self._batch_setitems(obj.items())
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 847, in _batch_setitems
        save(v)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 496, in save
        rv = reduce(self.proto)
      File "/Users/shriramholla/.pyenvs/python3.6/lib/python3.6/site-packages/absl/flags/_flagvalues.py", line 677, in __getstate__
        raise TypeError("can't pickle FlagValues")
    TypeError: can't pickle FlagValues
    Note: Nothing is being passed into or being returned by any task
    Anna Geller

    Anna Geller

    3 months ago
    Where did you get this syntax from? Prefect 1.0 has flow.register() - no need to register tasks separately
    Shriram Holla

    Shriram Holla

    3 months ago
    Oops sorry I meant
    flow.register
    Anna Geller

    Anna Geller

    3 months ago
    Shriram Holla

    Shriram Holla

    3 months ago
    If I use a script based flow storage, will it allow me to run concurrent tasks?
    Anna Geller

    Anna Geller

    3 months ago
    Sure, yes - storage is independent of the executor (in 2.0 task runner)
    Kevin Kho

    Kevin Kho

    3 months ago
    You are just returning something in a task that can’t be pickled. You can turn off checkpointing for the task too:
    @task(checkpoint=False)
    def mytask():
        return FlagValues
    Shriram Holla

    Shriram Holla

    3 months ago
    I tried that but it didn’t work. That’s the fishy part because I’m not returning anything from any task.
    Kevin Kho

    Kevin Kho

    3 months ago
    Ah sorry. Anna is completely right. If you have a global variable that is used by your tasks, it gets serialized along with the Flow
    Shriram Holla

    Shriram Holla

    3 months ago
    I see
    So here’s where I’m at right now. I can run a flow locally but cannot register it due to the above issue. I fixed this by using a script based flow storage. However, that requires a python script as an entrypoint. But since that would mean that it doesn’t run through our build system, I get import errors. The build system we use generates modules during compilation and converts them into executables. Do you know if there’s an alternate solution?
    Anna Geller

    Anna Geller

    3 months ago
    you can avoid that by switching to script storage - the docs I shared shows that if you register via CLI, it uses script storage by default:
    prefect register --project xyz -p yourflow.py
    Shriram Holla

    Shriram Holla

    3 months ago
    Yeah that’s what I used, which ended up giving me import errors.
    Essentially, do you know if there’s a way to do
    bazel run <register_flow_script>
    without specifying a python file
    Anna Geller

    Anna Geller

    3 months ago
    I don't know bazel 😃 but you would need Prefect CLI to trigger Prefect action and bazel CLI to trigger bazel action