• Domantas

    Domantas

    1 year ago
    Hello guys! Recently I have dealt with a problem running my prefect code in the server UI:
    Failed to load and execute Flow's environment: StorageError('An error occurred while unpickling the flow:\n AttributeError("type object \'Members\' has no attribute \'ids\'")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - prefect: (flow built with \'0.14.15\', currently running with \'0.14.12\')')
    Funny thing, that running locally(flow.run()) this error do not appear and it only occurs when executing code via server UI(will attach a screenshot with the error log). For the context, I'm running Rust code in the python script which seems the main problem when using UI(running locally it works perfectly fine, so code itself seems to be good). Maybe anyone knows how to solve this problem? If there are some relevant information missing or there is already another thread is created related with this problem - let me know!
    Domantas
    x
    +1
    13 replies
    Copy to Clipboard
  • Igor Bondartsov

    Igor Bondartsov

    1 year ago
    Hello! I am using AWS Fargate. I added to my agent env_vars. How I can import Secrets from agent to flow which will be run in the container?
    Igor Bondartsov
    Zach Angell
    +1
    11 replies
    Copy to Clipboard
  • s

    Sven Teresniak

    1 year ago
    I have 2 questions regarding the new
    prefect register
    :1. What is the purpose of
    prefect build
    ? I think I understand what it does. But when do I want to use
    prefect build
    +
    prefect register --json
    ever? 2. Is there any difference between
    prefect register -p flow.py
    on a flow file and calling
    flow.register(…)
    as part of a
    python flow.py
    call? The latter seems much more dynamic (e.g. do some initialization/behaviour changes in
    __main__
    )
    s
    Michael Adkins
    4 replies
    Copy to Clipboard
  • Rowen

    Rowen

    1 year ago
    Hi all, I am trying to run a very simple flow. It has succeeded when I run it locally by adding 
    flow.run()
     at the end of my python file. However, when I trigger the flow in the prefect cloud UI, it fails at the 
    extract
     stage, the error being 
    At least one upstream state has an unmappable result
    . Below is the code. I will elaborate more in the threads
    @task
    def transform(x):
        return x + 30
    
    
    
    @task
    def extract():
       return [200, 400, 500]
    
    
    
    with Flow("flow-name", storage=S3(bucket="bucket_name")) as flow:
        e = extract()
        t = transform.map(e) # fails when i trigger flow in the UI
    Rowen
    Zach Angell
    16 replies
    Copy to Clipboard
  • Stéphan Taljaard

    Stéphan Taljaard

    1 year ago
    Hi. How would you recommend limiting concurrency of a mapped task (900+ DB queries) using Prefect Server?
    Stéphan Taljaard
    Kevin Kho
    8 replies
    Copy to Clipboard
  • Jonas

    Jonas

    1 year ago
    Hello, I am learning Prefect and I am trying to make a flow inside a flow which takes a value/parameter from the above flow. I got a database file with a flow which fetches a dataframe and I want to pass that data to a new flow in another file to work with the data. I came across this doc https://docs.prefect.io/core/idioms/flow-to-flow.html, but I am still not sure how you could share data between flows.
    flow_database = StartFlowRun(flow_name="database", project_name="test", wait=True)
    flow_transform = StartFlowRun(flow_name="transform", project_name="test", wait=True)
    
    with Flow("parent-flow") as flow:
       
        data = flow_database() #get a dataframe from flow
    
        flow_transform(data) # take dataframe and use it another flow
    How should I go about this?
    Jonas
    ale
    +1
    3 replies
    Copy to Clipboard
  • c

    Charles Liu

    1 year ago
    Is it possible for the S3Upload feature to return an s3 path to an unpickled file?
    c
    Kevin Kho
    +1
    27 replies
    Copy to Clipboard
  • Adam Lewis

    Adam Lewis

    1 year ago
    Hi everyone, does Prefect have the concept of flow priority if more flows are scheduled to run than there are computational resources to run them all simultaneously? I've been looking in the docs but not able to discover it yet. Thanks!
    Adam Lewis
    1 replies
    Copy to Clipboard
  • m

    Matthew Blau

    1 year ago
    Hello all, we are using a .env file for secret management with prefect instead of the config.toml and are running into issues with injecting updated .env values at runtime with LocalRun(env=env_dict). The only way that the flow picks up the new .env contents is upon registration of the flow, which is not ideal as we need the .env file to be written to during runtime and the new values read from the same file. Any pointers on how to have Prefect pick up the new values? The flow creates a container and injects env values into it with
    enviornment=env_vars
    in the CreateContainer() function but we need to have it inject updated values at runtime. I have had success with
    flow.run_config = DockerRun(env={f"PREFECT__CONTEXT__SECRETS__{k}": v for k, v in config.context.secrets.items()})
    but we are not utilizing prefect's config.toml for this project as well as the flow not being executed within a docker container
    m
    Michael Adkins
    4 replies
    Copy to Clipboard
  • Ricardo Portela

    Ricardo Portela

    1 year ago
    Hello people! How you doing?. I would like to know There is a way with Prefect to prevent a flow execution if that flow was executed early in that day successfully?. please feel free to share github samples as well. Thanks in advance.
    Ricardo Portela
    Kevin Kho
    +1
    15 replies
    Copy to Clipboard