https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • a

    Aqib Fayyaz

    11/16/2021, 6:48 PM
    HI, i have set the prefect server using helm chart on gke following this video

    https://www.youtube.com/watch?v=EwsMecjSYEU&t=332s▾

    on prefect channel. I can see on my cluster prefect server and all other dependencies like apollo etc are deployed including prefect agent that i deployed using --set flag with helm install. For the first time this ui comes on prefect server
    🙌 2
    k
    23 replies · 2 participants
  • t

    Tom Shaffner

    11/17/2021, 4:53 PM
    Hi all, I'm new to Prefect and trying to set up a local server on an Azure (linux) VM inside a VNET. I'm having issues getting the up and running in the first place. I'm in a corporate environment; unfortunately this means MANY blocked ports. If I do an NMAP scan on the VM I get the first picture below. To even see the server I have to use the expose flag, and set the ui port to 80. With some testing it seems if I set the server port to 1720 it all starts okay (
    prefect server start --expose --ui-port=80 --server-port=1720
    ), I can at least get to the UI then, but the UI gives me a failure message about not connecting to graphql and a link to this slack (second picture below). I've tried setting the graphql port flag 5060 or 443 or even also to 1720 or using the no port mapping flag (https://docs.prefect.io/api/latest/cli/server.html), all to no avail. I'm new to this so I'm probably doing something stupid but I can't figure out what; any thoughts? In case it's related, I'm also unable to start a user agent. When I try
    prefect agent local start
    I get
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=1720): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d60a34e20>: Failed to establish a new connection: [Errno 111] Connection refused'))
    . I'm unclear on whether the server is automatically starting agents of its own or if I'd have to do this myself in addition to the server. I could potentially have the option to open some additional ports, but it takes several days, a long approval process that might get rejected, and I'm not even clear on which I'd need to open or why. After all, these are mostly internal ports to the same machine, right? I know that's a lot but I'm a bit at a loss here; any help?
    k
    a
    72 replies · 3 participants
  • a

    Aqib Fayyaz

    11/18/2021, 10:58 AM
    i have set the prefect server and agent using helm chart on gke. I have set the value in .prefect/config.toml
    [server]
    host = "http://<EXTERNAL-IP>"
    as mentioned in docs to point the cli to server. i have created the tenat using command
    prefect backend server && prefect server create-tenant --name default --slug default
    now when i open the server ui i shows the getting-started page and if i click on dashboard
    a
    9 replies · 2 participants
  • a

    Aqib Fayyaz

    11/18/2021, 3:16 PM
    Don't we have secret option available for server Ui? as my flow is stored on github repo which is private so i need secretes there
    n
    k
    +1
    6 replies · 4 participants
  • t

    Tom Klein

    11/19/2021, 2:16 PM
    Hello, i'm trying to test out prefect locally on a linux machine and running into something (that it seems a bunch of others have run into) but not sure how to solve it
    a
    r
    +1
    15 replies · 4 participants
  • t

    Tom Klein

    11/19/2021, 2:21 PM
    and, on a separate issue, when i try to test out prefect cloud and lift up a local agent (according to instruction given) - i'm seeing the following:
    k
    5 replies · 2 participants
  • a

    Anders Lindstrom

    11/19/2021, 7:05 PM
    I have a basic question about the use of
    Module
    storage in a Docker container. I've been trying to get it working for half a day now and I'm stumped. I'm sure it's a very simple problem but I'm unable to find any working examples as a basis. So I have a flow that is installed (via
    pip install
    ) in a Docker image. The flow is called
    flows.docker_2
    . It looks like this:
    import prefect
    from prefect import task, Flow
    from prefect.run_configs import DockerRun
    from prefect.storage import Module
    
    @task
    def say_hello():
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>("Hello, docker 2!")
    
    with Flow("docker_2_flow") as flow:
        flow.storage = Module("flows.docker_2")
        flow.run_config = DockerRun(
            image="prefect_docker_2_image",
            host_config={
                "auto_remove": False
            }
        )
        say_hello()
    
    if __name__ == "__main__":
        flow.register(project_name="docker_2_project", labels=["docker_flows"])
    I am able to register the flow by executing it inside the container:
    docker run --network prefect-server --rm --name prefect_docker_2 -it prefect_docker_2:latest python -m flows.docker_2
    So I know that the module is installed and can be run inside the container. However, when I run the flow, the module is not found. The docker agent reports:
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
    ModuleNotFoundError: No module named 'flows'
    Is there something else I have to do (other than `pip install`ing the module) to make the flow code visible?
    k
    5 replies · 2 participants
  • o

    Ovo Ojameruaye

    11/20/2021, 4:44 AM
    Hi everyone, This might sound pretty straight forward but I have spent a few days on it but I am not sure what's possible. Is it possible to run prefect server in a container (I have read that running docker compose in a container doesn't work)? Q2: Has anyone been able to run prefect server in an Azure Web App? Thanks
    a
    a
    3 replies · 3 participants
  • r

    Ryan Sattler

    11/22/2021, 5:42 AM
    Hi, I’m trying to get some basic (short-term) persistence going with the included Postgresql in the Prefect Server Helm Chart. When I change
    postgresql: persistence: enabled:
    from
    false
    to
    true
    in the chart and
    helm install
    , the chart appears to install successfully as normal and the various server pods are created, but the Postgres pod never appears and there are no error messages, nothing in
    kubcetl get deployments
    or
    kubectl describe replicasets
    etc. With the value
    false
    it works fine (except for the part where it loses all data when the pod dies). Does anyone know how to debug this?
    a
    3 replies · 2 participants
  • s

    Sylvain Hazard

    11/22/2021, 11:01 AM
    Hello ! Is it possible to use the
    create_flow_run
    paradigm to create multiple flow runs with differing parameters using
    map
    ? Been trying it out and can't seem to make it work. Code below.
    a
    k
    16 replies · 3 participants
  • s

    Sylvain Hazard

    11/22/2021, 2:13 PM
    Hey again, Got another question, sorry for bothering. How can I get the result of a subflow to pass as input to another task ? I'd like to do that because I have a specific task in my flow that is quite resource hungry and I would like to run it as a separate flow in order to optimize costs. The issue is that the output of this task is currently used downstream and I'd like to do the same with it running as its own flow. I may have missed it in the documentation, but I don't think I saw this use case described. Basically, I'd like to do something like this :
    with Flow(name="Parent Flow") as flow:
        docs = Parameters("docs", default=[1, 2, 3])
        small_task = SmallTask().map(docs)
        big_task = create_flow_run.map(flow_name=unmapped("Child flow"), project_name="whatever", parameters=small_task)
        wait_big_task = wait_for_flow_run.map(big_task, raise_final_state=unmapped(True))
        another_small_task = AnotherSmallTask().map(wait_big_task)
    but I'm not sure how or where to define the result of a flow in order to use it downstream.
    a
    2 replies · 2 participants
  • v

    Vaibhav Ariyur

    11/22/2021, 3:00 PM
    shown left: I run a flow, flow enters subflow, completes successfully, and enters the "clean" task successfully when there is a constantly running "keep-alive" task running in parallel. The keep-alive task just logs "keeping alive" every 10 seconds for an indefinite period of time. shown right: I run a flow, flow enters subflow, subflow completes successfully but the main flow does not understand that (hence the grey status for the subflow), main flow restarts instead of entering the clean task (hence the greyed out box in one of the prior tasks --> where I cancelled this flow). Anybody understand why this happens, and a non-hackish fix for this?
    k
    m
    +1
    16 replies · 4 participants
  • v

    volkerjaenisch

    11/22/2021, 11:14 PM
    Hi I am Volker from Inqbus Scientific Computing. We are currently evaluating Prefect/Orion as the future workflow engine for a not so small customer. Prefect is out of question but we like to give Orion a try. We have some deeply reaching questions about Orion: Class decorators, optimization without knowing the DAG beforehand, future development of the UI etc. Where do we place these questions best?
    k
    4 replies · 2 participants
  • v

    volkerjaenisch

    11/22/2021, 11:16 PM
    @Kevin Kho Thanks for the fast answer!
  • v

    volkerjaenisch

    11/22/2021, 11:17 PM
    OK here we go. I noticed that the UI of Orion is way back behind the funtionality of Prefect. When can one expect a catchup?
    k
    m
    31 replies · 3 participants
  • d

    Diego Alonso Roque Montoya

    11/23/2021, 2:28 AM
    Has feature development fully moved to Orion? When can we expect a stable version for Orion?
    k
    8 replies · 2 participants
  • s

    Stéphan Taljaard

    11/23/2021, 8:36 AM
    Hi. This sounds like a noob question. How do I change the default folder for results? (I don't mind, or actually want to, move the whole config folder) I've tried adding
    home_dir = "/mnt/disks/prefect_data/.prefect"
    to
    .prefect/config.toml
    , but results are still saved on my main disk
    a
    k
    8 replies · 3 participants
  • b

    Bruno Kuasney

    11/23/2021, 10:26 AM
    Hi, honestly I’m not sure if it’s prefect related, but, when I try to run the graphql query to turn off a specific flow:
    mutation {
      set_schedule_inactive(input: {flow_id: "67d364da-375d-4d0f-b151-d46435351c50"}) {
        success
      }
    }
    I get:
    "message": "field \"set_schedule_active\" not found in type: 'mutation_root'"
    I’m basically trying to use the query from the documentation here
    a
    17 replies · 2 participants
  • s

    Sylvain Hazard

    11/23/2021, 10:45 AM
    Hey ! Are tasks results available in flow-level state handlers ? I'd like to use those to generate some kind of report and send it to Slack.
    a
    4 replies · 2 participants
  • a

    Alexis Lucido

    11/23/2021, 3:58 PM
    Hello all. I have been trying to configure Prefect Server on a Debian/GNU Linux 10 VM and I encounter a problem. I have a conda env with prefect installed, docker and docker-compose installed as well. I can launch the server with prefect server start --expose (cannot seem to access the UI from my computer without the "--expose" part), but I got a "Couldn't connect to Prefect Server at http://localhost:4200/graphql" issue. It seems to be the same problem that the one Krishna had here: https://prefect-community.slack.com/archives/CL09KU1K7/p1636636632389000. I have attached the return of a prefect diagnostics command, and it seems that my docker containers have enough memory (16GB per container). My company is against the fact that I use Prefect Cloud. Finally, I can run Flows manually with 'prefect run -n "flow_to_test" --execute'. Anyone has any idea about my issue? Thank you very much!
    a
    m
    9 replies · 3 participants
  • s

    Scarlett King

    11/23/2021, 5:45 PM
    Hey, so I’m using prefect server on AKS. For the job run, I want to pass a few environmental variables to the image by default. What is the best way to add this? On my local machine, I start the kubernetes agent with —env, can I do similar thing for the Kubernetes agent deployed on AKS?
    j
    1 reply · 2 participants
  • l

    Lana Dann

    11/23/2021, 7:45 PM
    Hi there! I’m attempting to set up an ECS Agent as an ECS Fargate service but I need to scope the IAM permissions to the minimum requirements. I was wondering if someone could explain why we need these permissions so that I can figure out how to scope them out:
    "ec2:AuthorizeSecurityGroupIngress",
    "ec2:CreateSecurityGroup",
    "ec2:CreateTags",
    "ec2:DescribeNetworkInterfaces",
    "ec2:DescribeSecurityGroups",
    "ec2:DescribeSubnets",
    "ec2:DescribeVpcs",
    "ec2:DeleteSecurityGroup"
    a
    10 replies · 2 participants
  • j

    Joël Luijmes

    11/24/2021, 11:09 AM
    Hey there! I’m trying out the artifact task, and I think it works well. However, I’m trying to get it work with larger artifacts. I submitted earlier a PR to make Apollo’s body size configurable. But I’m now running into another limitation. In the code there is a hard coded limit of 1Mb. I’m willing to submit a PR, but am unfamiliar with this codebase (with graphql in general actually). Can someone point me how to make this limit check configurable?
    if data_size > 1000000:  # 1 mb max
        raise ValueError("Artifact data payload exceedes 1Mb limit.")
    a
    8 replies · 2 participants
  • a

    Alexis Lucido

    11/24/2021, 12:52 PM
    Hi all. Another question. Sorry for bothering. So I am trying to use a custom PostgreSQL database. I have set the a prefectdb postgre db with a username prefect and a given password. I have add the following lines of code into my config.toml file: [server.database] connection_url = "postgres://prefect:_mypassword_@0.0.0.0:5432/prefectdb" I launch my server with the command : prefect server start --external-postgres --expose However, I have the following error log coming up: hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"0.0.0.0\" I believe there is a mistake in my host definition, or a docker container link issue. I have tried to replace 0.0.0.0 with localhost and the hostname of my virtual machine, but I still find the same problem. Is there any more changes to add to the prefect server config file? Moreover, I have another error with the graphql container: Error: Can't load plugin: sqlalchemy.dialects:postgres And I am wondering whether I should set the connection_url element in config.toml as connection_url = "postgresql://...", as stated here: https://stackoverflow.com/questions/62688256/sqlalchemy-exc-nosuchmoduleerror-cant-load-plugin-sqlalchemy-dialectspostgre Could you please help me solve that issue? Thanks in advance, again!
    a
    2 replies · 2 participants
  • m

    Mariusz Olszewski

    11/24/2021, 2:38 PM
    Hi, I can not find anywhere info about how to upgrade prefect server and UI to the latest version. Can someone paste here the snippet?
    a
    1 reply · 2 participants
  • s

    Sylvain Hazard

    11/24/2021, 3:16 PM
    Hello there ! I'm having trouble using
    get_task_run_result
    to access data from a subflow. This example should reproduce the issue :
    from prefect import Flow, task, Parameter, unmapped
    from prefect.tasks.prefect import create_flow_run, get_task_run_result
    
    @task(slug="plus-one")
    def plus_one(x: int):
        return x+1
    
    with Flow("Child") as child_flow:
        x = Parameter("x", default=10)
        po = plus_one(x)
    
    @task
    def get_params():
        return [{"x": i} for i in range(5)]
    
    with Flow("Parent") as parent_flow:
        params = get_params()
        flow_run_ids = create_flow_run.map(flow_name=unmapped("Child"), parameters=params)
        results = get_task_run_result.map(flow_run_ids, unmapped("plus-one"))
    When running the parent flow, I get issues that look like :
    Error during execution of task: ValueError("No task runs found while querying for task runs where {'task': {'slug': {'_eq': 'plus-one'}}, 'flow_run_id': {'_eq': '3316e187-adda-461d-814f-29a6ffc059a4'}, 'map_index': {'_eq': -1}}")
    Is there something I'm missing with how slugs work ?
    a
    6 replies · 2 participants
  • a

    Aleksandr Glushko

    11/24/2021, 4:34 PM
    Hi all! Could anyone explain please, if I register flow with some parameters, for example: a=10. And in the flow i have:
    with Flow('flow_name') as flow:
      a = prefect.Parameter('a', 'required'=True)
      <http://logger.info|logger.info>(a)
    how can i get the parameter value to be used? So i want
    a
    to be 1, but I get:
    a = <Parameter: a>
    Im running the flow in the following way:
    prefect_client.graphql(query= """
                                    mutation{
                                        create_flow_run(input: { flow_id: %s , parameters: %s}) {
                                        id
                                        }
                                    }
                                    """ % (train_flow_id, json.dumps(flow_params).replace('"', '\"'))
                                    )
    where flow_params is a dictionary
    a
    4 replies · 2 participants
  • s

    Saurabh Indoria

    11/25/2021, 3:23 AM
    Hi All, We are using Prefect Cloud along with Kubernetes agent for our deployment. Right now, we just have one prefect-agent pod running in Kubernetes, which in turn creates Kubernetes Jobs for all flow runs. I was wondering if this is the best approach? Should we run multiple prefect-agent pods instead? We are very close to our production release, and would really appreciate your inputs!
    k
    2 replies · 2 participants
  • r

    Ryan Sattler

    11/25/2021, 3:40 AM
    Hi, I’m trying to connect the Prefect Server Helm Chart to an existing postgres database. I’ve configured the username and password fields inside the values file. However my pods fail to start with this error:
    Error: secret "prefect-server-postgresql" not found
    . I thought this secret would be created automatically? Am I supposed to create it manually and if so how?
    k
    a
    13 replies · 3 participants
  • q

    Qwame

    11/25/2021, 11:19 AM
    Hi, I'm new to prefect here and my question about the server is anytime I start the server and UI in a docker container, my flows from the previous runs disappear. How do I get around this?
    a
    1 reply · 2 participants
Powered by Linen
Title
q

Qwame

11/25/2021, 11:19 AM
Hi, I'm new to prefect here and my question about the server is anytime I start the server and UI in a docker container, my flows from the previous runs disappear. How do I get around this?
a

Anna Geller

11/25/2021, 11:25 AM
@Qwame the easiest way would be to sign up for a free tier of Prefect Cloud. You get 20,000 free task runs each month (no credit card required). This way, you don’t have to worry about setting up and maintaining the UI and the underlying database persistence layer: https://universal.prefect.io/signin/register
View count: 1