https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • j

    james.lamb

    04/05/2021, 7:40 PM
    đź‘‹ hello from Chicago! At https://docs.prefect.io/orchestration/concepts/services.html#lazarus, I see this description of the Lazarus process
    The Lazarus process is meant to gracefully retry failures caused by factors outside of Prefect's control. The most common situations requiring Lazarus intervention are infrastructure issues, such as Kubernetes pods not spinning up or being deleted before they're able to complete a run.
    Once every 10 minutes, the Lazarus process searches for distressed flow runs. Each flow run found in this manner is rescheduled; this intervention by Lazarus is reflected in the flow run's logs.
    Where can I find more specifics on how a flow could enter this "distressed" state? For example, if I have a KubernetesAgent up, Prefect Cloud triggers a flow run, and then the agent isn't able to start up a k8s job because I misconfigured its RBAC stuff, would that be something Lazarus retries? Will give more context in thread.
    đź‘‹ 1
    d
    k
    • 3
    • 15
  • c

    Christoph Wiese

    04/06/2021, 9:51 AM
    I saw in the release notes for 0.14.15 that if I change
    date
    in the context, dependant context variables will now align accordingly. Does this apply to
    CronSchedule()
    as well? Currently I’m using
    CronSchedule("00 02,12 * * *", start_date=pendulum.now("Europe/Berlin"))
    - could I replace this by overriding date in the context? What is actually the best way to override date - in the configuration.toml as
    date=pendulum.today("Europe/Berlin")
    ?
    k
    m
    • 3
    • 8
  • m

    Michael Wheeler

    04/06/2021, 2:11 PM
    Hi everyone! I posted a question about the relationship between Prefect's agents and executors on Stack Overflow. I attempted to answer it myself but would really appreciate it if I could get somebody else's eyes on it. https://stackoverflow.com/questions/66959695/prefect-relationship-between-agent-and-executor
    🙌 3
    d
    k
    • 3
    • 6
  • n

    Nathan Walker

    04/06/2021, 4:44 PM
    I'm trying to dynamically update a bunch of Flow schedules using API calls. First, is there documentation for the API somewhere? I see it mentioned in the Server docs on occasion, but I want the nitty-gritty -- I have the schema and that's super helpful, but I'd like more. Second, I see that I can set_flow_group_schedule, but that doesn't include Filters. Is there a way to add Filters that I'm just missing? Third, if I want to extend API capabilities and write my own queries/mutations (i.e., adding to the schema), where would I begin looking in to this?
    d
    • 2
    • 7
  • f

    Florian KĂĽhnlenz

    04/07/2021, 8:03 AM
    Hi, is there a way to setup email cloud-hooks on a local instance of the prefect server? Or is this only available in the cloud version?
    k
    • 2
    • 5
  • n

    Nathan Walker

    04/07/2021, 2:09 PM
    Hey folks, I'm running the following mutation in the API and getting back an error message saying it expected parameter_defaults to be JSON type, but found {my_param: "my_param_value"} instead-- ObjectValueNode object has no attribute 'value'. Formatting it like JSON doesn't fix the problem. From what I've read, JSON isn't a built-in scalar type, so I'm thinking I need to declare something or wrap the parameter_defaults in something, but I'm not sure exactly what. Can somebody point me in the right direction?
    mutation {
      set_flow_group_schedule(input:{
        flow_group_id: "f67a1118-41e6-4e3b-9f14-9203e8f5ed55",
        interval_clocks: [
          {interval: 60 
           parameter_defaults: {my_param: "my_param_value"}
          }
        ],
        timezone: "America/New_York"
        }
      ) {success}
    }
    m
    • 2
    • 4
  • x

    xyzy

    04/07/2021, 3:50 PM
    Does the version of server need to be above or equal to the agent version? I know agent must be equal or above core version, but didn't find anything about server
    d
    • 2
    • 1
  • x

    xyzy

    04/07/2021, 3:54 PM
    In the DevOps Guide video on the Prefect Youtube Channel (

    https://www.youtube.com/watch?v=EwsMecjSYEUâ–ľ

    ) a webpage showing DevOps guidelines is shown. Is this page publicly available? If not, will it be in the future?
    k
    • 2
    • 3
  • h

    Hygor Knust

    04/07/2021, 6:51 PM
    Hi everyone, I'm using the Prefect Helm Chart to deploy a Prefect Server in my self-hosted Kubernetes Cluster. The setup works fine, the tenant and agent are being set by Helm. I have also created a L7 Ingress pointing to Apollo. The UI is working, but when I try to register a simple Flow, I receive this error:
    requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://{endpoint}:80/graphql/alpha
    where endpoint is the L7 ingress endpoint. Is there a way to debug what's going wrong?
    âś… 1
    k
    m
    • 3
    • 14
  • c

    Christoph Wiese

    04/09/2021, 1:51 PM
    I’m using
    database = prefect.context.get("database", "JOYN_SNOW_BI_1841")
    in one of my flows, but if I set the database key via the Cloud interface like the below, it is not being used in the flow run - anybody got an idea what I’m doing wrong? Do I maybe need to transfer the context.get into a task?
    k
    • 2
    • 5
  • n

    Nate Jahncke

    04/09/2021, 10:18 PM
    Hi everyone, I'm new to Prefect and assume it will be a great product to use once I actually have it working. Thank you for all the effort you've put into it, and for making it free to evaluate effectively. I'm attempting to integrate Prefect into an existing docker-compose development environment. Running
    prefect server start
    yields a functional deployment with a working dashboard, but calling
    docker-compose up
    on the
    docker-compose.yml
    produced by executing
    prefect server config
    does not (the UI is responsive, but the dashboard pane below the top nav bar is blank). Also, any attempt to start a local agent in an adjacent docker container gives me a "No tenant found" message via a ClientError exception. Is the
    prefect server config
    command not actually producing everything it should be at present? I'm sure I can manually build a functional docker-compose.yml by reviewing the prefect docker-compose wrapper, but that might be very time-consuming, so I'm hoping the problem is just something simple that
    prefect server config
    is leaving out. I'm going to be evaluating the two deployments back to back now to see if I can tell what's missing. Any ideas would be greatly appreciated. Thanks again!
    👍 1
    m
    m
    m
    • 4
    • 29
  • m

    Michael Wheeler

    04/12/2021, 4:52 PM
    I'm using Prefect's Docker image
    prefecthq/prefect:latest
    to spin up a container that calls
    prefect server create-tenant
    , and getting a connection error because it's pointed at localhost:4200 by default. It doesn't look like there's an option to provide another url as a CLI argument. Is there a configuration file or environment variable I can set to correct this?
    k
    • 2
    • 4
  • m

    Michael Wheeler

    04/12/2021, 5:01 PM
    Also added another Stack Overflow question, this time without an answer from me: https://stackoverflow.com/questions/67062496/prefect-server-what-is-the-role-of-a-tenant
    👍 1
    k
    • 2
    • 1
  • n

    Nathan Walker

    04/12/2021, 6:06 PM
    Hey folks, I'd like to be able to push a Flow Group's start date back arbitrarily to act as a hacky "Delay Run" method. I'm seeing that the mutation set_flow_group_schedule should do the trick, but its input parameters only give "timezone" as an option. This makes sense since you can pass in multiple clocks, but both the cron_clock_input and interval_clock_input lack start_date as well in the API. It looks like when you use the API to create/update a Flow Group's schedule, it just plops datetime.now() in as the start date, but I'd like to be able to define it per clock. Can somebody point me in the right direction on adding functionality like this to Apollo/Hasura?
    k
    • 2
    • 10
  • f

    Felipe Saldana

    04/12/2021, 11:36 PM
    I see there is a flag to set a parameters file when running a flow. I dont see this flag when registering a flow. Is there a way for me to set a parameters file for a flow so that once registered any run could default to the values from the file?
    Run a flow that is registered to the Prefect API
    
    Options:
        --id, -i                    TEXT        The ID of a flow to run
        --version-group-id          TEXT        The ID of a flow version group to run
        --name, -n                  TEXT        The name of a flow to run
        --project, -p               TEXT        The name of a project that contains the flow
        --version, -v               INTEGER     A flow version to run
        --parameters-file, -pf      FILE PATH   A filepath of a JSON file containing
                                                parameters
        --parameters-string, -ps    TEXT        A string of JSON parameters (note: to ensure these are
                                                parsed correctly, it is best to include the full payload
                                                within single quotes)
    k
    n
    • 3
    • 35
  • n

    Nate Jahncke

    04/13/2021, 12:49 AM
    Hey everyone, quick question regarding flow storage metadata: I'm running a docker-compose Prefect stack with an agent container and due to the
    storage.directory
    metadata that is automatically added to flows that are configured as mine are, I'm forced to register the flow from within the agent container, rather than just running it in vscode (which tacks on my own home dir as the
    storage.directory
    and of course yields an exception on flow execution). This certainly works, and is not a big deal at all, but I'm wondering if I can force the metadata somehow? Passing a directory to the Local() object of course won't allow me to use a path that only exists in the agent container, as it's for local storage, and while I could probably fake it and use a dir common to both, I would really prefer not to do that even if it works (as that directory in my local system would get filled up with files I don't want to exist outside of the container). Maybe what I have in mind just doesn't work? Some of this is definitely down to my two-day-old understanding of Prefect's architecture, but the real goal is just keeping absolutely everything related to Prefect in the docker environment except for flow development. If possible. As I said, not crucial as I have a totally working solution now and am eager to build some flows, but thanks in advance for any advice!
    k
    m
    m
    • 4
    • 20
  • a

    Aurélien Vallée

    04/14/2021, 6:09 AM
    Hello everyone, I am having a somewhat hard time trying to configure a self hosted server for prefect. Most of my struggle is based on what is the proper way of doing things. I started by using
    prefect server start
    but ran with a bunch of issues: • Most container services are listening on 0.0.0.0, which is not desirable in my case. I have a specific IP on which I would like exposed service to listen. • I am confused as to which services actually need to be exposed beyond the loopback. My understanding is that the UI (of course) needs to be, as well as the GraphQL endpoint (since the UI JS seems to communicate with it). What about the others? I have to admit, I am foreign to hasura/apollo/etc. Since I wanted to customize that, I figured I should somehow generate the
    docker-compose.yml
    with
    prefect server config
    and tweak that accordingly, which I did. That did not seem to be a good solution though, since even after tweaking the published ports and trial and error with the various configurations in
    docker-compose.yml
    , the UI is still trying to reach
    <http://localhost:4200>
    , even though I tried to tweak the config to have it use
    http://<server IP>:4200/
    so I guess everything is not configurable through editing the
    docker-compose.yml
    . I spent quite some bunch of time reading various threads on internet, and it seems to me now the proper way of configuring the server is not to tweak the
    docker-compose.yml
    but rather to tweak
    config.toml
    , and
    prefect server start
    will use that when generating its temporary
    docker-compose.yml
    . So I did that, but I'm still confused as to what exactly should be exposed and what should not. I also get lost in the options of
    prefect server start
    , it allows me to no publish container listened ports to the host, but of course them of them should be for the UI/agent/graphql to be available. Not sure I can even articulate the questions I have precisly, I'm just lost in the proper way to configure the server and would need some guidance.
    n
    m
    • 3
    • 23
  • a

    Aurélien Vallée

    04/14/2021, 6:14 AM
    Ideally I would like to: • Have all ports that have no reason to be exposed otherwise limited to listening on the loopback • Have all ports that should be externally accessible published on the host (UI and GraphQL from my understanding) • For all externally accessible ports, have control over which IP these ports are listening
  • a

    Aurélien Vallée

    04/14/2021, 6:15 AM
    What I did until now: • Tweak
    config.toml
    to set
    host=
    as my desired public IP for externally accessible services (:4200 and :8080) • Tweak
    config.toml
    to set
    host=127.0.0.1
    on ports that have no reason for a local deployment to be exposed (:4201 and :5432)
  • a

    Aurélien Vallée

    04/14/2021, 6:16 AM
    • Use
    --no-hasura-port
    and
    --no-postgresql-port
    on the command line
  • m

    Minakshi

    04/16/2021, 12:01 AM
    Hi team, I am trying to get prefect server to be installed on our dev boxes. It works for me locally. But our boxes don't support downloading the docker images from external websites. Where can i find the images for the services that prefect uses (postgres, hasura, graphql,apollo, towel) which i be able to upload to our internal registry? Also is there any other suggestion for doing this type of deployment?
    n
    • 2
    • 5
  • m

    Minakshi

    04/17/2021, 1:48 AM
    Whats the process to start the prefect server using a mounted volume? are there ore details on how to use the --use-volume command? How to setup a sample postgres mounted volume etc?
    m
    • 2
    • 12
  • j

    Joe McDonald

    04/19/2021, 7:07 PM
    In the ECS agent, is there a way to pass botocore_config at command line? I don’t see any option to pass that along… if not I’m writing a way to read that from a config file like run_task_kwargs because the way the settings for retry is written it overrides any other setting method… https://github.com/PrefectHQ/prefect/blob/bbb6c49bb3bfefd88e420a1b9e50565278dbbc62/src/prefect/agent/ecs/agent.py#L126
  • k

    Kevin Kho

    04/19/2021, 7:14 PM
    Hi @Joe McDonald! Unfortunately, it seems like you’re right that there is no way to pass botocore_config yet through the CLI.
    j
    m
    • 3
    • 9
  • m

    Minakshi

    04/20/2021, 4:07 AM
    Hi team, are prefect agents deployed as part of prefect server itself? Would it be okay to run the agent on the same box as prefect server? Are there any advantages of running the agent on a separate box?
    n
    • 2
    • 4
  • v

    Vladislav Bogucharov

    04/21/2021, 4:14 PM
    Hello everyone! I am trying to install prefect according to the tutorial, I run the following commands:
    python3 -m venv prefect_venv
    source prefect_venv/bin/activate
    pip install wheel
    pip install prefect
    prefect backend server
    prefect server start
    And then I get the error: subprocess.CalledProcessError: Command '[' docker-compose ',' down ']' returned non-zero exit status 255. When I run
    prefect diagnostics
    everything seems to be fine:
    {
      "config_overrides": {},
      "env_vars": [],
      "system_information": {
        "platform": "Linux-4.15.0-142-generic-x86_64-with-Ubuntu-18.04-bionic",
        "prefect_backend": "server",
        "prefect_version": "0.14.16",
        "python_version": "3.6.9"
      }
    }
    Has anyone faced this problem?
    c
    • 2
    • 3
  • g

    Garret Cook

    04/22/2021, 2:04 AM
    Is there a way to get a dockerstorage based flow to listen on a port? Similar to publishing a port when a container is run? I’ve got a device that will only backup to an ftp server. I’m wanting my flow to listen for connections on a public port. Is there a different design pattern recommended?
    k
    • 2
    • 6
  • i

    Ivan Ksaver Šušnjara

    04/22/2021, 1:37 PM
    Hello. I'm having a problem with LocalDaskExecutor employing "processes". When calling flow.run() method, following error appears:
    RuntimeError: 
            An attempt has been made to start a new process before the
            current process has finished its bootstrapping phase.
    
            This probably means that you are not using fork to start your
            child processes and you have forgotten to use the proper idiom
            in the main module:
    
                if __name__ == '__main__':
                    freeze_support()
                    ...
    
            The "freeze_support()" line can be omitted if the program
            is not going to be frozen to produce an executable.
    What is more intriguing, some flows continue and complete nonetheless, while other require manual KeyboardInterrupt (ctrl+c), and then they start executing their respctive tasks. Dask provides solution for the issue (https://github.com/dask/distributed/issues/2520), however I don't see a workaround for this in Prefect. Problem does not appear when "threads" are employed instead of "processes" with LocalDaskExecutor's scheduler argument. I'm using Prefect version 0.14.16 with Python 3.8.7. Has anyone faced the same issue? Thanks in advance!
    k
    m
    • 3
    • 13
  • j

    Joe McDonald

    04/22/2021, 3:47 PM
    So in HA setup in AWS ECS we have 3 of each of the services for server: hasura, apollo, graphql, ui, and towel. We have in another cluster three agents running on ECS. I’ve seen a log where two agents submitted execution of the same flow run. The first two lines of the log, with the same timestamp to the second, was the “Submitted for execution: Task arn:aws:ecs:…” from two different agents. Is there any type of locking occurring that should prevent this? Where in the code from server is this handled? I’m willing to look into it, but be nice to have a starting point of which service is responsible for handing jobs to agents and would allow for adding a check.
    n
    • 2
    • 4
  • e

    Enda Peng

    04/22/2021, 3:49 PM
    Is there a way to overwrite the local server port? After running
    prefect server start
    It automatically binds to 8080. I try to find .compose file but don’t get any hint. Thank you!
    k
    • 2
    • 1
Powered by Linen
Title
e

Enda Peng

04/22/2021, 3:49 PM
Is there a way to overwrite the local server port? After running
prefect server start
It automatically binds to 8080. I try to find .compose file but don’t get any hint. Thank you!
k

Kevin Kho

04/22/2021, 3:51 PM
Hi @Enda Peng! Did you see the options for the start command?
👍 2
View count: 2