prefect-community #prefect-server

james.lamb

04/05/2021, 7:40 PM

👋 hello from Chicago! At https://docs.prefect.io/orchestration/concepts/services.html#lazarus, I see this description of the Lazarus process

The Lazarus process is meant to gracefully retry failures caused by factors outside of Prefect's control. The most common situations requiring Lazarus intervention are infrastructure issues, such as Kubernetes pods not spinning up or being deleted before they're able to complete a run.

Once every 10 minutes, the Lazarus process searches for distressed flow runs. Each flow run found in this manner is rescheduled; this intervention by Lazarus is reflected in the flow run's logs.

Where can I find more specifics on how a flow could enter this "distressed" state? For example, if I have a KubernetesAgent up, Prefect Cloud triggers a flow run, and then the agent isn't able to start up a k8s job because I misconfigured its RBAC stuff, would that be something Lazarus retries? Will give more context in thread.

👋 1

Christoph Wiese

04/06/2021, 9:51 AM

I saw in the release notes for 0.14.15 that if I change

date

in the context, dependant context variables will now align accordingly. Does this apply to

CronSchedule()

as well? Currently I’m using

CronSchedule("00 02,12 * * *", start_date=pendulum.now("Europe/Berlin"))

- could I replace this by overriding date in the context? What is actually the best way to override date - in the configuration.toml as

date=pendulum.today("Europe/Berlin")

Michael Wheeler

04/06/2021, 2:11 PM

Hi everyone! I posted a question about the relationship between Prefect's agents and executors on Stack Overflow. I attempted to answer it myself but would really appreciate it if I could get somebody else's eyes on it. https://stackoverflow.com/questions/66959695/prefect-relationship-between-agent-and-executor

🙌 3

Nathan Walker

04/06/2021, 4:44 PM

I'm trying to dynamically update a bunch of Flow schedules using API calls. First, is there documentation for the API somewhere? I see it mentioned in the Server docs on occasion, but I want the nitty-gritty -- I have the schema and that's super helpful, but I'd like more. Second, I see that I can set_flow_group_schedule, but that doesn't include Filters. Is there a way to add Filters that I'm just missing? Third, if I want to extend API capabilities and write my own queries/mutations (i.e., adding to the schema), where would I begin looking in to this?

Florian Kühnlenz

04/07/2021, 8:03 AM

Hi, is there a way to setup email cloud-hooks on a local instance of the prefect server? Or is this only available in the cloud version?

Nathan Walker

04/07/2021, 2:09 PM

Hey folks, I'm running the following mutation in the API and getting back an error message saying it expected parameter_defaults to be JSON type, but found {my_param: "my_param_value"} instead-- ObjectValueNode object has no attribute 'value'. Formatting it like JSON doesn't fix the problem. From what I've read, JSON isn't a built-in scalar type, so I'm thinking I need to declare something or wrap the parameter_defaults in something, but I'm not sure exactly what. Can somebody point me in the right direction?

Copy code

mutation {
  set_flow_group_schedule(input:{
    flow_group_id: "f67a1118-41e6-4e3b-9f14-9203e8f5ed55",
    interval_clocks: [
      {interval: 60 
       parameter_defaults: {my_param: "my_param_value"}
      }
    ],
    timezone: "America/New_York"
    }
  ) {success}
}

xyzy

04/07/2021, 3:50 PM

Does the version of server need to be above or equal to the agent version? I know agent must be equal or above core version, but didn't find anything about server

xyzy

04/07/2021, 3:54 PM

In the DevOps Guide video on the Prefect Youtube Channel (

https://www.youtube.com/watch?v=EwsMecjSYEU▾

) a webpage showing DevOps guidelines is shown. Is this page publicly available? If not, will it be in the future?

Hygor Knust

04/07/2021, 6:51 PM

Hi everyone, I'm using the Prefect Helm Chart to deploy a Prefect Server in my self-hosted Kubernetes Cluster. The setup works fine, the tenant and agent are being set by Helm. I have also created a L7 Ingress pointing to Apollo. The UI is working, but when I try to register a simple Flow, I receive this error:

requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://{endpoint}:80/graphql/alpha

where endpoint is the L7 ingress endpoint. Is there a way to debug what's going wrong?

✅ 1

Christoph Wiese

04/09/2021, 1:51 PM

I’m using

database = prefect.context.get("database", "JOYN_SNOW_BI_1841")

in one of my flows, but if I set the database key via the Cloud interface like the below, it is not being used in the flow run - anybody got an idea what I’m doing wrong? Do I maybe need to transfer the context.get into a task?

Nate Jahncke

04/09/2021, 10:18 PM

Hi everyone, I'm new to Prefect and assume it will be a great product to use once I actually have it working. Thank you for all the effort you've put into it, and for making it free to evaluate effectively. I'm attempting to integrate Prefect into an existing docker-compose development environment. Running

prefect server start

yields a functional deployment with a working dashboard, but calling

docker-compose up

on the

docker-compose.yml

produced by executing

prefect server config

does not (the UI is responsive, but the dashboard pane below the top nav bar is blank). Also, any attempt to start a local agent in an adjacent docker container gives me a "No tenant found" message via a ClientError exception. Is the

prefect server config

command not actually producing everything it should be at present? I'm sure I can manually build a functional docker-compose.yml by reviewing the prefect docker-compose wrapper, but that might be very time-consuming, so I'm hoping the problem is just something simple that

prefect server config

is leaving out. I'm going to be evaluating the two deployments back to back now to see if I can tell what's missing. Any ideas would be greatly appreciated. Thanks again!

👍 1

Michael Wheeler

04/12/2021, 4:52 PM

I'm using Prefect's Docker image

prefecthq/prefect:latest

to spin up a container that calls

prefect server create-tenant

, and getting a connection error because it's pointed at localhost:4200 by default. It doesn't look like there's an option to provide another url as a CLI argument. Is there a configuration file or environment variable I can set to correct this?

Michael Wheeler

04/12/2021, 5:01 PM

Also added another Stack Overflow question, this time without an answer from me: https://stackoverflow.com/questions/67062496/prefect-server-what-is-the-role-of-a-tenant

👍 1

Nathan Walker

04/12/2021, 6:06 PM

Hey folks, I'd like to be able to push a Flow Group's start date back arbitrarily to act as a hacky "Delay Run" method. I'm seeing that the mutation set_flow_group_schedule should do the trick, but its input parameters only give "timezone" as an option. This makes sense since you can pass in multiple clocks, but both the cron_clock_input and interval_clock_input lack start_date as well in the API. It looks like when you use the API to create/update a Flow Group's schedule, it just plops datetime.now() in as the start date, but I'd like to be able to define it per clock. Can somebody point me in the right direction on adding functionality like this to Apollo/Hasura?

Felipe Saldana

04/12/2021, 11:36 PM

I see there is a flag to set a parameters file when running a flow. I dont see this flag when registering a flow. Is there a way for me to set a parameters file for a flow so that once registered any run could default to the values from the file?

Copy code

Run a flow that is registered to the Prefect API

Options:
    --id, -i                    TEXT        The ID of a flow to run
    --version-group-id          TEXT        The ID of a flow version group to run
    --name, -n                  TEXT        The name of a flow to run
    --project, -p               TEXT        The name of a project that contains the flow
    --version, -v               INTEGER     A flow version to run
    --parameters-file, -pf      FILE PATH   A filepath of a JSON file containing
                                            parameters
    --parameters-string, -ps    TEXT        A string of JSON parameters (note: to ensure these are
                                            parsed correctly, it is best to include the full payload
                                            within single quotes)

Nate Jahncke

04/13/2021, 12:49 AM

Hey everyone, quick question regarding flow storage metadata: I'm running a docker-compose Prefect stack with an agent container and due to the

storage.directory

metadata that is automatically added to flows that are configured as mine are, I'm forced to register the flow from within the agent container, rather than just running it in vscode (which tacks on my own home dir as the

storage.directory

and of course yields an exception on flow execution). This certainly works, and is not a big deal at all, but I'm wondering if I can force the metadata somehow? Passing a directory to the Local() object of course won't allow me to use a path that only exists in the agent container, as it's for local storage, and while I could probably fake it and use a dir common to both, I would really prefer not to do that even if it works (as that directory in my local system would get filled up with files I don't want to exist outside of the container). Maybe what I have in mind just doesn't work? Some of this is definitely down to my two-day-old understanding of Prefect's architecture, but the real goal is just keeping absolutely everything related to Prefect in the docker environment except for flow development. If possible. As I said, not crucial as I have a totally working solution now and am eager to build some flows, but thanks in advance for any advice!

Aurélien Vallée

04/14/2021, 6:09 AM

Hello everyone, I am having a somewhat hard time trying to configure a self hosted server for prefect. Most of my struggle is based on what is the proper way of doing things. I started by using

prefect server start

but ran with a bunch of issues: • Most container services are listening on 0.0.0.0, which is not desirable in my case. I have a specific IP on which I would like exposed service to listen. • I am confused as to which services actually need to be exposed beyond the loopback. My understanding is that the UI (of course) needs to be, as well as the GraphQL endpoint (since the UI JS seems to communicate with it). What about the others? I have to admit, I am foreign to hasura/apollo/etc. Since I wanted to customize that, I figured I should somehow generate the

docker-compose.yml

with

prefect server config

and tweak that accordingly, which I did. That did not seem to be a good solution though, since even after tweaking the published ports and trial and error with the various configurations in

docker-compose.yml

, the UI is still trying to reach

<http://localhost:4200>

, even though I tried to tweak the config to have it use

http://<server IP>:4200/

so I guess everything is not configurable through editing the

docker-compose.yml

. I spent quite some bunch of time reading various threads on internet, and it seems to me now the proper way of configuring the server is not to tweak the

docker-compose.yml

but rather to tweak

config.toml

, and

prefect server start

will use that when generating its temporary

docker-compose.yml

. So I did that, but I'm still confused as to what exactly should be exposed and what should not. I also get lost in the options of

prefect server start

, it allows me to no publish container listened ports to the host, but of course them of them should be for the UI/agent/graphql to be available. Not sure I can even articulate the questions I have precisly, I'm just lost in the proper way to configure the server and would need some guidance.

Aurélien Vallée

04/14/2021, 6:14 AM

Ideally I would like to: • Have all ports that have no reason to be exposed otherwise limited to listening on the loopback • Have all ports that should be externally accessible published on the host (UI and GraphQL from my understanding) • For all externally accessible ports, have control over which IP these ports are listening

Aurélien Vallée

04/14/2021, 6:15 AM

What I did until now: • Tweak

config.toml

to set

host=

as my desired public IP for externally accessible services (:4200 and :8080) • Tweak

config.toml

to set

host=127.0.0.1

on ports that have no reason for a local deployment to be exposed (:4201 and :5432)

Aurélien Vallée

04/14/2021, 6:16 AM

• Use

--no-hasura-port

and

--no-postgresql-port

on the command line

Minakshi

04/16/2021, 12:01 AM

Hi team, I am trying to get prefect server to be installed on our dev boxes. It works for me locally. But our boxes don't support downloading the docker images from external websites. Where can i find the images for the services that prefect uses (postgres, hasura, graphql,apollo, towel) which i be able to upload to our internal registry? Also is there any other suggestion for doing this type of deployment?

Minakshi

04/17/2021, 1:48 AM

Whats the process to start the prefect server using a mounted volume? are there ore details on how to use the --use-volume command? How to setup a sample postgres mounted volume etc?

Joe McDonald

04/19/2021, 7:07 PM

In the ECS agent, is there a way to pass botocore_config at command line? I don’t see any option to pass that along… if not I’m writing a way to read that from a config file like run_task_kwargs because the way the settings for retry is written it overrides any other setting method… https://github.com/PrefectHQ/prefect/blob/bbb6c49bb3bfefd88e420a1b9e50565278dbbc62/src/prefect/agent/ecs/agent.py#L126

Kevin Kho

04/19/2021, 7:14 PM

Hi @Joe McDonald! Unfortunately, it seems like you’re right that there is no way to pass botocore_config yet through the CLI.

Minakshi

04/20/2021, 4:07 AM

Hi team, are prefect agents deployed as part of prefect server itself? Would it be okay to run the agent on the same box as prefect server? Are there any advantages of running the agent on a separate box?

Vladislav Bogucharov

04/21/2021, 4:14 PM

Hello everyone! I am trying to install prefect according to the tutorial, I run the following commands:

Copy code

python3 -m venv prefect_venv
source prefect_venv/bin/activate
pip install wheel
pip install prefect
prefect backend server
prefect server start

And then I get the error: subprocess.CalledProcessError: Command '[' docker-compose ',' down ']' returned non-zero exit status 255. When I run prefect diagnostics
everything seems to be fine:

Copy code

{
  "config_overrides": {},
  "env_vars": [],
  "system_information": {
    "platform": "Linux-4.15.0-142-generic-x86_64-with-Ubuntu-18.04-bionic",
    "prefect_backend": "server",
    "prefect_version": "0.14.16",
    "python_version": "3.6.9"
  }
}

Has anyone faced this problem?

Garret Cook

04/22/2021, 2:04 AM

Is there a way to get a dockerstorage based flow to listen on a port? Similar to publishing a port when a container is run? I’ve got a device that will only backup to an ftp server. I’m wanting my flow to listen for connections on a public port. Is there a different design pattern recommended?

Ivan Ksaver Šušnjara

04/22/2021, 1:37 PM

Hello. I'm having a problem with LocalDaskExecutor employing "processes". When calling flow.run() method, following error appears:

Copy code

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

What is more intriguing, some flows continue and complete nonetheless, while other require manual KeyboardInterrupt (ctrl+c), and then they start executing their respctive tasks. Dask provides solution for the issue (https://github.com/dask/distributed/issues/2520), however I don't see a workaround for this in Prefect. Problem does not appear when "threads" are employed instead of "processes" with LocalDaskExecutor's scheduler argument. I'm using Prefect version 0.14.16 with Python 3.8.7. Has anyone faced the same issue? Thanks in advance!

Joe McDonald

04/22/2021, 3:47 PM

So in HA setup in AWS ECS we have 3 of each of the services for server: hasura, apollo, graphql, ui, and towel. We have in another cluster three agents running on ECS. I’ve seen a log where two agents submitted execution of the same flow run. The first two lines of the log, with the same timestamp to the second, was the “Submitted for execution: Task arnawsecs:…” from two different agents. Is there any type of locking occurring that should prevent this? Where in the code from server is this handled? I’m willing to look into it, but be nice to have a starting point of which service is responsible for handing jobs to agents and would allow for adding a check.

Enda Peng

04/22/2021, 3:49 PM

Is there a way to overwrite the local server port? After running

Copy code

prefect server start

It automatically binds to 8080. I try to find .compose file but don’t get any hint. Thank you!