james.lamb
04/05/2021, 7:40 PMThe Lazarus process is meant to gracefully retry failures caused by factors outside of Prefect's control. The most common situations requiring Lazarus intervention are infrastructure issues, such as Kubernetes pods not spinning up or being deleted before they're able to complete a run.
Once every 10 minutes, the Lazarus process searches for distressed flow runs. Each flow run found in this manner is rescheduled; this intervention by Lazarus is reflected in the flow run's logs.Where can I find more specifics on how a flow could enter this "distressed" state? For example, if I have a KubernetesAgent up, Prefect Cloud triggers a flow run, and then the agent isn't able to start up a k8s job because I misconfigured its RBAC stuff, would that be something Lazarus retries? Will give more context in thread.
Christoph Wiese
04/06/2021, 9:51 AMdate
in the context, dependant context variables will now align accordingly. Does this apply to CronSchedule()
as well?
Currently I’m using CronSchedule("00 02,12 * * *", start_date=pendulum.now("Europe/Berlin"))
- could I replace this by overriding date in the context? What is actually the best way to override date - in the configuration.toml as date=pendulum.today("Europe/Berlin")
?Michael Wheeler
04/06/2021, 2:11 PMNathan Walker
04/06/2021, 4:44 PMFlorian Kühnlenz
04/07/2021, 8:03 AMNathan Walker
04/07/2021, 2:09 PMmutation {
set_flow_group_schedule(input:{
flow_group_id: "f67a1118-41e6-4e3b-9f14-9203e8f5ed55",
interval_clocks: [
{interval: 60
parameter_defaults: {my_param: "my_param_value"}
}
],
timezone: "America/New_York"
}
) {success}
}
xyzy
04/07/2021, 3:50 PMHygor Knust
04/07/2021, 6:51 PMrequests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://{endpoint}:80/graphql/alpha
where endpoint is the L7 ingress endpoint.
Is there a way to debug what's going wrong?Christoph Wiese
04/09/2021, 1:51 PMdatabase = prefect.context.get("database", "JOYN_SNOW_BI_1841")
in one of my flows, but if I set the database key via the Cloud interface like the below, it is not being used in the flow run - anybody got an idea what I’m doing wrong? Do I maybe need to transfer the context.get into a task?Nate Jahncke
04/09/2021, 10:18 PMprefect server start
yields a functional deployment with a working dashboard, but calling docker-compose up
on the docker-compose.yml
produced by executing prefect server config
does not (the UI is responsive, but the dashboard pane below the top nav bar is blank). Also, any attempt to start a local agent in an adjacent docker container gives me a "No tenant found" message via a ClientError exception.
Is the prefect server config
command not actually producing everything it should be at present? I'm sure I can manually build a functional docker-compose.yml by reviewing the prefect docker-compose wrapper, but that might be very time-consuming, so I'm hoping the problem is just something simple that prefect server config
is leaving out. I'm going to be evaluating the two deployments back to back now to see if I can tell what's missing. Any ideas would be greatly appreciated.
Thanks again!Michael Wheeler
04/12/2021, 4:52 PMprefecthq/prefect:latest
to spin up a container that calls prefect server create-tenant
, and getting a connection error because it's pointed at localhost:4200 by default. It doesn't look like there's an option to provide another url as a CLI argument. Is there a configuration file or environment variable I can set to correct this?Michael Wheeler
04/12/2021, 5:01 PMNathan Walker
04/12/2021, 6:06 PMFelipe Saldana
04/12/2021, 11:36 PMRun a flow that is registered to the Prefect API
Options:
--id, -i TEXT The ID of a flow to run
--version-group-id TEXT The ID of a flow version group to run
--name, -n TEXT The name of a flow to run
--project, -p TEXT The name of a project that contains the flow
--version, -v INTEGER A flow version to run
--parameters-file, -pf FILE PATH A filepath of a JSON file containing
parameters
--parameters-string, -ps TEXT A string of JSON parameters (note: to ensure these are
parsed correctly, it is best to include the full payload
within single quotes)
Nate Jahncke
04/13/2021, 12:49 AMstorage.directory
metadata that is automatically added to flows that are configured as mine are, I'm forced to register the flow from within the agent container, rather than just running it in vscode (which tacks on my own home dir as the storage.directory
and of course yields an exception on flow execution).
This certainly works, and is not a big deal at all, but I'm wondering if I can force the metadata somehow? Passing a directory to the Local() object of course won't allow me to use a path that only exists in the agent container, as it's for local storage, and while I could probably fake it and use a dir common to both, I would really prefer not to do that even if it works (as that directory in my local system would get filled up with files I don't want to exist outside of the container).
Maybe what I have in mind just doesn't work? Some of this is definitely down to my two-day-old understanding of Prefect's architecture, but the real goal is just keeping absolutely everything related to Prefect in the docker environment except for flow development. If possible.
As I said, not crucial as I have a totally working solution now and am eager to build some flows, but thanks in advance for any advice!Aurélien Vallée
04/14/2021, 6:09 AMprefect server start
but ran with a bunch of issues:
• Most container services are listening on 0.0.0.0, which is not desirable in my case. I have a specific IP on which I would like exposed service to listen.
• I am confused as to which services actually need to be exposed beyond the loopback. My understanding is that the UI (of course) needs to be, as well as the GraphQL endpoint (since the UI JS seems to communicate with it). What about the others? I have to admit, I am foreign to hasura/apollo/etc.
Since I wanted to customize that, I figured I should somehow generate the docker-compose.yml
with prefect server config
and tweak that accordingly, which I did.
That did not seem to be a good solution though, since even after tweaking the published ports and trial and error with the various configurations in docker-compose.yml
, the UI is still trying to reach <http://localhost:4200>
, even though I tried to tweak the config to have it use http://<server IP>:4200/
so I guess everything is not configurable through editing the docker-compose.yml
.
I spent quite some bunch of time reading various threads on internet, and it seems to me now the proper way of configuring the server is not to tweak the docker-compose.yml
but rather to tweak config.toml
, and prefect server start
will use that when generating its temporary docker-compose.yml
.
So I did that, but I'm still confused as to what exactly should be exposed and what should not.
I also get lost in the options of prefect server start
, it allows me to no publish container listened ports to the host, but of course them of them should be for the UI/agent/graphql to be available.
Not sure I can even articulate the questions I have precisly, I'm just lost in the proper way to configure the server and would need some guidance.Aurélien Vallée
04/14/2021, 6:14 AMAurélien Vallée
04/14/2021, 6:15 AMconfig.toml
to set host=
as my desired public IP for externally accessible services (:4200 and :8080)
• Tweak config.toml
to set host=127.0.0.1
on ports that have no reason for a local deployment to be exposed (:4201 and :5432)Aurélien Vallée
04/14/2021, 6:16 AM--no-hasura-port
and --no-postgresql-port
on the command lineMinakshi
04/16/2021, 12:01 AMMinakshi
04/17/2021, 1:48 AMJoe McDonald
04/19/2021, 7:07 PMKevin Kho
Minakshi
04/20/2021, 4:07 AMVladislav Bogucharov
04/21/2021, 4:14 PMpython3 -m venv prefect_venv
source prefect_venv/bin/activate
pip install wheel
pip install prefect
prefect backend server
prefect server start
And then I get the error:
subprocess.CalledProcessError: Command '[' docker-compose ',' down ']' returned non-zero exit status 255.
When I run prefect diagnostics
everything seems to be fine:
{
"config_overrides": {},
"env_vars": [],
"system_information": {
"platform": "Linux-4.15.0-142-generic-x86_64-with-Ubuntu-18.04-bionic",
"prefect_backend": "server",
"prefect_version": "0.14.16",
"python_version": "3.6.9"
}
}
Has anyone faced this problem?Garret Cook
04/22/2021, 2:04 AMIvan Ksaver Šušnjara
04/22/2021, 1:37 PMRuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
What is more intriguing, some flows continue and complete nonetheless, while other require manual KeyboardInterrupt (ctrl+c), and then they start executing their respctive tasks. Dask provides solution for the issue (https://github.com/dask/distributed/issues/2520), however I don't see a workaround for this in Prefect. Problem does not appear when "threads" are employed instead of "processes" with LocalDaskExecutor's scheduler argument.
I'm using Prefect version 0.14.16 with Python 3.8.7.
Has anyone faced the same issue? Thanks in advance!Joe McDonald
04/22/2021, 3:47 PMEnda Peng
04/22/2021, 3:49 PMprefect server start
It automatically binds to 8080. I try to find .compose file but don’t get any hint. Thank you!