prefect-community #prefect-community

Gabriel Santos

02/05/2021, 6:29 AM

Hello everyone! Does anyone use the prefect on heroku?

Laura Vaida

02/05/2021, 12:56 PM

hi folks, i want to connect prefect with my kubernetes engine on gcp. i used the following config for the dask scheduler:

Copy code

worker:
  replicas: 3
  resources:
    limits:
      cpu: 1
      memory: 3G
    requests:
      cpu: 1
      memory: 3G
  env:
    - name: EXTRA_CONDA_PACKAGES
      value: prefect
jupyter:
  enabled: false

Laura Vaida

02/05/2021, 12:58 PM

but somehow this doesnt really work, so there is no load balancer in gcp, does anybody know why?

Michael Hadorn

02/05/2021, 1:32 PM

Hi there Sadly i didn't understand really the docker-run and gitlab-storage concept. 😞 I built an image on top of the prefect image (prefecthq/prefect:0.14.6-python3.8) and installed my python deps successfully. In my flow, I added to parts:

Copy code

with Flow(
    prefect.config.general.flow_name
    , executor=LocalDaskExecutor()
    , storage=GitLab(
        host="<https://gitlab.xxx.ch/>"
        , repo="cdwh/cdwh-prefect"
        , path="cdwh_flow.py"
        , ref="master"
        , secrets=["GITLAB_ACCESS_TOKEN"]
    )
    , run_config=DockerRun(
        image="mrjck/cdwh-flow:1.0"
    )
    ,...

Then I successfully registered the flow to the backend. If I let them run, the prefect docker agent deploying the flow but crashed immediately with the following message:

404 Client Error for <http+docker://localhost/v1.40/images/create?tag=1.0&fromImage=mrjck%2Fcdwh-flow>: Not Found ("manifest for mrjck/cdwh-flow:1.0 not found: manifest unknown: manifest unknown")

If I run 'docker images' the used images is there with the correct tag. How does my Image know, how the storage is pulled? UPDATE: Was solved the next day. If not, looks like this: https://github.com/PrefectHQ/prefect/issues/2781 Try to start the agent with: --no-docker-interface

Milly gupta

02/05/2021, 2:04 PM

Hey all, can we create subtasks with in a task?

✅ 1

Jeremy Phelps

02/05/2021, 6:39 PM

Hello, Prefect community, I was wondering if there's a way to tell exactly where a log message, as seen in the Prefect Server's web console, actually came from. There seem to be at least three places any given log message can come from, and I have to log into different machines to hunt for their origin in each machine's syslog: • The Kubernetes pod created by the Kubernetes Agent. • The Dask scheduler machine. • A Dask worker machine. Knowing where the log message comes from is necessary to determine which part of the infrastructure is misconfigured. Thanks in advance.

Neeraj Vyas

02/05/2021, 6:41 PM

Hi all, can someone can share the steps to store flows in my docker-image and use it using docker-hub with kubernetes agent? It will surely help

jeff n

02/05/2021, 8:37 PM

Howdy, We are trying to streamline configurations in our tasks but are unable to use the Prefect Secrets due to security. Our rough draft used

os.getenv()

inside the tasks so that we pulled the secrets in at runtime but this is becoming incredibly unruly. I want to use a config library like dynaconfig so that I can mix settings files with env files. What is the best practice to load up the variables at run time in a smart way? Would the loading be a task that then pushes the config into the Prefect context?

Giovanni Giacco

02/05/2021, 9:42 PM

I need a tip from the community. In our new use cases we need to create flows at runtime, when a REST call arrives. In those cases where we can store the flows? What's the best solution? Maybe S3 storage?

Verun Rahimtoola

02/05/2021, 11:00 PM

hi, for flow params, is there a way to specify the type? kinda like how in

argparse.add_argument(...)

you can pass in something like

type=int

or some callable

james.lamb

02/05/2021, 11:17 PM

👋 hello from Chicago! Is there a status page for Prefect Cloud, where I could look for Prefect-Cloud-wide incidents?

Fina Silva-Santisteban

02/05/2021, 11:42 PM

Hi Prefect community! We’re currently running the prefect server, prefect docker agent with a local docker image and I’m using Docker storage which points at my local Dockerfile. That works fine, but switching the backend from server to cloud somehow doesn’t. We have Prefect Cloud seats and I’ve successfully authenticated my agent, and I’ve successfully registered my flows. When I try to run a flow using the Prefect Cloud UI, my prefect docker agent console says

Deploying flow run...

but the flow gets stuck in the

Submitted

stage. What am I missing?

Matt Denno

02/06/2021, 11:28 PM

Hello - I am having an issue getting my flows to run from the server after registering them. I can run them by calling

flow.run()

but if I

flow.register(name="Task Name")

and then try to run via the UI it fails. The message I get is:

Copy code

Failed to load and execute Flow's environment: ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.')

I feel like I have done exactly this setup before with Prefect and it has worked fine, but now it is not working. I see it is related to django not being setup, but can't understand why it isn't...I guess because I don't complete understand how Prefect works. A simple example that demonstrates the issue is as follows:

Copy code

import django
import os
import prefect

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'afs.settings')
django.setup()

from prefect import Flow, task
from api.models import Location # This is a django model import


@task()
def django_task():
    locs = Location.objects.all()
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(f"Number of Locations: {len(locs)}")

with Flow(f"Django Task") as flow:
    locs = django_task()


# flow.run()
flow.register(project_name="Kerala")

Searching the Slack channel for Django returned one result which indicated the maybe using a resource manager was the way to set this up, but I wasn't able to get it to work using resource manager either. The other thing I tried was to set DJANGO_SETTINGS_MODULE via a LocalRun:

Copy code

flow.run_config = LocalRun(env={'DJANGO_SETTINGS_MODULE': 'afs.settings'})
flow.register(project_name="Kerala")

But that results in another error about INSTALLED_APPS not being configured...clearly I am missing something. Last thing, I also added the following to a a separate non django task, and it looks like it is using the correct virtual env to execute the task and has the correct paths.

Copy code

<http://logger.info|logger.info>(sys.path)
<http://logger.info|logger.info>(sys.version)
<http://logger.info|logger.info>(sys.executable)

Really stumped... Any help would be greatly appreciated.

Dimosthenis Schizas

02/07/2021, 12:19 AM

Hi everyone, sorry if my question is really dumb, but I couldn't find anything in the documentation. How can I register a flow with a date parameter, that will run on a schedule and will have by default the date of running. When i create the parameter like this:

Copy code

auction_date = Parameter(
        "auction_date", default=prefect.context.date.date()
    )

I get the following error message when trying to register:

Copy code

AttributeError: 'Context' object has no attribute 'date'

I am missing something here

Nathan Atkins

02/07/2021, 12:36 AM

Glad to be back doing some work with Prefect. I can see a lot has changed since I was last working with it. I'm having some problems with

prefect

commands against my local server. I guess it could be a installation or configuration problem, but debugging makes it look like there is an issue with initializing Client objects to talk to the local server. Specifically:

prefect create project my_project

and

flow.register(project_name='tutorial')

Both generate

prefect.utilities.exceptions.ClientError: Malformed response received from API.

errors. If I debug both problems I wind up in

prefect.client.Client.___init___()

where it looks like it sets the api_server to the cloud host because no

api_server

is passed in. I have the backend set to to

server

. In

prefect.cli.create.project()

Copy code

Client().create_project(project_name=name, project_description=description)

it looks like the

api_server=localhost

needs to be passed to

Client.___init___()

or Client needs to do something other than

Copy code

self.api_server = api_server or prefect.context.config.cloud.get("graphql")

Same issue in

prefect.core.flow.py

I can connect to the server fine from my local browser or through cloud.prefect.io switched to the local server. (Nice feature by the way.)

Dolor Oculus

02/07/2021, 1:42 AM

This is a 🍒 on 🔝 thing, but is there any way to make a flow with conditions look better? I have a flow with some nested case() statements (on the order of, "if this file exists, and this other file exists, then do this..."). Was hoping for , got

Rob Fowler

02/08/2021, 11:24 AM

odd, I have had a prefect flow running for months, not I am getting: Unexpected error occured in FlowRunner: ValueError('Could not infer an active Flow context.')

Rob Fowler

02/08/2021, 12:02 PM

seems

Copy code

executor = LocalDaskExecutor(scheduler="processes", num_workers=10)
    for ii in flow.run(executor=executor).result[result].result:
        print(ii)

Rob Fowler

02/08/2021, 12:02 PM

does not work anymore

James Phoenix

02/08/2021, 12:52 PM

Hey everyone.

James Phoenix

02/08/2021, 12:52 PM

I would like to execute flows from inside of a flask route.

James Phoenix

02/08/2021, 12:53 PM

Is it possible to run flows this way?

James Phoenix

02/08/2021, 12:53 PM

Copy code

import prefect
client = prefect.Client()

James Phoenix

02/08/2021, 1:22 PM

Is it this?

James Phoenix

02/08/2021, 1:22 PM

Copy code

client.create_flow_run(flow_id="<flow id>")

James Phoenix

02/08/2021, 1:22 PM

https://docs.prefect.io/orchestration/concepts/flow_runs.html#ui

rafaqat ali

02/08/2021, 1:29 PM

Hi, I'm new here so don't have much idea about perfect. Can anyone let me know how can I create approval workflow using perfect?

Avinash Varma Kanumuri

02/08/2021, 2:32 PM

Hi, I’m new to prefect. Does prefect have subdag like @composite_solid in dagster, if yes can you please let me know the prefect term for it. (task calling other tasks)

Samuel Hinton

02/08/2021, 4:37 PM

Hi team! I have a few files which contain some different flows. There are some common functions that I want them all to share, all in a common.py file in the same directory. I could turn this into a github package and pip installable by itself, but is there a simple way of simply flagging to the storage that I also want common.py uploaded into my S3 bucket alongside the flow file?

Pedro Martins

02/08/2021, 6:06 PM

Hey team! I have a flow in which the parameters default value comes from a

config.toml

. See below:

Copy code

with Flow("model-deployment-pipeline", **custom_confs,) as flow:
        model_uri = Parameter("model_uri", default=config["model"]["modelUri"])
        environment = Parameter("environment", default=config["model"]["environment"])

        deploy_model(model_uri=model_uri, namespace=environment)

This flow is triggered when there are modification on the config.toml. How can I make sure that this flow will access the most up to date config file?