prefect-community #prefect-community

Channels

prefect-community

Fina Silva-Santisteban

02/05/2021, 12:15 AM

Hi prefect community! We’ve created a flow which creates and saves an excel workbook. We’re trying to upload that workbook into our S3 bucket and we’ve provided the necessary credentials. The problem is that the bucket fills up with

*.prefect_result

files instead of a workbook. How can we fix that? This is the code snippet:

flow.result = S3Result(bucket=bucket)

David Clements

02/05/2021, 12:39 AM

💃 Just arrived! Looking forward to doing my spike on prefect. it is a bake off with dagster!!

Brett Naul

02/05/2021, 12:56 AM

anyone else noticed the following UI behavior? if you open two links to flow runs in new tabs, the initial URLs are correct (https://cloud.prefect.io/flow-run/abc and https://cloud.prefect.io/flow-run/def), but then both will redirect to the same page (for us https://cloud.prefect.io/replica/flow-run/abc). thought I was going crazy for a while...

Marwan Sarieddine

02/05/2021, 1:20 AM

Hi Folks, we ran a flow that triggered 11 downstream flows on prefect cloud using

StartFlowRun

, the 11 flows got “queued”, but since we are on team plan (i.e. limited by flow concurrency) - only 2 of the flows were scheduled and the rest were late runs - so far so good, then the late runs get scheduled and run, but for some unkown reason 5 of the 11 late runs disappeared on their own and I don’t see why that happened

Gabriel Santos

02/05/2021, 6:29 AM

Hello everyone! Does anyone use the prefect on heroku?

Laura Vaida

02/05/2021, 12:56 PM

hi folks, i want to connect prefect with my kubernetes engine on gcp. i used the following config for the dask scheduler:

worker:
  replicas: 3
  resources:
    limits:
      cpu: 1
      memory: 3G
    requests:
      cpu: 1
      memory: 3G
  env:
    - name: EXTRA_CONDA_PACKAGES
      value: prefect
jupyter:
  enabled: false

Laura Vaida

02/05/2021, 12:58 PM

but somehow this doesnt really work, so there is no load balancer in gcp, does anybody know why?

Michael Hadorn

02/05/2021, 1:32 PM

Hi there Sadly i didn't understand really the docker-run and gitlab-storage concept. 😞 I built an image on top of the prefect image (prefecthq/prefect:0.14.6-python3.8) and installed my python deps successfully. In my flow, I added to parts:

with Flow(
    prefect.config.general.flow_name
    , executor=LocalDaskExecutor()
    , storage=GitLab(
        host="<https://gitlab.xxx.ch/>"
        , repo="cdwh/cdwh-prefect"
        , path="cdwh_flow.py"
        , ref="master"
        , secrets=["GITLAB_ACCESS_TOKEN"]
    )
    , run_config=DockerRun(
        image="mrjck/cdwh-flow:1.0"
    )
    ,...

Then I successfully registered the flow to the backend. If I let them run, the prefect docker agent deploying the flow but crashed immediately with the following message:

404 Client Error for <http+docker://localhost/v1.40/images/create?tag=1.0&fromImage=mrjck%2Fcdwh-flow>: Not Found ("manifest for mrjck/cdwh-flow:1.0 not found: manifest unknown: manifest unknown")

If I run 'docker images' the used images is there with the correct tag. How does my Image know, how the storage is pulled? UPDATE: Was solved the next day. If not, looks like this: https://github.com/PrefectHQ/prefect/issues/2781 Try to start the agent with: --no-docker-interface

Milly gupta

02/05/2021, 2:04 PM

Hey all, can we create subtasks with in a task?

✅ 1

Jeremy Phelps

02/05/2021, 6:39 PM

Hello, Prefect community, I was wondering if there's a way to tell exactly where a log message, as seen in the Prefect Server's web console, actually came from. There seem to be at least three places any given log message can come from, and I have to log into different machines to hunt for their origin in each machine's syslog: • The Kubernetes pod created by the Kubernetes Agent. • The Dask scheduler machine. • A Dask worker machine. Knowing where the log message comes from is necessary to determine which part of the infrastructure is misconfigured. Thanks in advance.

Neeraj Vyas

02/05/2021, 6:41 PM

Hi all, can someone can share the steps to store flows in my docker-image and use it using docker-hub with kubernetes agent? It will surely help

jeff n

02/05/2021, 8:37 PM

Howdy, We are trying to streamline configurations in our tasks but are unable to use the Prefect Secrets due to security. Our rough draft used

os.getenv()

inside the tasks so that we pulled the secrets in at runtime but this is becoming incredibly unruly. I want to use a config library like dynaconfig so that I can mix settings files with env files. What is the best practice to load up the variables at run time in a smart way? Would the loading be a task that then pushes the config into the Prefect context?

Giovanni Giacco

02/05/2021, 9:42 PM

I need a tip from the community. In our new use cases we need to create flows at runtime, when a REST call arrives. In those cases where we can store the flows? What's the best solution? Maybe S3 storage?

Verun Rahimtoola

02/05/2021, 11:00 PM

hi, for flow params, is there a way to specify the type? kinda like how in

argparse.add_argument(...)

you can pass in something like

type=int

or some callable

james.lamb

02/05/2021, 11:17 PM

👋 hello from Chicago! Is there a status page for Prefect Cloud, where I could look for Prefect-Cloud-wide incidents?

Fina Silva-Santisteban

02/05/2021, 11:42 PM

Hi Prefect community! We’re currently running the prefect server, prefect docker agent with a local docker image and I’m using Docker storage which points at my local Dockerfile. That works fine, but switching the backend from server to cloud somehow doesn’t. We have Prefect Cloud seats and I’ve successfully authenticated my agent, and I’ve successfully registered my flows. When I try to run a flow using the Prefect Cloud UI, my prefect docker agent console says

Deploying flow run...

but the flow gets stuck in the

Submitted

stage. What am I missing?

Matt Denno

02/06/2021, 11:28 PM

Hello - I am having an issue getting my flows to run from the server after registering them. I can run them by calling

flow.run()

but if I

flow.register(name="Task Name")

and then try to run via the UI it fails. The message I get is:

Failed to load and execute Flow's environment: ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.')

I feel like I have done exactly this setup before with Prefect and it has worked fine, but now it is not working. I see it is related to django not being setup, but can't understand why it isn't...I guess because I don't complete understand how Prefect works. A simple example that demonstrates the issue is as follows:

import django
import os
import prefect

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'afs.settings')
django.setup()

from prefect import Flow, task
from api.models import Location # This is a django model import


@task()
def django_task():
    locs = Location.objects.all()
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(f"Number of Locations: {len(locs)}")

with Flow(f"Django Task") as flow:
    locs = django_task()


# flow.run()
flow.register(project_name="Kerala")

Searching the Slack channel for Django returned one result which indicated the maybe using a resource manager was the way to set this up, but I wasn't able to get it to work using resource manager either. The other thing I tried was to set DJANGO_SETTINGS_MODULE via a LocalRun:

flow.run_config = LocalRun(env={'DJANGO_SETTINGS_MODULE': 'afs.settings'})
flow.register(project_name="Kerala")

But that results in another error about INSTALLED_APPS not being configured...clearly I am missing something. Last thing, I also added the following to a a separate non django task, and it looks like it is using the correct virtual env to execute the task and has the correct paths.

<http://logger.info|logger.info>(sys.path)
<http://logger.info|logger.info>(sys.version)
<http://logger.info|logger.info>(sys.executable)

Really stumped... Any help would be greatly appreciated.

Dimosthenis Schizas

02/07/2021, 12:19 AM

Hi everyone, sorry if my question is really dumb, but I couldn't find anything in the documentation. How can I register a flow with a date parameter, that will run on a schedule and will have by default the date of running. When i create the parameter like this:

auction_date = Parameter(
        "auction_date", default=prefect.context.date.date()
    )

I get the following error message when trying to register:

AttributeError: 'Context' object has no attribute 'date'

I am missing something here

Nathan Atkins

02/07/2021, 12:36 AM

Glad to be back doing some work with Prefect. I can see a lot has changed since I was last working with it. I'm having some problems with

prefect

commands against my local server. I guess it could be a installation or configuration problem, but debugging makes it look like there is an issue with initializing Client objects to talk to the local server. Specifically:

prefect create project my_project

and

flow.register(project_name='tutorial')

Both generate

prefect.utilities.exceptions.ClientError: Malformed response received from API.

errors. If I debug both problems I wind up in

prefect.client.Client.___init___()

where it looks like it sets the api_server to the cloud host because no

api_server

is passed in. I have the backend set to to

server

. In

prefect.cli.create.project()

Client().create_project(project_name=name, project_description=description)

it looks like the

api_server=localhost

needs to be passed to

Client.___init___()

or Client needs to do something other than

self.api_server = api_server or prefect.context.config.cloud.get("graphql")

Same issue in

prefect.core.flow.py

I can connect to the server fine from my local browser or through cloud.prefect.io switched to the local server. (Nice feature by the way.)

Dolor Oculus

02/07/2021, 1:42 AM

This is a 🍒 on 🔝 thing, but is there any way to make a flow with conditions look better? I have a flow with some nested case() statements (on the order of, "if this file exists, and this other file exists, then do this..."). Was hoping for , got

Rob Fowler

02/08/2021, 11:24 AM

odd, I have had a prefect flow running for months, not I am getting: Unexpected error occured in FlowRunner: ValueError('Could not infer an active Flow context.')

Rob Fowler

02/08/2021, 12:02 PM

seems

executor = LocalDaskExecutor(scheduler="processes", num_workers=10)
    for ii in flow.run(executor=executor).result[result].result:
        print(ii)

Rob Fowler

02/08/2021, 12:02 PM

does not work anymore

James Phoenix

02/08/2021, 12:52 PM

Hey everyone.

James Phoenix

02/08/2021, 12:52 PM

I would like to execute flows from inside of a flask route.

James Phoenix

02/08/2021, 12:53 PM

Is it possible to run flows this way?

James Phoenix

02/08/2021, 12:53 PM

import prefect
client = prefect.Client()

James Phoenix

02/08/2021, 1:22 PM

Is it this?

James Phoenix

02/08/2021, 1:22 PM

client.create_flow_run(flow_id="<flow id>")

James Phoenix

02/08/2021, 1:22 PM

https://docs.prefect.io/orchestration/concepts/flow_runs.html#ui