https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • f

    Fina Silva-Santisteban

    02/05/2021, 12:15 AM
    Hi prefect community! We’ve created a flow which creates and saves an excel workbook. We’re trying to upload that workbook into our S3 bucket and we’ve provided the necessary credentials. The problem is that the bucket fills up with
    *.prefect_result
    files instead of a workbook. How can we fix that? This is the code snippet:
    flow.result = S3Result(bucket=bucket)
    m
    c
    m
    • 4
    • 7
  • d

    David Clements

    02/05/2021, 12:39 AM
    💃 Just arrived! Looking forward to doing my spike on prefect. it is a bake off with dagster!!
  • b

    Brett Naul

    02/05/2021, 12:56 AM
    anyone else noticed the following UI behavior? if you open two links to flow runs in new tabs, the initial URLs are correct (https://cloud.prefect.io/flow-run/abc and https://cloud.prefect.io/flow-run/def), but then both will redirect to the same page (for us https://cloud.prefect.io/replica/flow-run/abc). thought I was going crazy for a while...
    n
    m
    • 3
    • 3
  • m

    Marwan Sarieddine

    02/05/2021, 1:20 AM
    Hi Folks, we ran a flow that triggered 11 downstream flows on prefect cloud using
    StartFlowRun
    , the 11 flows got “queued”, but since we are on team plan (i.e. limited by flow concurrency) - only 2 of the flows were scheduled and the rest were late runs - so far so good, then the late runs get scheduled and run, but for some unkown reason 5 of the 11 late runs disappeared on their own and I don’t see why that happened
    n
    a
    • 3
    • 10
  • g

    Gabriel Santos

    02/05/2021, 6:29 AM
    Hello everyone! Does anyone use the prefect on heroku?
    n
    • 2
    • 4
  • l

    Laura Vaida

    02/05/2021, 12:56 PM
    hi folks, i want to connect prefect with my kubernetes engine on gcp. i used the following config for the dask scheduler:
    worker:
      replicas: 3
      resources:
        limits:
          cpu: 1
          memory: 3G
        requests:
          cpu: 1
          memory: 3G
      env:
        - name: EXTRA_CONDA_PACKAGES
          value: prefect
    jupyter:
      enabled: false
  • l

    Laura Vaida

    02/05/2021, 12:58 PM
    but somehow this doesnt really work, so there is no load balancer in gcp, does anybody know why?
    n
    • 2
    • 3
  • m

    Michael Hadorn

    02/05/2021, 1:32 PM
    Hi there Sadly i didn't understand really the docker-run and gitlab-storage concept. 😞 I built an image on top of the prefect image (prefecthq/prefect:0.14.6-python3.8) and installed my python deps successfully. In my flow, I added to parts:
    with Flow(
        prefect.config.general.flow_name
        , executor=LocalDaskExecutor()
        , storage=GitLab(
            host="<https://gitlab.xxx.ch/>"
            , repo="cdwh/cdwh-prefect"
            , path="cdwh_flow.py"
            , ref="master"
            , secrets=["GITLAB_ACCESS_TOKEN"]
        )
        , run_config=DockerRun(
            image="mrjck/cdwh-flow:1.0"
        )
        ,...
    Then I successfully registered the flow to the backend. If I let them run, the prefect docker agent deploying the flow but crashed immediately with the following message:
    404 Client Error for <http+docker://localhost/v1.40/images/create?tag=1.0&fromImage=mrjck%2Fcdwh-flow>: Not Found ("manifest for mrjck/cdwh-flow:1.0 not found: manifest unknown: manifest unknown")
    If I run 'docker images' the used images is there with the correct tag. How does my Image know, how the storage is pulled? UPDATE: Was solved the next day. If not, looks like this: https://github.com/PrefectHQ/prefect/issues/2781 Try to start the agent with: --no-docker-interface
    n
    m
    • 3
    • 46
  • m

    Milly gupta

    02/05/2021, 2:04 PM
    Hey all, can we create subtasks with in a task?
    ✅ 1
    n
    b
    • 3
    • 4
  • j

    Jeremy Phelps

    02/05/2021, 6:39 PM
    Hello, Prefect community, I was wondering if there's a way to tell exactly where a log message, as seen in the Prefect Server's web console, actually came from. There seem to be at least three places any given log message can come from, and I have to log into different machines to hunt for their origin in each machine's syslog: • The Kubernetes pod created by the Kubernetes Agent. • The Dask scheduler machine. • A Dask worker machine. Knowing where the log message comes from is necessary to determine which part of the infrastructure is misconfigured. Thanks in advance.
  • n

    Neeraj Vyas

    02/05/2021, 6:41 PM
    Hi all, can someone can share the steps to store flows in my docker-image and use it using docker-hub with kubernetes agent? It will surely help
    m
    • 2
    • 7
  • j

    jeff n

    02/05/2021, 8:37 PM
    Howdy, We are trying to streamline configurations in our tasks but are unable to use the Prefect Secrets due to security. Our rough draft used
    os.getenv()
    inside the tasks so that we pulled the secrets in at runtime but this is becoming incredibly unruly. I want to use a config library like dynaconfig so that I can mix settings files with env files. What is the best practice to load up the variables at run time in a smart way? Would the loading be a task that then pushes the config into the Prefect context?
    n
    • 2
    • 3
  • g

    Giovanni Giacco

    02/05/2021, 9:42 PM
    I need a tip from the community. In our new use cases we need to create flows at runtime, when a REST call arrives. In those cases where we can store the flows? What's the best solution? Maybe S3 storage?
    m
    • 2
    • 4
  • v

    Verun Rahimtoola

    02/05/2021, 11:00 PM
    hi, for flow params, is there a way to specify the type? kinda like how in
    argparse.add_argument(...)
    you can pass in something like
    type=int
    or some callable
    n
    • 2
    • 5
  • j

    james.lamb

    02/05/2021, 11:17 PM
    👋 hello from Chicago! Is there a status page for Prefect Cloud, where I could look for Prefect-Cloud-wide incidents?
    c
    m
    • 3
    • 5
  • f

    Fina Silva-Santisteban

    02/05/2021, 11:42 PM
    Hi Prefect community! We’re currently running the prefect server, prefect docker agent with a local docker image and I’m using Docker storage which points at my local Dockerfile. That works fine, but switching the backend from server to cloud somehow doesn’t. We have Prefect Cloud seats and I’ve successfully authenticated my agent, and I’ve successfully registered my flows. When I try to run a flow using the Prefect Cloud UI, my prefect docker agent console says
    Deploying flow run...
    but the flow gets stuck in the
    Submitted
    stage. What am I missing?
    c
    m
    • 3
    • 10
  • m

    Matt Denno

    02/06/2021, 11:28 PM
    Hello - I am having an issue getting my flows to run from the server after registering them. I can run them by calling
    flow.run()
    but if I
    flow.register(name="Task Name")
    and then try to run via the UI it fails. The message I get is:
    Failed to load and execute Flow's environment: ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.')
    I feel like I have done exactly this setup before with Prefect and it has worked fine, but now it is not working. I see it is related to django not being setup, but can't understand why it isn't...I guess because I don't complete understand how Prefect works. A simple example that demonstrates the issue is as follows:
    import django
    import os
    import prefect
    
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'afs.settings')
    django.setup()
    
    from prefect import Flow, task
    from api.models import Location # This is a django model import
    
    
    @task()
    def django_task():
        locs = Location.objects.all()
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>(f"Number of Locations: {len(locs)}")
    
    with Flow(f"Django Task") as flow:
        locs = django_task()
    
    
    # flow.run()
    flow.register(project_name="Kerala")
    Searching the Slack channel for Django returned one result which indicated the maybe using a resource manager was the way to set this up, but I wasn't able to get it to work using resource manager either. The other thing I tried was to set DJANGO_SETTINGS_MODULE via a LocalRun:
    flow.run_config = LocalRun(env={'DJANGO_SETTINGS_MODULE': 'afs.settings'})
    flow.register(project_name="Kerala")
    But that results in another error about INSTALLED_APPS not being configured...clearly I am missing something. Last thing, I also added the following to a a separate non django task, and it looks like it is using the correct virtual env to execute the task and has the correct paths.
    <http://logger.info|logger.info>(sys.path)
    <http://logger.info|logger.info>(sys.version)
    <http://logger.info|logger.info>(sys.executable)
    Really stumped... Any help would be greatly appreciated.
    j
    a
    • 3
    • 4
  • d

    Dimosthenis Schizas

    02/07/2021, 12:19 AM
    Hi everyone, sorry if my question is really dumb, but I couldn't find anything in the documentation. How can I register a flow with a date parameter, that will run on a schedule and will have by default the date of running. When i create the parameter like this:
    auction_date = Parameter(
            "auction_date", default=prefect.context.date.date()
        )
    I get the following error message when trying to register:
    AttributeError: 'Context' object has no attribute 'date'
    I am missing something here
    a
    • 2
    • 4
  • n

    Nathan Atkins

    02/07/2021, 12:36 AM
    Glad to be back doing some work with Prefect. I can see a lot has changed since I was last working with it. I'm having some problems with
    prefect
    commands against my local server. I guess it could be a installation or configuration problem, but debugging makes it look like there is an issue with initializing Client objects to talk to the local server. Specifically:
    prefect create project my_project
    and
    flow.register(project_name='tutorial')
    Both generate
    prefect.utilities.exceptions.ClientError: Malformed response received from API.
    errors. If I debug both problems I wind up in
    prefect.client.Client.___init___()
    where it looks like it sets the api_server to the cloud host because no
    api_server
    is passed in. I have the backend set to to
    server
    . In
    prefect.cli.create.project()
    Client().create_project(project_name=name, project_description=description)
    it looks like the
    api_server=localhost
    needs to be passed to
    Client.___init___()
    or Client needs to do something other than
    self.api_server = api_server or prefect.context.config.cloud.get("graphql")
    Same issue in
    prefect.core.flow.py
    I can connect to the server fine from my local browser or through cloud.prefect.io switched to the local server. (Nice feature by the way.)
    j
    • 2
    • 13
  • d

    Dolor Oculus

    02/07/2021, 1:42 AM
    This is a 🍒 on 🔝 thing, but is there any way to make a flow with conditions look better? I have a flow with some nested case() statements (on the order of, "if this file exists, and this other file exists, then do this..."). Was hoping for , got
    a
    • 2
    • 2
  • r

    Rob Fowler

    02/08/2021, 11:24 AM
    odd, I have had a prefect flow running for months, not I am getting: Unexpected error occured in FlowRunner: ValueError('Could not infer an active Flow context.')
    m
    j
    • 3
    • 11
  • r

    Rob Fowler

    02/08/2021, 12:02 PM
    seems
    executor = LocalDaskExecutor(scheduler="processes", num_workers=10)
        for ii in flow.run(executor=executor).result[result].result:
            print(ii)
  • r

    Rob Fowler

    02/08/2021, 12:02 PM
    does not work anymore
  • j

    James Phoenix

    02/08/2021, 12:52 PM
    Hey everyone.
  • j

    James Phoenix

    02/08/2021, 12:52 PM
    I would like to execute flows from inside of a flask route.
  • j

    James Phoenix

    02/08/2021, 12:53 PM
    Is it possible to run flows this way?
  • j

    James Phoenix

    02/08/2021, 12:53 PM
    import prefect
    client = prefect.Client()
  • j

    James Phoenix

    02/08/2021, 1:22 PM
    Is it this?
  • j

    James Phoenix

    02/08/2021, 1:22 PM
    client.create_flow_run(flow_id="<flow id>")
    e
    • 2
    • 4
  • j

    James Phoenix

    02/08/2021, 1:22 PM
    https://docs.prefect.io/orchestration/concepts/flow_runs.html#ui
Powered by Linen
Title
j

James Phoenix

02/08/2021, 1:22 PM
https://docs.prefect.io/orchestration/concepts/flow_runs.html#ui
View count: 1