https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • h

    Hawkar Mahmod

    03/08/2021, 4:31 PM
    I have a Flow I wish to register but that fails the Prefect healthcheck. It attempts to import a library that requires AWS credentials, which are then not found, and a Exception is raised, this causes the healthcheck to fail. I am using Docker Storage and and ECS Run Config. The reason it fails is that one of my Flow's dependencies uses AWS credentials that locally I provide via environment variables and that in production I provide by an EC2 instance metadata. I don't want to load the Flow, or the Docker image I use for Storage with any credential information for obvious reasons. How do I overcome this? I know about Prefect Secrets but it's something I want to avoid because it's unnecessary considering in production the credentials are available automatically using the EC2 metadata service. Health check fails at :
    /opt/prefect/healthcheck.py
    d
    m
    f
    • 4
    • 12
  • t

    Trevor Kramer

    03/08/2021, 4:47 PM
    It looks like Prefect swallows errors when using BatchSubmit. I am getting a generic "FAIL signal raised: FAIL("Failed to submit job 'blah' to AWS Batch.")" error. This makes it hard to debug since this flow runs fine locally but is failing on my ECSRunner. Can you log the stack trace or error message from boto3?
    👀 2
    d
    m
    • 3
    • 6
  • a

    Andor Tóth

    03/08/2021, 5:09 PM
    Just a simple question. Can I override
    yesterday
    in context? Like this:
    with prefect.context(yesterday="2021-01-01"):
        flow.run(run_on_schedule=False, executor=LocalExecutor())
    d
    • 2
    • 34
  • a

    Andor Tóth

    03/08/2021, 5:11 PM
    I'd like a flow to run on yesterday's data by default, but for a backfills I should be able to override it.
  • a

    Andor Tóth

    03/08/2021, 5:11 PM
    Actually, I've tried that above, but it does not work
  • t

    Trevor Kramer

    03/08/2021, 5:29 PM
    I'm having trouble understanding how Contexts work. This code always seems to return None
    prefect.context.get('flow_run_name')
    How do I get the flow run name (like zealous-dormouse) from inside the flow?
    d
    m
    • 3
    • 16
  • k

    kevin

    03/08/2021, 5:47 PM
    can the Prefect
    SlackTask()
    be configured to send messages to different channels depending on what flow it is being run in?
    d
    • 2
    • 11
  • l

    Luis Gallegos

    03/08/2021, 6:44 PM
    Hi all, Can i prevent run two instance of a scheduled flow?. I dont found any configuration to do that. Thanks for your help
    k
    • 2
    • 2
  • j

    james.lamb

    03/08/2021, 8:59 PM
    👋 hello from Chicago! When configuring a flow using run configs, is it possible to provide extra data (think a small dictionary whose values are strings or ints) that will show up in
    flow.serialize()
    ? Will share more details in thread, thanks!
    j
    • 2
    • 10
  • s

    Soren Daugaard

    03/08/2021, 9:25 PM
    👋 hello all! I’ve been playing around with the great expectations validation task using the documentation here: https://docs.prefect.io/api/latest/tasks/great_expectations.html#rungreatexpectationsvalidation. I don’t have
    validation_operators
    in my
    great_expectations.yml
    file so I call the task as follows:
    validation_task = RunGreatExpectationsValidation()
        validation_results = validation_task(
            batch_kwargs=validation.get_batch_kwargs(records, 'pandas_s3'), expectation_suite_name=GE_EXPECTATION_SUITE_NAME, validation_operator=None
        )
    However when I execute my flow I get the following error:
    Unexpected error: DataContextError('No validation operator `None` was found in your project. Please verify this in your great_expectations.yml')
    Traceback (most recent call last):
      File "/Users/soren.daugaard/Code/data-pipelines/.direnv/python-3.8.6/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 1272, in run_validation_operator
        validation_operator = self.validation_operators[validation_operator_name]
    KeyError: None
    According to the documentation that parameter should be optional, but I am not able to figure out how to set it to nothing. I tried
    None
    and the empty string but with similar errors. Any pointers would be much appreciated.
    k
    • 2
    • 7
  • m

    MasatoShima

    03/09/2021, 7:56 AM
    Hi ! I am trying to deploy Prefect Server and UI on Amazon ECS (Fargate). But I have a problem… Deploying seems to succeed. However, when I opened the Prefect UI screen, the page did not seem to display correctly. When I look at the console of Google Chrome, I get the following error.
    Uncaught (in promise) Error: passed invalid or empty tenant object
        at l.setDefaultTenant (index.js:51)
        at vuex.esm.js:840
        at vuex.esm.js:462
        at Array.forEach (<anonymous>)
        at vuex.esm.js:461
        at l._withCommit (vuex.esm.js:620)
        at l.commit (vuex.esm.js:460)
        at l.commit (vuex.esm.js:405)
        at tenantNavGuard.js:12
        at u (runtime.js:63)
    Does anyone have any idea how to solve this problem ? By the way, the architecture on AWS is as follows.
    client - ALB - ECS (Fargate)
    ✅ 1
    n
    • 2
    • 2
  • t

    Till

    03/09/2021, 9:41 AM
    hello, hello
  • t

    Till

    03/09/2021, 9:43 AM
    started playing with Prefect a bit on the weekend, i have some questions about the "docker-compose" setup, e.g., how I would role my own. e.g., I don't have python on the host and would like to run it in containers a 100%. I also tried the suggested docker image, but that drops me into a shell. the docs are a bit opaque here. would anyone have some pointers? 🙂
    j
    f
    • 3
    • 12
  • v

    Vitaly Shulgin

    03/09/2021, 10:07 AM
    Hello team, is it possible to use custom adjustment implementation for shcedule?
    m
    • 2
    • 1
  • l

    Laura Vaida

    03/09/2021, 10:38 AM
    hii, does anybody now how to declare "from datetime import datetime" as python dependency in docker storage?
    flow.storage = Docker(registry_url="<http://gcr.io/keller-steering-enabling/flows|gcr.io/keller-steering-enabling/flows>", image_name="billwerk-contracts",
                          python_dependencies=["pandas", "oauthlib ", "requests", "requests_oauthlib", "datetime", "oauth2client", "snowflake", "datetime"], secrets=["GCP_CREDENTIALS"])
    above doesn't properly work thanks!
    ✅ 1
    s
    a
    • 3
    • 8
  • t

    Trevor Kramer

    03/09/2021, 1:05 PM
    Is there a way to authenticate the client other than using prefect auth login? I am trying to use prefect inside a CI/CD pipeline and it would be ideal to set an environment variable instead.
    j
    • 2
    • 4
  • f

    Florian Kühnlenz

    03/09/2021, 2:45 PM
    Hi, I tried to install prefect on my local machine to test it. However I am stuck after starting it via docker-compose. I created the docker-compose.yml file via prefect server. However I can not connect to the server with the prefect agent local start command. This exits with a 'message': 'No tenant found.'. Also the UI on localhost is mostly empty. What am I missing? Logs from docker-compose up seem fine.
    j
    • 2
    • 12
  • k

    Konstantinos

    03/09/2021, 2:59 PM
    Hello, need some help with mapped/ upstream tasks. I'd like to run a mapped task, that will only start upon the success of another task but I am doing sth obviously wrong in the following simple example ( probably thinking of the parameter upstream_tasks wrong). Thanks in advance for your help
    g
    j
    • 3
    • 8
  • a

    Adam Brusselback

    03/09/2021, 3:45 PM
    Question, if I have multiple agents with the same labels, they seem to be running the tasks in a flow once each...that seems wrong
    j
    • 2
    • 17
  • m

    Mitchell Bregman

    03/09/2021, 4:11 PM
    Hey there, I had 2 flows that were running for a few days without realizing they were using up all the concurrency slots I had available in Cloud. I am trying to cancel all the “late” runs and essentially reset my scheduler… I have tried to do so via the interface, but it does not seem to clear the late runs out.. further, as I try to cancel the 2 flows that have been running, I am stuck in a “Cancelling” state. Is there any way someone can help me reset the current state of my Cloud? I can certainly kill the k8 pod that is running, but would rather it happen via UI. Thanks!
    j
    • 2
    • 20
  • h

    haf

    03/09/2021, 4:39 PM
    What happens when you run
    prefect register flow --file ... --project ...
    ? Is the python code in the file executed? Similarly, if I am using Prefect cloud, and a k8s agent that spawns jobs; what env vars are available when: 1. The job is starts the python file (thereby declaring the flow / DAG) 2. The flow is run
    • 1
    • 1
  • c

    Charles Liu

    03/09/2021, 5:33 PM
    Hi just moving this over from the wrong channel: I recently finished porting one of our pipelines over to prefect and have been eager to move out of local and over to an EKS/ECR configuration. I saw some guides by Anna and Jimmy that were really helpful so far, but I've hit multiple collisions and was seeking some help to work through some of them. Specifically two blockages I've encountered involve using our own custom base image, which is hosted on ECR, which is actually pulling code from CodeArtifact; that approach is hitting healthcheck issues when building. Another issue I'm having is being unable to create new EKS clusters through eksctl (where CloudFormation spits out max # of elastic IPs reached) , but I'm assuming that's more of a provisioning/permissioning issue and for that I'm just wondering if anyone else has gotten stuck at these points as well. I can however deploy an EKS Agent and view it in kubectl get pods, and publish a flow on Prefect Cloud regardless of these blockers, so I feel like I'm tangled up in the last few steps in getting this to work. (Thanks Dylan for pointing me to this channel!)
    m
    • 2
    • 13
  • l

    Laura Vaida

    03/09/2021, 6:03 PM
    hi folks, how to specify a secret with key, value pairs and to read it with PrefectSecret or Secret().get()? Thanks!
    ✅ 1
    j
    h
    c
    • 4
    • 28
  • h

    haf

    03/09/2021, 8:19 PM
    Can I supply secrets from env vars from the k8s agent's deployment?
    • 1
    • 1
  • h

    haf

    03/09/2021, 8:21 PM
    What call translates a Task into an ordinary value? E.g. what translates a
    Parameter
    to a value in a
    def
    ?
    j
    j
    • 3
    • 20
  • b

    Brian Mesick

    03/09/2021, 8:30 PM
    Hi all, question about logging… 🧵
    m
    • 2
    • 9
  • h

    haf

    03/09/2021, 9:04 PM
    ?
    b
    • 2
    • 2
  • h

    haf

    03/09/2021, 9:57 PM
    I'm trying to configure the final bits of my install on Kubernetes. I'm looking at https://docs.prefect.io/api/latest/cli/agent.html#kubernetes-install — vs the
    start
    command that also takes
    --job-template <path>
    — how do I configure my own job template when using
    prefect agent kubernetes install
    ?
    b
    b
    • 3
    • 15
  • m

    Maria

    03/09/2021, 10:42 PM
    Hi, I'm trying to find out what is the recommended way to bulk load data from csv into the database (eg postgres) regularly in prefect. I understand I can do "insert into table ... " and use postgres.PostgresExecuteMany task, but I'm concerned that for very large files it might be slow. Is there a way to execute pg copy (or COPY), or "insert into" is how I should do it in prefect?
    h
    • 2
    • 3
  • a

    Adam Brusselback

    03/09/2021, 11:11 PM
    @Maria You could use a shell task to execute psql COPY, etc
Powered by Linen
Title
a

Adam Brusselback

03/09/2021, 11:11 PM
@Maria You could use a shell task to execute psql COPY, etc
View count: 1