• 夏文思

    夏文思

    9 months ago
    💃 刚刚加入!
  • Yong Tian

    Yong Tian

    9 months ago
    🎉Glad to Join! Good Job! Prefect Inc.
  • a

    André Petersen

    9 months ago
    Hello again! I want to create a very rough price estimate based on the Prefect Setup described here (https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df) by @Anna Geller. I have not much experience with AWS, yet, which makes it difficult to estimate the price using the price estimator. We will only have a couple of batch jobs running a couple of times a day and the processing will take place exclusively on snowflake, so the prefect flows will only deligate work. Also we would probably use prefect cloud. I would guess that in the price calculator here https://calculator.aws/#/createCalculator/Fargate we could go with Linux OS, 2 tasks/pods running 2 hours on average with 2 vCPU allocated and 4GB memory (minimum) and 20GB of storage (minimum).1. Do you agree? If not, why? I can not really believe that this would only cost $14 per month. 2. Do I need to calculate additional costs for the ECS tasks? If I understand correctly, these are included in the Fargate price estimation. I know that we could save money by using spot instances. My target is to make a pessimistic/conservative price estimation. Thanks in advance! Really appreciate the help in this channel!
    a
    Anna Geller
    10 replies
    Copy to Clipboard
  • Daniil Ponizov

    Daniil Ponizov

    9 months ago
    Hi! is it possible to pass some information to the flow run in the UI? Often there is problem, that run depends on current time: for example it downloads data with timestamp equals to previous day and if this run fails and you find later than that day, you have to run the script manually with the appropriate arguments
    Daniil Ponizov
    Amanda Wee
    +1
    7 replies
    Copy to Clipboard
  • haf

    haf

    9 months ago
    KeyError: 'data'
    — have you seen this error message before?
    haf
    Anna Geller
    8 replies
    Copy to Clipboard
  • haf

    haf

    9 months ago
    What might be causing Heartbeat failures from Prefect / Dask? How would one go about debugging this?
    haf
    Anna Geller
    +1
    26 replies
    Copy to Clipboard
  • Vince Bob

    Vince Bob

    9 months ago
    hello, I am struggling on a great_expectations integration problem. I obviously use RunGreatExpectationsValidation task on a checkpoint I created in GE with:
    validation_task(
    context_root_dir=root_dir,
    checkpoint_name=expectation_checkpoint_name
    )
    When I run the command on GE (great_expectations --V3-api checkpoint run my_checkpoint), it works, but on prefect task, I have an exception: With GE V3 api:
    .....
    for batch in ge_checkpoint["batches"]:
    TypeError: 'Checkpoint' object is not subscriptable
    The same with GE V2 api
    ...
    for batch in ge_checkpoint["batches"]:
    TypeError: 'LegacyCheckpoint' object is not subscriptable
    Great_expectations=0.13.43 (also tried with 0.12.10 version) prefect=0.15.9 Anyone experienced this pb?
    Vince Bob
    Anna Geller
    5 replies
    Copy to Clipboard
  • Elijah Roussos

    Elijah Roussos

    9 months ago
    Hi all! I got a quick question if anyone can answer. We’ve got a couple flows running on ECS with a JSON secret for postgres DB access. I want to be able to test flows locally without deploying to Prefect cloud, but obviously that means setting the secret in
    ~/.prefect/config.toml
    locally. From the docs it seems like you can only set strings in the toml, but I need JSON. I’ve tried setting it as a JSON string and also in toml syntax to no avail. Is there any way to set a local JSON secret?
    Elijah Roussos
    Anna Geller
    3 replies
    Copy to Clipboard
  • a

    Adam Everington

    9 months ago
    .map().... is there an upper limit? ...and i'm sure i've asked before, but there's no way of batching it so say like 5 run at a time?
    a
    1 replies
    Copy to Clipboard
  • Anh Nguyen

    Anh Nguyen

    9 months ago
    Hi all. I cannt run flow . How to fix that ?
    Anh Nguyen
    Anna Geller
    2 replies
    Copy to Clipboard