https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • m

    Michael Eggert

    08/19/2021, 4:12 AM
    Hello. I'm struggling to get prefect to run/install at all. I find that the documentation is severely lacking in terms of having a useful guide on walking you through an install from scratch if you've never used Prefect before. I have a new Ubuntu 18.04 Server installation, docker and docker-compose installed, python 3.7 setup and set as the default. Run "pip install prefect" it's installed, the documentation says to now run "prefect server backend" or really "prefect" anything and I get nothing. Where are the steps in between to getting this running? https://github.com/prefecthq/prefect
    a
    k
    +1
    11 replies · 4 participants
  • a

    Aiden Price

    08/19/2021, 6:14 AM
    Hi everyone, I'm having trouble with a custom
    resource_manager
    class, the flow times out and errors like this appear;
    Error during execution of task: AttributeError("'NoneType' object has no attribute 'setup'")
    k
    18 replies · 2 participants
  • m

    Matias Godoy

    08/19/2021, 10:57 AM
    Hello everyone! Hope you are all ok. I am trying to use the GraphQL API to get some information about our flow_runs. Basically what I need is some information about every flow_run and how long they took to finish (or has taken since creation until now for the case of the ones still running). The thing is that I'm very new to GraphQL and I'm not sure what I want is doable. It should be something like this:
    query{
      flow_run{
        name,
        created,
        state,
        flow_id,
    	duration: <here goes the flow_run execution duration>
      }
    }
    Ideally I could do something like
    duration = end_time - start_time
    , but I think GraphQL does not allow operations like these? I also checked the
    flow_run_aggregate
    query, but it does not contain such information. The real problem is that I want to consume this data from Grafana, so I have nowhere to run these calculations and the data should come already calculated in the response. Any ideas on how could I do this? Thanks a lot!
    b
    k
    5 replies · 3 participants
  • i

    Italo Barros

    08/19/2021, 11:43 AM
    Hello everyone, there's a way to configure Prefect so he can use a proxy server? I'm trying to log in on Prefect cloud (using the CLI), but I'm receiving the following SSLError:
    requests.exceptions.SSLError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))
    k
    27 replies · 2 participants
  • p

    Pierre Monico

    08/19/2021, 12:11 PM
    On the installation page, the possibility of installing “extras” is mentioned, but I can’t figure out where the full list is. I guess a link would help. (Further, the reason why I landed on this page is because it seems I need the
    [git]
    extra to be installed for the Git Storage to work, which is not mentioned in the doc)
    n
    4 replies · 2 participants
  • k

    Krapi Shah

    08/19/2021, 12:40 PM
    Hi, We have a flow that consists of multiple flows. There are two parent flows on which multiple flows depend. The two parent flows are independent of each other. I see the flows execute serially satisfying dependency constraints... but don't see any nondependent flows running in parallel. Should we expect the two independent flows to be scheduled in parallel or they would still run serially one after the other?
    k
    c
    7 replies · 3 participants
  • a

    Abuzar Shaikh

    08/19/2021, 1:22 PM
    Hey guys, I am running the same flow on two different agents both with same configuration, the flow runs successfully on one ages while it fails on the other agent, with this error
    Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n  TypeError("int() argument must be a string, a bytes-like object or a number, not \'NaTType\'")')
    Any idea why?
    k
    5 replies · 2 participants
  • m

    Marwan Sarieddine

    08/19/2021, 1:22 PM
    Hi Folks, a question about tasks that return a tuple of items and prefect task mapping.
    k
    4 replies · 2 participants
  • c

    Charles Leung

    08/19/2021, 2:00 PM
    Hey Team, When our agents go down due to a hosed host, we find that we can get an email notification via automations setup (agent is unhealthy). However, when we change it to pagerduty, we find we do not get alerts. How can we resolve this?
    k
    j
    +1
    13 replies · 4 participants
  • j

    Jack Sundberg

    08/19/2021, 2:56 PM
    Docker doesn't play well with academic clusters because of the root-privilege requirements, but has anyone used Prefect with Singularity? It's an alternative to Docker but can still pull from Docker Hub images. Prefect storage and agent types for Singularity would be really helpful in academic HPC use cases.
    w
    4 replies · 2 participants
  • y

    YD

    08/19/2021, 4:00 PM
    is there a simple way, like a pinging some ip address, that can tell me if I have access from inside a production environment to the cloud service ?
    k
    w
    36 replies · 3 participants
  • k

    Kathryn Klarich

    08/19/2021, 4:51 PM
    What is the purpose of the
    mapped
    argument in
    Task.set_upstream
    ? is this just another way of applying mapping, or should we always be setting
    mapped=True
    if the upstream task is mapped? (we usually use
    Task.map()
    when mapping tasks)
    k
    5 replies · 2 participants
  • j

    Jean Da Rolt

    08/19/2021, 5:19 PM
    Hi folks, can I have nested tasks?
    @task
    def f1():
      ...
    @task
    def f2():
      f1()
      ...
    k
    m
    3 replies · 3 participants
  • a

    An Hoang

    08/19/2021, 5:25 PM
    Can I/Is it a good practice to put a result of a task inside the
    prefect.context
    to be used in all subsequent tasks? I have a stateful variable that many tasks will use and would love to use it inside many tasks without passing in as parameter
    ✅ 1
    k
    14 replies · 2 participants
  • j

    Joe Schmid

    08/19/2021, 8:30 PM
    Running into an issue where some mapped tasks fail with state TimedOut without an exception or apparent failures in the task run logs. Are there specific conditions that would cause a task to end in TimedOut state? (Feel free to point me to docs or code if needed.) Example log output in thread reply:
    c
    8 replies · 2 participants
  • j

    John Ramirez

    08/19/2021, 8:42 PM
    I want to use prefect to validate api data from an endpoint; is there a good way to report the results of the validations for example data for this ID was wrong
    k
    3 replies · 2 participants
  • m

    Michael Welsh

    08/19/2021, 8:46 PM
    Hi general question here When running flows lately I have been noticing lately long waits for workers, hour+, and when I do happen to get a worker upon creation of a cluster its only ever one worker. Is this common? I tried just creating coiled clusters and they have no problem getting workers. Any help is appreciated thanks.
    k
    g
    14 replies · 3 participants
  • a

    Aric Huang

    08/19/2021, 9:39 PM
    Hi, it seems that when running flows locally with the Prefect CLI it will abort if passed parameters that don't exist in the flow, but when running registered flows it will pass them through and run with no errors. Is there a way for the CLI to validate parameters and show an error if a param doesn't exist? We'd like to catch this in case someone passes a misspelled param name, for example.
    # Running the flow locally
    aric@aric-dev:~/src/conductor$ prefect run -p conductor/samples/hello_world_flow.py --param foo=bar
    Retrieving local flow... Done
    Configured local flow run
    └── Parameters: {'foo': 'bar'}
    Running flow locally...
    Flow runner encountered an exception!
      Traceback (most recent call last):
        File "/home/aric/.local/lib/python3.8/site-packages/prefect/cli/run.py", line 590, in run
        result_state = flow.run(parameters=params_dict)
        File "/home/aric/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1250, in run
        raise ValueError(
      ValueError: Flow.run received the following unexpected parameters: foo
    
    Flow run failed!
    
    # Running the same flow registered on Prefect Cloud
    aric@aric-dev:~/src/conductor$ prefect run -n "Hello World" --param foo=bar
    Looking up flow metadata... Done
    Creating run for flow 'Hello World'... Done
    └── Name: monumental-bat
    └── UUID: 39c7896b-0bee-4015-be99-648d90da201b
    └── Labels: []
    └── Parameters: {'foo': 'bar'}
    └── Context: {}
    └── URL: <https://cloud.prefect.io/><URL>
    m
    4 replies · 2 participants
  • s

    Sepehr Sadighpour

    08/20/2021, 1:10 AM
    Hi all, has anyone toyed with deployment of flows from S3? I’d like to speed up our CI/CD process, and wondering if it’d make sense to have a “meta” Flow fetch a python package of other Flows from S3, download and register them, etc. Airflow has had this on their to-do list for years (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher), and I’d love for Prefect to beat them to the punch. Reducing deployment from 20-30 minutes to seconds would be a game-changing differentiator
    k
    3 replies · 2 participants
  • a

    Aiden Price

    08/20/2021, 5:57 AM
    What is the best approach to calling an async function within a Prefect task? I have to call the Azure Eventhub client which is async all the way down, from my straight synchronous Prefect task. I've tried calling the async functions like so;
    asyncio.run(client.send_batch(event_data_batch))
    but I get errors like
    RuntimeError: Event loop is closed
    Any advice?
    n
    k
    +1
    5 replies · 4 participants
  • m

    Maria

    08/20/2021, 8:26 AM
    Hi all, I'm triggering my flow with AWS lambda and this great example answered most of my questions: https://github.com/cicdw/prefect-cloud-lambda . But I'm just wondering, is there an easy way to get latest flow_id for my flow_group, rather than hardcoding flow_id in there (and changing every time I deploy a new version)? Is there another env var I can specify maybe?
    b
    k
    3 replies · 3 participants
  • p

    Pedro Machado

    08/20/2021, 12:43 PM
    Hi everyone. I have a flow that triggers another flow using
    create_flow_run
    and
    get_task_run_result
    . I am using these two tasks inside of another task so this can be retried as a unit (see code in thread). Last night, the child flow was triggered three times (triggered by the retries in the parent flow) and all three times the flow succeeded so it looks like there was an issue where the parent flow misread the child flow's status. I am attaching the logs in the thread. Am I misusing these tasks? Any ideas as to why this could be happening? This has been running every night for a couple of weeks and it's the first time this problem happens. Thanks!
    s
    k
    +1
    15 replies · 4 participants
  • b

    Bastian Röhrig

    08/20/2021, 1:05 PM
    Hey everyone, do you know any resources for best practises / guides regarding production deployment of agents? I'm looking at https://docs.prefect.io/orchestration/agents/overview.html#agent-types and am wondering whether a single vm using a docker agent would be fine. I am looking for something with very little overhead as our team is very small (3 people, so I feel kubernetes agent is not really an option). Thanks!
    k
    k
    6 replies · 3 participants
  • d

    Dmitry Kuleshov

    08/20/2021, 1:48 PM
    Hi all! Can someone rub me into the documentation where I can find information on how prefect works with long-run jobs. We are interested in what will happen if there will be temporary network issue. What will happen with the job and do we have any choices to configure behaviour for that.
    k
    3 replies · 2 participants
  • d

    David McGuire

    08/20/2021, 3:03 PM
    Has anyone had trouble with a localized
    CronClock
    schedule skipping
    Flow
    runs, entirely?
    k
    23 replies · 2 participants
  • c

    Constantino Schillebeeckx

    08/20/2021, 4:15 PM
    Does prefect support intersphinx for documentation purposes?
    k
    8 replies · 2 participants
  • w

    Wilson Bilkovich

    08/20/2021, 4:38 PM
    Has anybody used Arrow for anything with Prefect/Dask? https://arrow.apache.org/docs/python/index.html It seems appealing, I’m just wondering if there are sharp edges.
    k
    11 replies · 2 participants
  • k

    Kevin Kho

    08/20/2021, 6:21 PM
    Archiving this myself: SSL Errors when using Prefect agents can be fixed with updating the package “certifi”, running:
    conda update certifi
    Follows the overflow post  where I found a similar problem with this solution: https://stackoverflow.com/questions/41691327/ssl-sslerror-ssl-certificate-verify-failed-certificate-verify-failed-ssl-c
    m
    2 replies · 2 participants
  • k

    Kyle Hansen

    08/20/2021, 6:40 PM
    I had a quick question about the
    .map
    within prefect. Is it guaranteed that the order is preserved across multiple maps, like a python
    multiprocessing.pool.map
    ? Or is it not a guarantee that the order is preserved, like
    multiprocessing.pool.imap_unordered
    . I couldn't find whether this was guaranteed or not in the
    prefect docs
    .
    k
    8 replies · 2 participants
  • j

    Jean Da Rolt

    08/21/2021, 12:56 AM
    folks, I'm coming from a luigi background, where we can add custom loggers on top of luigi loggers:
    [core]
    logging_conf_file = logging.cfg
    then
    [loggers]
    keys=root,app
    
    [handlers]
    keys=consoleHandler,StackDriveHandler
    ...
    Is there anything similar in Prefect? I do not want to add additional logs inside my tasks, but make prefect log using additional loggers.
    k
    m
    5 replies · 3 participants
Powered by Linen
Title
j

Jean Da Rolt

08/21/2021, 12:56 AM
folks, I'm coming from a luigi background, where we can add custom loggers on top of luigi loggers:
[core]
logging_conf_file = logging.cfg
then
[loggers]
keys=root,app

[handlers]
keys=consoleHandler,StackDriveHandler
...
Is there anything similar in Prefect? I do not want to add additional logs inside my tasks, but make prefect log using additional loggers.
k

Kevin Kho

08/21/2021, 1:24 AM
Hey @Jean Da Rolt, yep we have this .
j

Jean Da Rolt

08/21/2021, 2:32 PM
Hi Kevin, thanks for the reply. I may be wrong, but what the extra_loggers do is to take all messages from the listed loggers and send it to stdout by attaching a streamhandler for each of the elements. What I wanted is the oposite: attach a customHandler to Prefect Logger, inside the config. I know that this works:
import logging

from logging import StreamHandler

class BlablaHandler(StreamHandler):

    def __init__(self):
        StreamHandler.__init__(self)


    def emit(self, record):
        msg = self.format(record)
        print("Printing inside handler")
        print(msg)

logger = logging.getLogger()
logger.addHandler(BlablaHandler())
however I would prefer to have that in the prefect config file. Is that possible?
m

Michael Adkins

08/21/2021, 3:07 PM
Hey Jean, that's not possible right now. We have a proposal to add config via a logging file like you showed above but it's not done yet.
You have to write a custom
get_logger
function that, on logger retrieval, adds the extra handlers you want
j

Jean Da Rolt

08/21/2021, 6:22 PM
ok, thank you
View count: 3