https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • i

    itay livni

    12/24/2020, 4:19 PM
    Hi - I've asked this question previously many versions ago. Is
    apply_map
    the only way to map over
    ifelse
    and
    case
    statements?
    d
    • 2
    • 5
  • w

    wiretrack

    12/24/2020, 8:18 PM
    hey guys, trying prefect for the first time now, seems an amazing piece of work
    👋 3
    d
    • 2
    • 1
  • w

    wiretrack

    12/24/2020, 8:18 PM
    been having some problems organizing flow of flows:
    customer_flow = Flow('customer_flow')
    customer_flow.register(project_name="All Flows")
    
    customer_uptodate = check_status(source='db.sqlite3', target='db_target.sqlite3', table='customers')
    test_target_db = test_target_db()
    test_source_db = test_source_db()
    
    customer_flow.add_task(test_source_db)
    customer_flow.add_task(test_target_db)
    customer_flow.add_task(customer_uptodate)
    I get the error “ValueError: Could not infer an active Flow context.” (edited) but I couldn’t understand what exactly i’m doing wrong using the imperative API. registered the flow, registerd the tasks, add_task to flow. i didn’t understand what could I do differently, since the error appears right at my first task (customer_uptodate = check_status(source=‘db.sqlite3’, target=‘db_target.sqlite3’, table=‘customers’)
  • w

    wiretrack

    12/24/2020, 8:19 PM
    any ideas, hints, suggestions, links to articles that could be of any help?
    d
    • 2
    • 29
  • j

    Joël Luijmes

    12/25/2020, 11:02 AM
    Hey, can I create a task which is triggered by a Retry? Or create some retry logic for multiple tasks? For instance, I spawn off a kubernetes job, but if the job fails, I first want to cleanup that job before retrying.
    d
    • 2
    • 2
  • j

    jeff n

    12/26/2020, 4:43 AM
    I am just starting with Prefect and using the local server. One question I have is around object sharing. If I have a client I need to setup to call something like Reddit. When I have multiple tasks that need to access it where do I set it up? Do I need to set it up repeatedly in each task? Pass it to the tasks via a simple parameter to the task? Looking for best practices on this please.
    a
    • 2
    • 1
  • j

    Jay Shah

    12/27/2020, 9:48 AM
    Hi, Can someone please help me with the - RuntimeError: Unable to find any timezone configuration? I have Helm Dask chart installed on AWS EKS cluster. The deployment has two load balancer services that expose dask-scheduler and Jupyter notebook. I am trying to register the flow to the prefect cloud from the deployed Jupyter Notebook. The workflow runs fine on the dask-scheduler. self._access_token_expires_at is part of the init so I can't set the timezone.
    self._access_token_expires_at = pendulum.now()
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-5-fa84dadfda8e> in <module>
    ----> 1 flow.register("prefect-workflow")
    
    /opt/conda/lib/python3.8/site-packages/prefect/core/flow.py in register(self, project_name, build, labels, set_schedule_active, version_group_id, no_url, idempotency_key, **kwargs)
       1658             self.result = self.storage.result
       1659 
    -> 1660         client = prefect.Client()
       1661 
       1662         registered_flow = client.register(
    
    /opt/conda/lib/python3.8/site-packages/prefect/client/client.py in __init__(self, api_server, api_token)
         92         self._access_token = None
         93         self._refresh_token = None
    ---> 94         self._access_token_expires_at = pendulum.now()
         95         self._active_tenant_id = None
         96         self._attached_headers = {}  # type: Dict[str, str]
    
    /opt/conda/lib/python3.8/site-packages/pendulum/__init__.py in now(tz)
        209 
        210     if tz is None or tz == "local":
    --> 211         dt = _datetime.datetime.now(local_timezone())
        212     elif tz is UTC or tz == "UTC":
        213         dt = _datetime.datetime.now(UTC)
    
    /opt/conda/lib/python3.8/site-packages/pendulum/tz/__init__.py in local_timezone()
         58     Return the local timezone.
         59     """
    ---> 60     return get_local_timezone()
    
    /opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py in get_local_timezone()
         33 
         34     if _local_timezone is None:
    ---> 35         tz = _get_system_timezone()
         36 
         37         _local_timezone = tz
    
    /opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py in _get_system_timezone()
         61         return _get_darwin_timezone()
         62 
    ---> 63     return _get_unix_timezone()
         64 
         65 
    
    /opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py in _get_unix_timezone(_root)
        240         return TimezoneFile(tzpath)
        241 
    --> 242     raise RuntimeError("Unable to find any timezone configuration")
        243 
        244 
    
    RuntimeError: Unable to find any timezone configuration
    I get a similar error when I run: import pendulum pendulum.now()
  • s

    Slackbot

    12/27/2020, 10:36 AM
    This message was deleted.
    j
    a
    • 3
    • 11
  • a

    Ajith Kumara Beragala Acharige Lal

    12/28/2020, 11:58 AM
    Hi Prefect Experts, UI fails to connect to the
    graphql
    , not sure what causing this , any idea
    a
    • 2
    • 20
  • m

    Marwan Sarieddine

    12/28/2020, 2:31 PM
    Hi prefect team, I am wondering if there is a plan to implement executor callbacks now that 0.14.0 has migrated away from environments. I see this discussed here https://prefect-community.slack.com/archives/CL09KU1K7/p1602269846235100 but I don't see any mention of this in the changelog.
    m
    j
    • 3
    • 12
  • a

    Ajith Kumara Beragala Acharige Lal

    12/28/2020, 5:17 PM
    Hi Prefect experts, anyone can help with this ?
    m
    • 2
    • 70
  • m

    Matthew Blau

    12/28/2020, 7:41 PM
    Hello all, I am attempting to run a docker container from within a prefect flow. Currently the file that I am trying to run lives within the file system as seen https://pastebin.com/NjuF5yFR here in my code. I have a volume that needs to be mounted for it to be able to locate the file, but I am unsure of the syntax. What am I doing wrong?
    m
    • 2
    • 34
  • m

    matta

    12/28/2020, 11:21 PM
    Any idea what this error message could mean when trying to run a flow from the GUI?
    {'_schema': 'Invalid data type: None'}
    m
    • 2
    • 3
  • m

    matta

    12/28/2020, 11:21 PM
    That's all it says
  • m

    matta

    12/28/2020, 11:21 PM
    Not seeing logs either
  • m

    matta

    12/28/2020, 11:22 PM
    It runs fine in a notebook
  • m

    matta

    12/28/2020, 11:22 PM
    We're using Prefect Core on a kubernetes cluster
  • m

    Matthew Blau

    12/29/2020, 1:23 PM
    Hello all, my company is happy with what I was able to do with Prefect and wants to move forth with deploying it, however, I was told that we would prefer for Prefect itself to be in a docker container when we do the deployment. I do see that their is a docker image on docker hub, however I cannot locate any docs on how to build the image. Can someone point me in the right direction?
    a
    j
    • 3
    • 31
  • j

    John Grubb

    12/29/2020, 4:54 PM
    Hi, probably dumb Q but from your docs on Task Concurrency -
    This means that Prefect Cloud will ensure that no more than 10 tasks with the "database" tag will be running at any given time.
    this also applies to mapped tasks, yes? So say I want to backfill some data off of a slow API endpoint, I don't want to bury the application with tons of concurrent requests but I also don't want to switch over to the LocalExecutor for the job. Could I set a concurrency limit of
    2
    for this API and will the mapped tasks behave as intended?
    j
    • 2
    • 2
  • r

    Ryan Abernathey

    12/29/2020, 4:57 PM
    Quick q for the prefect devs: I'm curious why task concurrency limits are a cloud feature. This feels to me more like a core feature, since it is presumably something that has to be managed by agents, which are independent of cloud. https://docs.prefect.io/orchestration/concepts/task-concurrency-limiting.html
    j
    j
    +2
    • 5
    • 20
  • p

    Peter Erickson

    12/29/2020, 6:37 PM
    It looks like the latest
    prefecthq/prefect:0.14.1
    image doesn't contain the
    dask-kubernetes
    python module. Does anyone know if there's a different image to use or is it best to build my own image?
    j
    • 2
    • 2
  • h

    Henrik VÀisÀnen

    12/29/2020, 6:56 PM
    Hey, could someone point, how to get the number of threads and cores Dask worker has within Prefect task? I'm not spawning them from prefect but using the existing workers
    j
    • 2
    • 6
  • j

    Jeff Brainerd

    12/29/2020, 9:11 PM
    Hello 😛refect: team— qq on versioning flows:
    j
    a
    • 3
    • 10
  • d

    Dilip Thiagarajan

    12/29/2020, 10:02 PM
    hi all, is there a reason that
    CloudFlowRunner
    and
    CloudTaskRunner
    are used when running a flow using a
    LocalAgent
    ? I was expecting
    FlowRunner
    and
    TaskRunner
    to be used (the backend is also “server”)
    j
    • 2
    • 24
  • a

    Amanda Wee

    12/29/2020, 10:08 PM
    Thought I'd mention this as it might be by design: the slack notification utilities used to work with prefect core alone, but after 0.11.5 they seem to require prefect cloud/server. I discovered this issue when trying to upgrade to 0.13.x in my quest to get my team's existing code to work with prefect server.
    j
    • 2
    • 1
  • j

    jeff n

    12/30/2020, 12:41 AM
    I am struggling a bit with a parameter. It is being passed into my task at run time still a parameter. When I try to pass it to something expecting a string it doesn’t evaluate to the value. What am I missing on how to use parameters as normal values at run time.
    j
    • 2
    • 7
  • e

    Equipe AI HOC

    12/30/2020, 9:56 AM
    Hello, just arrived.
    j
    • 2
    • 1
  • s

    Sébastien

    12/30/2020, 10:06 AM
    One of my flows, which usually works just fine, failed yesterday with this unusual error:
    prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <https://api.prefect.io/graphql>
    
    The following error messages were provided by the GraphQL server:
    
        INTERNAL_SERVER_ERROR: Variable "$input" got invalid value null at
            "input.states[0].task_run_id"; Expected non-nullable type UUID! not to be null.
    
    The GraphQL query was:
    
        mutation($input: set_task_run_states_input!) {
                set_task_run_states(input: $input) {
                    states {
                        id
                        message
                        status
                }
            }
        }
    Runs (for different flows) after the failed run work again. This failed flow will run again at 4pm (in ~5h). Is this a known issue? Did I do something wrong?
    j
    • 2
    • 1
  • m

    Matthew Blau

    12/30/2020, 2:24 PM
    Hello all, I have attempted to install prefect on a Ubuntu server via
    pip3 install prefect
    and when I run
    prefect backend server
    I get a
    prefect: command not found
    my python version is 3.6.8. What am I doing wrong?
    j
    • 2
    • 26
  • k

    kevin

    12/30/2020, 5:37 PM
    Hey guys, just a quick question: does the
    defaults_from_attrs
    decorator on
    PostgresFetch.run()
    allow me to build a bunch of queries out like this:
    with Flow('foo') as flow:
        pw = PrefectSecret('pass')
        pg_template = PostgresFetch(host='host',user='user',db_name='name', port=1234)
        pg_query1 = pg_template(password=pw, query='select * from foo;')
        pg_query2 = pg_template(password=pw, query='select * from bar;')
    j
    • 2
    • 7
Powered by Linen
Title
k

kevin

12/30/2020, 5:37 PM
Hey guys, just a quick question: does the
defaults_from_attrs
decorator on
PostgresFetch.run()
allow me to build a bunch of queries out like this:
with Flow('foo') as flow:
    pw = PrefectSecret('pass')
    pg_template = PostgresFetch(host='host',user='user',db_name='name', port=1234)
    pg_query1 = pg_template(password=pw, query='select * from foo;')
    pg_query2 = pg_template(password=pw, query='select * from bar;')
j

josh

12/30/2020, 6:08 PM
Looking at the code for the task it seems like that should be possible. Are you seeing otherwise?
k

kevin

12/30/2020, 6:09 PM
I haven't executed the code yet, I was just wondering if I was going in the right direction and using the internal functions correctly
It seems kind of odd to me that password is deferred to the
run()
method on the task rather than at instantiation
j

josh

12/30/2020, 6:11 PM
The
PrefectSecret
task is a preferred way of passing sensitive information to the task and for the value to be read it needs to be passed in at runtime
k

kevin

12/30/2020, 6:12 PM
gotcha, so can I do something like
PrefectSecret(os.environ.get('db_pw'))
?
j

josh

12/30/2020, 6:13 PM
If the value in your environment for
db_pw
is the name of the secret then you should be good to go. Take a look at this doc on secrets for some more information about how they are used: https://docs.prefect.io/core/concepts/secrets.html
k

kevin

12/30/2020, 6:14 PM
awesome, thanks for your help!
View count: 1