https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Arun Dass

    08/21/2021, 4:32 AM
    Hey I am deploying a Kubernetes Agent using Cloud backend. I am confused on how the agent knows which flows belong to it. Is there a value in the metadata that tells cloud server that this flow belongs with this agent?
    m
    1 reply · 2 participants
  • a

    Anh Nguyen

    08/21/2021, 7:14 AM
    helo! Beside the logger is write in normal mode, I wanna write logging into table in Database. How to do that?
    k
    1 reply · 2 participants
  • c

    Chintan Shah

    08/21/2021, 4:19 PM
    Hello, trying to run a workflow using ecs agent - getting this error while starting ecs start ValueError: Failed to infer default networkConfiguration, please explicitly configure using
    --run-task-kwargs
    k
    5 replies · 2 participants
  • w

    Wilson Bilkovich

    08/21/2021, 11:01 PM
    I’ve seen some confusing examples, possibly representing different versions of Prefect over time. Is there a canonical example .prefect/config.toml? For example I saw
    host = foo
    in one video, and
    [server] endpoint = baz
    in another
    k
    20 replies · 2 participants
  • w

    Wilson Bilkovich

    08/21/2021, 11:32 PM
    Unrelated topic, I’m having a bit of trouble using S3 storage for a flow. Nothing seems to get created at the key I specify, but I’m not getting any errors.
    k
    14 replies · 2 participants
  • b

    Blake List

    08/22/2021, 3:26 AM
    Hi there, I am trying to build a "flow of flows", where two flows (a and b) are run by a parent flow (p). I have a cron clock which will run 5 times a day at specific times. I need to pass two varying parameters (t1 and t2) to flows a and b at the beginning of each schedule. These parameters represent timestamps where, the first time everything is scheduled, t1 will be
    '0000-00-00 00:00:00'
    and t2 will be
    time.now()
    . The next time schedule runs, t1 needs to be the previous value of t2, and t2 will once again be
    time.now()
    . I can see that I may need to use a task to compute the parameters within the script of the parent flow (similar to here) and use the
    StartFlowRun
    task (as seen here), but I'm not quite sure how to put it together. Any help would be appreciated, thank you!!
    k
    9 replies · 2 participants
  • c

    Chintan Shah

    08/22/2021, 11:32 AM
    HI there, i have issue while running a workflow through ecs agent ,
    An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Container.image should not be null or empty
    . this is run time error while trying to execute flow . any pointers
    👀 1
    k
    30 replies · 2 participants
  • b

    Ben Muller

    08/22/2021, 10:27 PM
    Hey community, I have previously worked with Dockerstorage for all my flows but as my repo has grown it has slown down my CI process, so I am switching to S3 stroage. I have a docker image baked in with all the dependancies and files etc for the repo. I have set up my storage like so:
    S3(
                bucket="es-prefect-flows-staging",
                stored_as_script=True,
            )
    Now after registering my flow I am getting an error after the flow is downloaded: in 🧵
    ✅ 1
    m
    6 replies · 2 participants
  • g

    Gaylord Cherencey

    08/23/2021, 3:51 AM
    Hello, I have 2 questions about Prefect Cloud: • Is there a list somewhere that detail the information sent to Prefect Cloud from Agents? • Is there documentation around Okta SSO implementation with Prefect Cloud? (more specifically when I when to have multi-tenancy) Thanks in advance for the help :)
    k
    n
    10 replies · 3 participants
  • e

    Eddie Atkinson

    08/23/2021, 5:06 AM
    Hey folks, Haven’t been able to figure this one out by reading the docs / threads here. I am trying to process data for sites in a date range one day at a time (with a task running for each combination of date and site_id). My current solution uses
    map
    to process the data for each site individually, but does so for the entire range at once. The complication arises from the fact that the date range is specified as a parameter, and the site_ids are the result of a task that runs.
    k
    6 replies · 2 participants
  • c

    Chhaya Vankhede

    08/23/2021, 6:47 AM
    I'm trying to update the flow schedule using graphql, there is an option in UI under setting where I can modify schedule. I want something similar in graphql
    k
    2 replies · 2 participants
  • i

    Italo Barros

    08/23/2021, 11:48 AM
    Hello everyone, I noticed something strange today and I don't know if it was a bug or not. I had registered a Flow using the API and started an agent at the same folder using the CLI. The Flow was schedule to Run on the Prefect Cloud and I did some "Quick Runs" to test/debug. Everything was running fine, however, the VM that I'm running the Agent was shut down for maintenance on the weekend, and I restarted the agent again (now in a different folder than before) an received the following error:
    Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'Submodules\'",)\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.',)
    The error was solved after shutting down the agent and starting again at the same folder of his first execution. I think this is related to the agent metadata, is this behavior normal?
    k
    3 replies · 2 participants
  • w

    Wilson Bilkovich

    08/23/2021, 12:29 PM
    When exactly do I need a
    RunConfig
    object? Do I need one if I’m already specifying an Executor?
    k
    4 replies · 2 participants
  • j

    Jelle Vegter

    08/23/2021, 1:05 PM
    Hey all, what is the best practice for creating variables (i.e. date stamp) that you want to re-use across tasks? I can't just declare the variable as the date stamp doesn't update after the initial registry.
    w
    k
    3 replies · 3 participants
  • b

    Bastian Röhrig

    08/23/2021, 2:14 PM
    Hey everyone, I'm getting a weird error when trying to run flows with the docker agent against the cloud backend. The error message asks me to make sure I'm authenticated. However, when I run
    prefect auth list-tenants
    I get my tenants, so if feel like I should be authenticated. I'm kind of stumped, maybe someone here has an idea?
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 738, in _request
        json_resp = response.json()
      File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 900, in json
        return complexjson.loads(self.text, **kwargs)
      File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
        return _default_decoder.decode(s)
      File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 53, in flow_run
        result = client.graphql(query)
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 550, in graphql
        retry_on_api_error=retry_on_api_error,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 454, in post
        retry_on_api_error=retry_on_api_error,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 744, in _request
        ) from exc
    prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.
    m
    7 replies · 2 participants
  • y

    YD

    08/23/2021, 4:46 PM
    Teams question If you have several teams, how do you register flows from the same server to a different team? I do not see a
    team
    option in
    flow.register
    . is it when you create the project in the cloud UI? what happens if one team create a project with the same name as a different team ?
    n
    d
    16 replies · 3 participants
  • a

    Alex Furrier

    08/23/2021, 6:34 PM
    Is there a way to assign a task result handler from an imported task? Like the equivalent of this:
    from prefect import task
    from prefect.engine.result.local_result import LocalResult
    from my_prefect_tasks import imported_task
    
    @task(result=LocalResult())
    def assigned_result_task(x):
        return x
    
    imported_task.result_handler = LocalResult()
    k
    3 replies · 2 participants
  • z

    Zach Schumacher

    08/23/2021, 9:07 PM
    when i run flows locally, my logging works fine, but if i run flows in prefect cloud, i get zero logging output. am i doing something blaringly wrong?
    k
    c
    39 replies · 3 participants
  • v

    Vincent

    08/23/2021, 10:00 PM
    Hi all. I am seeing many of our jobs fail with this failure mode
    requests.exceptions.RetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by ResponseError('too many 429 error responses'))
    Ours jobs are quite parallel, and wonder if there is anything that must be resolved, either user or server side that can enable this parallel execution.
    k
    8 replies · 2 participants
  • a

    Alex Furrier

    08/23/2021, 10:37 PM
    Is there a way to extract the number of remaining tasks in a flow from something like the
    prefect.client.client.FlowRunInfoResult
    that's returned by
    client.get_flow_run_info()
    ?
    k
    3 replies · 2 participants
  • w

    Wilson Bilkovich

    08/23/2021, 10:46 PM
    I’m getting
    Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'prefect'")
    when I try to run a flow after making a small tweak to the pod spec. Anyone seen that before?
    k
    7 replies · 2 participants
  • m

    Maria

    08/24/2021, 1:14 AM
    Hi, I have a job configured with ECS agent and flow runs in Fargate. It works fine, but when 10 jobs got triggered simultaneously (externally with lambda), couple of the jobs failed with
    State message: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
    Any advices?
    k
    r
    4 replies · 3 participants
  • j

    Jacob Blanco

    08/24/2021, 6:40 AM
    I’m encountering strange behaviour when trying to invite Team Members to Cloud. I select the role as User, but the menu seems to randomly change to Administrator without me doing anything.
    k
    2 replies · 2 participants
  • r

    Robert Hales

    08/24/2021, 10:36 AM
    Getting some odd behaviour with a task, runs fine and finishes successfully - but is then started again? The task timer duration stops but is printing to logs. Using
    DaskExecutor
    m
    d
    +1
    49 replies · 4 participants
  • l

    Luiz Bezerra de Oliveira Lima Neto

    08/24/2021, 12:09 PM
    💃 Just arrived!
    👋 2
    k
    c
    2 replies · 3 participants
  • c

    Charles Leung

    08/24/2021, 3:11 PM
    Hey Prefect team, I'm having some difficulty understanding the proper arguments to modify/update an action. I am running: mutation { update_action (input:{ action_id: "<Action ID>" name: "CH4 Prefect Agents Down" }) { success } } But in return i get an internal server error; i just want to rename the action specified in the ID • nevermind resolved - just resupplied the config argument
    n
    m
    7 replies · 3 participants
  • w

    Wilson Bilkovich

    08/24/2021, 4:00 PM
    Hmm, now I’m getting
    ERROR - prefect.FlowRunner | Unexpected error: TypeError('code() takes at most 15 arguments (16 given)')
    Is this potentially because I’m running Python 3.9?
    k
    22 replies · 2 participants
  • k

    Kyle Pierce

    08/24/2021, 5:03 PM
    We are upgrading from 13.7. Getting an error when run flows
    Error downloading Flow from S3: No module named 'boto3'
    Used the
    prefecthq/server:core-0.15.3
    image. Is there something for extras or something we are missing?
    k
    19 replies · 2 participants
  • l

    Leon Kozlowski

    08/24/2021, 7:12 PM
    Has anyone had success with a single agent deployment handling multiple environments via labels? (more info in thread)
    k
    16 replies · 2 participants
  • w

    wiretrack

    08/24/2021, 8:50 PM
    Hey all, I was wondering if there’s anyway to use Prefect (I have the services self hosted on a kubernetes cluster) with multiple tenants. I see the
    tenant
    field in the database, but could find a way to make the UI tenant specific, neither to register the flow for a specific tenant. Any ideas?
    n
    7 replies · 2 participants
Powered by Linen
Title
w

wiretrack

08/24/2021, 8:50 PM
Hey all, I was wondering if there’s anyway to use Prefect (I have the services self hosted on a kubernetes cluster) with multiple tenants. I see the
tenant
field in the database, but could find a way to make the UI tenant specific, neither to register the flow for a specific tenant. Any ideas?
n

nicholas

08/24/2021, 9:11 PM
Hi @wiretrack - since Prefect Server doesn't ship with any form of auth, the
tenant
field is a convention only; the UI normally relies on implicit database sharding to filter views instead of adding an explicit
tenant_id
filter to every request
So the answer is that it's not really possible right now
w

wiretrack

08/24/2021, 9:14 PM
Hi @nicholas, thanks, understood. I’ve been looking into the code I though that it was not the case indeed. I’ve seen that
register()
method in the
Flow
class instantiates a
client
object, which accepts
tenant_id
as a parameter. So, excluding the authentication (security) aspect of it, it shouldn’t be a dramatic change to add
tenant_id
as a parameter for the
register
method, and allow for multitenancy. Maybe the UI would need an extra layer to allow to change tenants, but could also have just a
tenant_id
identifier. Far from production ready in the sense of security, but could be a great alternative for separation of concerns.
I was thinking of posting a feature request in the
server
repo to allow multi tenancy, I would be happy to contribute as well, both for the server and the UI
do you think this is a feature you guys would consider adding?
if I’m not mistaken the agent is already prepared to be tenant specific as well, can’t remember now
n

nicholas

08/25/2021, 12:18 AM
In theory I think we'd be willing to accept something like that but I think it may end up being a pretty non-trivial amount of work, particularly in the UI, since every query will need to be adjusted and every component that calls them will need to retrieve and pass it
View count: 4