https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • s

    Stéphan Taljaard

    07/04/2021, 4:22 PM
    Hi. Any pointers for limiting the server postgresql db size? Any specific things to watch out for while truncating the tables?
    k
    • 2
    • 5
  • s

    Sami Serbey

    07/05/2021, 12:31 PM
    Hello everyone, is it correct to run 1 agent for each flow?
    s
    c
    • 3
    • 5
  • b

    Beeshal Rizal

    07/06/2021, 1:24 AM
    Hey team, was wondering if there’s any docs around CI/CD and prefect
    k
    • 2
    • 5
  • b

    Bruno Murino

    07/06/2021, 7:27 AM
    Hi everyone — I just deployed my first flow using Prefect Server and something odd is happening with the schedule.. when creating the flow I’ve set up a schedule of 10 minutes, on the UI I can see the schedule exists, however no runs are actually being scheduled at all, like if the schedule didn’t exist.. has that happened with anyone?
    s
    • 2
    • 2
  • b

    Bruno Murino

    07/06/2021, 8:42 AM
  • b

    Bruno Murino

    07/06/2021, 8:45 AM
    this is how I’m defining my clock, then I attach it to the flow by doing
    flow.schedule = get_schedule()
    import datetime
    from datetime import datetime, timedelta
    from prefect.schedules import clocks, Schedule
    
    
    def get_schedule():
    
        now = datetime.utcnow()
    
        clock1 = clocks.IntervalClock(
            interval=timedelta(minutes=10),
            parameter_defaults={
                "scope": "full",
            },
        )
    
        my_schedule = Schedule(
            clocks=[
                clock1,
            ]
        )
    
        return my_schedule
    ✅ 1
    s
    s
    +2
    • 5
    • 27
  • s

    Scarlett King

    07/06/2021, 4:23 PM
    Hi, may I ask what is the version of python image installed by the Prefect server helm chart ?
    k
    m
    • 3
    • 16
  • m

    Madison Schott

    07/06/2021, 7:06 PM
    Hello! I'm trying to run a fairly simple dbt task but getting this error:
    Task 'DbtShellTask': Starting task run...
    [2021-07-06 15:05:27-0400] ERROR - prefect.DbtShellTask | Command failed with exit code 2
    [2021-07-06 15:05:27-0400] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 2')
    [2021-07-06 15:05:28-0400] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
    [2021-07-06 15:05:28-0400] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
    m
    b
    • 3
    • 7
  • е

    Евгений Берендяев

    07/07/2021, 8:35 AM
    Hi everyone I have deployed the helm chart and in perfect ui I’m receiving this error:
    Cannot connect to Prefect Server at http://localhost:4200/graphql
    This variable is set to https://prefect-apollo.example.com/graphql I’m wondering why the UI still points to stupid localhost
    ✔️ 1
    • 1
    • 2
  • е

    Евгений Берендяев

    07/07/2021, 10:56 AM
    Hi everyone I have deployed prefect-server in k8s using helm chart Now I’m going to deploy an agent on the standalone VM with docker installed. I’ve installed prefect using pip.
    prefect agent docker start --label common --api <https://prefect-apollo.example.com/graphql>
    it says
    No agent API token provided
    according to this API keys can be generated only by using Perfect Cloud, not onpremise version Can I attach docker agent to my onpremise installation?
    👍 1
    ✅ 1
    k
    • 2
    • 2
  • b

    bral

    07/07/2021, 5:22 PM
    Hello! After upgrading from 0.14.21 to 0.15.0 cron schedule like */15 * * * * ( every 15 minutues) does not work. It is showed in flow settings in ui, but there are not upcoming flow runs. If i set schedule manually from ui - it is working. How i can fix it ? Thanks !
    k
    • 2
    • 2
  • r

    Raed

    07/07/2021, 6:56 PM
    Hey guys are there any guides for setting server up on AKS with an external URL and authentication?
    k
    a
    е
    • 4
    • 5
  • c

    Clemens

    07/09/2021, 8:04 AM
    Hi! We are running prefect-server on a Kubernetes cluster, I have the issue that sometimes the resources in our cluster are limited and a flow cannot spawn a pod due to insufficient resources. However what seems to happen is that prefect keeps on spawning new jobs essentially blocking each other. Is there a way to prevent a flow from starting if there is another flow having resource issues? Alternatively, because the issue is with only one flow here, is there a way to prevent a flowrun from starting when there is already another flowrun of the same flow running (or pending)?
    n
    • 2
    • 7
  • е

    Евгений Берендяев

    07/09/2021, 8:24 AM
    Hi again I’m running prefect-server and trying to register a flow on this remote prefect-server. The code of the flow is on my laptop. How can I “point” prefect cli to register the flow on the remote server? i.e. some env variable like
    PERFECT_SERVER_URL
    or
    flow.register()
    param (I’m pretty new in data ops and still thinking in “devops” categories like ci/cd pipelines, sorry 😞 )
    ✅ 2
    k
    • 2
    • 2
  • c

    Chris Bowen

    07/09/2021, 5:11 PM
    Hello, I'm not sure if this is more of a Prefect server question or a UI question, but I'm running Prefect server, so figured I'd start here. I read through the documentation page and the code styling page and I don't completely understand how to utilize the doc strings to auto-generate documentation. I see that I can add a Markdown description to each flow via the UI, but I'd rather keep my documentation in DevOps and do it via code. On the documentation page, I see it says, "_Modules, functions, and classes must be explicitly added to the auto-generated documentation_" and I should reference the "_generated Markdown file_" in the
    TOML
    file. I'm not sure where/what generates the referenced Markdown file though. I put some docstrings into some of my flows and registered them, but I don't see a Markdown file generated anywhere, nor do I see any change on the UI that displays my doc string. Feels like I might be missing something (or another Prefect Docs page). Any advice is greatly appreciated! Thank you.
    z
    • 2
    • 4
  • g

    Goh Rui Zhi

    07/12/2021, 1:16 PM
    Hello, I'm trying to set up prefect server via the helm chart and I need an external db. I 've changed the values in value.yaml to my postgres db instance and deployed it. The pods for apollo, ui and towel seems to be working well, but graphql and hasura has failed. From the logs, I see that my password authentication has failed for my postgres instance. I am fairly certain that the username, password and external hostnames are correct. Why is this happening? Apologies for not providing any screenshots, the database configurations is confidential. Also, I do not have the luxury of running
    prefect server start --postgres-url postgres://<username>:<password>@hostname:<port>/<dbname>
    or similar commands and can only make use of value.yaml to change the configurations. Any help will be appreciated 🙂 Edit: I also realised that the database_url stated in the logs does not have the password field, it is just username@hostname:port/dbname. Is this expected?
    k
    n
    • 3
    • 11
  • s

    Scarlett King

    07/12/2021, 2:06 PM
    Hi, so we are building automated flow register with Azure DevOps. However, it seems that the Prefect version on the build server is 0.14.22 but the Prefect version on Prefect server is 0.15.0. Has anyone got this problem? What do I need to do to upgrade Prefect on build server to 0.15.0?
    k
    • 2
    • 1
  • r

    Raúl Mansilla

    07/12/2021, 9:47 PM
    Hello everyone, I´m trying to use gitlab storage in one test-flow…I stored the gitlab token inside config.toml under
    [context.secrets]
    n
    c
    • 3
    • 11
  • g

    Goh Rui Zhi

    07/13/2021, 2:20 AM
    Hello, I’m experiencing another issue during the deployment of the prefect helm chart.
    Error from server (BadRequest): container “graphql” in pod “prefect-poc-graphql-76462537d4-4v7c6” is waiting to start: PodInitializing
    All other pods seems to be deployed and running well. Anyone knows why this is happening? Also, my UI is not showing up as well, probably due to this issue.
    ✅ 1
    • 1
    • 3
  • е

    Евгений Берендяев

    07/13/2021, 8:03 AM
    Hi guys, trying to register and run a flow on the remote perfect-server. From my laptop I run
    python3 example1.py
    . The .py file mentioned in this command is copied from that page. unfortunately the registered flow does not start and shows me an error
    [13 July 2021 10:52am]: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/Users/mylocaluser/'")
    anyone knows why?
    ✅ 1
    k
    • 2
    • 3
  • z

    Zhilong Li

    07/13/2021, 9:26 AM
    Hi! I'm fairly new to prefect and I wanted to ask about the manual_only trigger for a task. For a task that's waiting for manual approval, does the server expose an API that I can call to send the approval signal? I have my Prefect server up on Kubernetes. Thanks in advance
    n
    • 2
    • 4
  • g

    Goh Rui Zhi

    07/13/2021, 12:59 PM
    Hello everyone, I'm experiencing a problem when connecting my prefect server's Ui. I've defined the
    apolloApiUrl
    in
    values.yaml
    before deploying the helm chart. And indeed I do see the
    settings.json
    reflecting this. However, it seems that
    graphql
    is still being requested from
    <https://localhost:4200/graphql>
    and the user interface is permanently stucked at its loading page. May I know what I did wrong? Also, I am not able to configure my
    config.toml
    as only the deployment agent has access to it, and therefore I can only play around with the chart, values and templates.
    a
    n
    • 3
    • 11
  • a

    Alex Furrier

    07/13/2021, 4:00 PM
    I'm using Prefect server to run a flow that queries a NoSQL database. If the query is large enough I get a 413 payload to large error for graphql. Any suggestions for how to tackle?
    Failed to set task state with error: HTTPError('413 Client Error: Payload Too Large for url: <http://prefect-apollo.prefect:4200/graphql/graphql>')
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
        state = self.client.set_task_run_state(
      File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 1518, in set_task_run_state
        result = self.graphql(
      File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 298, in graphql
        result = <http://self.post|self.post>(
      File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 213, in post
        response = self._request(
      File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 459, in _request
        response = self._send_request(
      File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 375, in _send_request
        response.raise_for_status()
      File "/opt/conda/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 413 Client Error: Payload Too Large for url: <http://prefect-apollo.prefect:4200/graphql/graphql>
    n
    • 2
    • 5
  • n

    Nathan Evans

    07/13/2021, 5:56 PM
    trying to run an ecs agent, getting
    AuthorizationError: No agent API token provided
    my task definition has a PREFECT__CLOUD__API_KEY, and its set to a service account api key. any ideas?
    k
    m
    • 3
    • 9
  • a

    Alex Furrier

    07/13/2021, 8:44 PM
    I'm trying to run a "flow of flows" following the documentation on dependent flows here. Running into an issue when trying to initialize the
    StartFlowRun(flow_name="my registered flow", project_name="my project")
    with a registered flow. I pass the name of the registered flow but when I try to execute the flow of flows I get this error:
    Unexpected error: ValueError("Flow 'my registered flow' not found.")
    To double check the name is correct in the Interactive API I queried for flows and found the flow name I passed. Any idea what I'm doing wrong? Attempting to execute this on k8s and a Dask cluster if that makes a difference.
    n
    k
    • 3
    • 6
  • k

    Kenneth Miller

    07/13/2021, 9:49 PM
    Hey all - question about interval schedules - if I schedule a flow to execute every minute: schedule = IntervalSchedule(interval=timedelta(minutes=1))...but, the flow run takes longer than a minute, will the next flow run wait till the current flow run is complete?
    n
    • 2
    • 1
  • j

    jaehoon

    07/14/2021, 3:10 AM
    Hi, prefect family! My flow is not like this normally, but once in a while, an error 'Too Many File' occurs. The problem is that once it occurs, the openfile buffer of the instance is consumed and a new process is not executed. I want to fundamentally solve this error that may pop up when. Do you know anything about this?
    k
    • 2
    • 3
  • r

    Raúl Mansilla

    07/14/2021, 10:30 AM
    Hello everyone!! I’m trying to register a flow from my Gitlab server as a part of a CI/CD pipeline…but I’m always getting this error:
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f373f0b32b0>: Failed to establish a new connection: [Errno 111] Connection refused'))
    k
    n
    • 3
    • 12
  • a

    Alex Furrier

    07/14/2021, 8:25 PM
    I'm running a mapped task that has many children tasks (~2800) and about 95% of them are sucessfull but at some point I get this error:
    Heartbeat process died with exit code 1
    and the few children tasks left incomplete are unsuccsful. Any ideas how to debug why the parent mapping task is dying?
    k
    n
    • 3
    • 21
  • k

    Kevin Weiler

    07/14/2021, 9:17 PM
    Hi there! I’m trying to set the
    extra_loggers
    config at the level of my flow (as described here) but it’s not having the desired effect (logging stuff from my imported code)
    k
    • 2
    • 7
Powered by Linen
Title
k

Kevin Weiler

07/14/2021, 9:17 PM
Hi there! I’m trying to set the
extra_loggers
config at the level of my flow (as described here) but it’s not having the desired effect (logging stuff from my imported code)
I’m on
0.15.1
this is my flow:
from prefect import task, Flow
from prefect.executors import LocalDaskExecutor
from prefect.storage import Docker
from prefect.run_configs import LocalRun, DockerRun

from research.example_pkg import things

@task()
def run_thing1():
    if things.thing1():
        return True

@task()
def run_thing2():
    if things.thing2():
        return True

if __name__ == "__main__":
    with Flow("test_flow", executor=LocalDaskExecutor(scheduler="processes")) as flow:
        if run_thing1():
            thing2_var = run_thing2()

    flow.run_config = LocalRun(
        env={
            "PREFECT__LOGGING__LEVEL": "DEBUG",
            "PREFECT__LOGGING__EXTRA_LOGGERS": '["research"]'
        }
    )
    flow.run()
running that file doesn’t give me logged output from the imported “things”
but if I export
PREFECT__LOGGING__EXTRA_LOGGERS='["research"]'
prior to running that flow file - it seems to work
is there anyway to set that logging config when creating the flow?
k

Kevin Kho

07/15/2021, 1:59 PM
Hi @Kevin Weiler, is this a custom module? I’m not sure this will help but could you try
"['research']"
instead? double quotes on the outside.
View count: 2