https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • m

    Marc Lipoff

    12/18/2020, 5:29 PM
    https://github.com/PrefectHQ/prefect/blob/96ef85470872593268c9498b57ac9f0b5a268e01/src/prefect/utilities/filesystems.py#L75
  • m

    Marc Lipoff

    12/18/2020, 5:29 PM
    What's the best way to report a bug? On this line, the key for s3 is wrong. The leading
    /
    needs to be removed.
    n
    1 reply · 2 participants
  • p

    Pedro Machado

    12/21/2020, 12:06 AM
    Hi there. I was under the impression that I the
    prefect server stop
    command had been added. I lost the connection to the server. Is there a graceful way to stop it once it's running? If I wanted to use the docker compose file directly to run server, what is the easiest way to set all the variables that are needed in this file?
    m
    2 replies · 2 participants
  • j

    Joël Luijmes

    12/21/2020, 3:36 PM
    AFAIK the logs are stored in the database right? If I want to move that to a different backend (google bucket or stackdriver), are the logs then still retrievable from the UI? Or are there building blocks to implement that?
    m
    p
    6 replies · 3 participants
  • f

    Frank Cheng

    12/21/2020, 6:59 PM
    Hi! I'm new to Prefect and wish to get some help from experts. We have a POC system using prefect. We just do "prefect server start" to start a local server, and "prefect agent start" on the same server to run the code. Everything works fine for weeks. After some heavy load testing, we started to get error saying "Unexpected error: OSError(28, 'No space left on device') " If we run our code directly using flow.run(), it works fine. But if the same code for flow.register(), it failed with "Unexpected error: OSError(28, 'No space left on device') " We have tried to remove all images and clear docker volumes, but still getting this error. The disk space usage looks OK on that server. Can someone help to shed some light on this issue? Thanks in advance
    m
    11 replies · 2 participants
  • r

    Ron Gross

    12/22/2020, 8:03 AM
    Hi, a question about using
    with case
    and
    upstream_tasks
    together we have a set of 3 tasks that need to run one after the other (no values shared between them)
    A -> B -> C
    we also would like to have the ability to run only:
    A -> C
    that means that
    B
    should go inside
    with case
    , but then what happens is that
    C
    is skipped, because it has
    upstream_tasks
    related to
    B
    any solutions available?
    j
    3 replies · 2 participants
  • k

    Konstantin Tretiakov

    12/22/2020, 8:26 AM
    Hello there. Seems like there is no way to "detach" local 'prefect server start' from terminal out of the box, and making own docker-compose with all env's seems overwhelming since it will have to be checked and updated manually every time prefect updates. Am i missing something, or currently there is no way to actually deploy 'official' deployment of on-premise Prefect Core in any kind of infrastructure (service in VM, deployment in k8s, etc) and only way to use it is through Prefect Cloud?
    a
    2 replies · 2 participants
  • n

    nib

    12/22/2020, 10:17 AM
    Hi, a question about agent. I created 1k flow runs. LocalAgent started to deploy all these flows using popen(prefect execute flow-run). And it created 1k processes. All this processes used all available memory. Is it possible to limit number of deployed flows? I use OSS version. Thanks in advance
    👀 2
    j
    g
    4 replies · 3 participants
  • m

    Massoud Mazar

    12/24/2020, 12:05 AM
    Hi, has anyone had any issues with 0.14.0? I see a lot of things breaking... For example, GitLab storage does not add its default label any more. Also the agent starts with no default label other than the IP address And a bunch of name spaces not found anymore:
    No module named 'prefect.engine.result_handlers'
    on the agent
    m
    1 reply · 2 participants
  • j

    Joël Luijmes

    12/24/2020, 5:13 PM
    Hey, when pinning my helm deployed prefect to 0.14.0, I noticed the tags 0.14.0 and core-0.14.0. As I’m running it myself I probably need the core- version. However, when I used that version for the Kubernetes Agent, it fails to start (see thread for logs).
    👀 1
    c
    m
    +1
    7 replies · 4 participants
  • f

    fblaze f

    12/26/2020, 5:07 PM
    How to use any global object/variable/parameter in task definitions. For example, my flow fails when I declare the client in the global scope in the below code
    #!/usr/bin/python3
    from prefect import task, Flow import boto3 import json client = boto3.client('sagemaker', region_name='us-east-1'                       ) @task def list_training_jobs():     response = client.list_training_jobs()     print(response) with Flow("SageMakerFlow") as flow:     training_jobs = list_training_jobs()     print(training_jobs)      flow.register(project_name="tutorial") flow.run()
    s
    2 replies · 2 participants
  • f

    fblaze f

    12/26/2020, 5:26 PM
    Hi- I'm trying to understand why my runs are not showing up in the run. I have the local agent running and when I run the following code, I don't see the run in the UI
    import prefect
    from prefect import task, Flow
    @task
    def say_hello():
        
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Cloud!")
    with Flow("hello-flow") as flow:
        
    say_hello()
    # Register the flow under the "tutorial" project
    flow.register(project_name="tutorial")
    flow.run()
    a
    2 replies · 2 participants
  • m

    Matthew Blau

    12/28/2020, 2:03 PM
    Hello all, I am trialing out Prefect in my organization and am currently trying to use prefect to run existing docker containers. However, I am not finding documentation/tutorials that show how I can run existing built docker images with a prefect flow. I have prefect locally installed and am using the OSS version of it. I am basically wanting to just run the existing container, as is, with prefect. Any suggestions or documentation that I can look at to assist in this goal?
    a
    2 replies · 2 participants
  • p

    Pedro Machado

    12/31/2020, 3:00 AM
    Hi. We are trying to deploy Prefect Server at a client and their security team noticed that the server loads https://cdn.lr-ingest.io/logger-1.min.js and a js from js.stripe.com. Could you comment on the purpose of these scrips how they can be disabled? We are starting the server with this env var
    PREFECT_SERVER__TELEMETRY__ENABLED=false
    Thanks!
    n
    2 replies · 2 participants
  • f

    fblaze f

    12/31/2020, 5:40 PM
    Hi! Whenever I rerun a flow, it reuses values from the previous run for the parameters declared globally. How should I handle this?
    #!/usr/bin/python3
    from datetime import datetime
    from prefect import task, Flow, Parameter, context, case
    from prefect.utilities.debug import is_serializable
    logger = context.get("logger")
    timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
    @task
    def print_timestamp():
    <http://logger.info|logger.info>(timestamp)
    with Flow("Test") as flow:
        
    print_timestamp()
    serializable = is_serializable(flow)
    if serializable:
        
    flow.register(project_name="Default",
                      
    idempotency_key=flow.serialized_hash())
    flow.run()
    Attached screenshot from two runs. As you could see, the value of the global parameter
    timestamp
    hasn't changed
    d
    7 replies · 2 participants
  • p

    Pedro Martins

    01/04/2021, 12:46 PM
    Are there plans to add prefect-server helm chart to a helm repository?
    m
    m
    +1
    20 replies · 4 participants
  • k

    kevin

    01/04/2021, 9:18 PM
    If I want to run a flow every Saturday, would I use an
    IntervalClock
    or a
    CronClock
    ? Which is more in line with what Prefect expects?
    c
    2 replies · 2 participants
  • j

    jack

    01/06/2021, 4:51 PM
    Using a self-hosted server, how do I create an access token? Using the cloud option, I can click User -> User Settings -> Personal Access Tokens. But with the self-hosted server it doesn't even show a User icon in the top right.
    n
    j
    6 replies · 3 participants
  • a

    Arnaud Fombellida

    01/07/2021, 7:48 AM
    Hey, I'm encountering an issue trying to run the server on a remote machine. The apollo server does not seem to start up properly. After running
    prefect backend server
    , I start the server with
    prefect server start --postgres-port 4301 --hasura-port 4302 --graphql-port 4303 --ui-port 4304 --use-volume --volume-path /home/arnaud/mco/prefect/pgdata
    (Mapping ports as some ports are already in use). In the config.toml, I changed the following:
    [server]
    	[server.ui]
    	apollo_url = "<http://192.168.176.193>:${server.host_port}/graphql"
    After starting the server, all services seem to start properly and
    docker ps
    indicates all services are up (and all ports mapped correctly). However, as you can see below, the are no logs for the apollo server. Running
    prefect server create-tenant --name default --slug default
    results in
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.176.193', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4b14a1dc70>: Failed to establish a new connection: [Errno 111] Connection refused'))
    and I cannot connect to http://192.168.176.193:4200/graphql (nor can the UI). Can anyone help me with this ?
    m
    m
    7 replies · 3 participants
  • f

    Francisco Tanudjaja

    01/07/2021, 8:38 PM
    Hi I'm new to prefect. I'm trying the docker image and ran into this issue
    $docker run -it prefecthq/prefect:latest bash
    
    ... within the container
    root@8afe017ae8ef:/# prefect backend server
    Backend switched to server
    root@8afe017ae8ef:/# prefect server start
    Exception caught; killing services (press ctrl-C to force)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/cli/server.py", line 347, in start
        ["docker-compose", "pull"], cwd=compose_dir_path, env=env
      File "/usr/local/lib/python3.7/subprocess.py", line 358, in check_call
        retcode = call(*popenargs, **kwargs)
      File "/usr/local/lib/python3.7/subprocess.py", line 339, in call
        with Popen(*popenargs, **kwargs) as p:
      File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
        restore_signals, start_new_session)
      File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/cli/server.py", line 385, in start
        ["docker-compose", "down"], cwd=compose_dir_path, env=env
      File "/usr/local/lib/python3.7/subprocess.py", line 411, in check_output
        **kwargs).stdout
      File "/usr/local/lib/python3.7/subprocess.py", line 488, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
        restore_signals, start_new_session)
      File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'
    It's likely I'm missing something basic. Any tips appreciated. Thanks.
    c
    m
    9 replies · 3 participants
  • g

    Greg Roche

    01/08/2021, 11:33 AM
    Hi folks, quick question about data persistence... we are running Prefect Server on an EC2 instance, and we just upgraded from
    0.13.x
    to
    0.14.2
    by running these commands:
    pip install --upgrade prefect
    prefect server stop
    prefect server start --detach
    Unfortunately, after the upgrade had finished all of the projects, registered flows, and flow run history have been lost. Is there any way to prevent this from happening after future upgrades?
    p
    j
    +2
    4 replies · 5 participants
  • k

    kevin

    01/08/2021, 8:38 PM
    Would it be possible to extend
    PostgresFetch
    task to accept an optional boolean arg that indicates whether you want the column names returned with the query or not?
    n
    1 reply · 2 participants
  • a

    Ajith Kumara Beragala Acharige Lal

    01/11/2021, 11:56 AM
    Hi Experts, any idea why this code cannot trigger "Quick run" via Prefect UI? this job stays
    scheduled
    forever without getting triggered, , any idea?
    import prefect
    from prefect.tasks.shell import ShellTask
    from prefect import task, Flow
    
    task = ShellTask(helper_script="cd /prefect")
    
    with Flow("ShellScriptTest") as f:
        mv_file = task(command='cp -rf /prefect/test.py /prefect/kubernetes/')
    
    out = f.register(project_name="ShellScriptTest")
    d
    j
    34 replies · 3 participants
  • j

    Jeremy Phelps

    01/11/2021, 6:38 PM
    Hi, I'm having trouble with Prefect Server. It came up, I was able to register a project in it, and I was also able to register a flow. But when I try to run the flow, I get an error:
    agent@prefect-agent:~/.prefect/results$ prefect run flow --name 'My first Flow' --project test
        Traceback (most recent call last):
          File "/home/agent/.pyenv/versions/3.7.3/bin/prefect", line 10, in <module>
            sys.exit(cli())
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
            return self.main(*args, **kwargs)
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 782, in main
            rv = self.invoke(ctx)
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
            return _process_result(sub_ctx.command.invoke(sub_ctx))
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
            return _process_result(sub_ctx.command.invoke(sub_ctx))
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
            return ctx.invoke(self.callback, **ctx.params)
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
            return callback(*args, **kwargs)
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/cli/run.py", line 186, in flow
            run_name=run_name,
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 1094, in create_flow_run
            res = self.graphql(create_mutation, variables=dict(input=inputs))
          File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 319, in graphql
            raise ClientError(result["errors"])
        prefect.utilities.exceptions.ClientError: [{'message': "Required parameters were not supplied: {'numbers'}", 'locations': [{'line': 2, 'column': 5}], 'path': ['create_flow_run'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "Required parameters were not supplied: {'numbers'}"}}}]
    I checked the Docker logs and found this GraphQL error:
    GraphQL request:2:3
        1 | mutation ($input: create_flow_run_input!) {
        2 |   create_flow_run(input: $input) {
          |   ^
        3 |     id
        Traceback (most recent call last):
          File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result
            return await result
          File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve
            result = await result
          File "/prefect-server/src/prefect_server/graphql/runs.py", line 119, in resolve_create_flow_run
            run_config=input.get("run_config"),
          File "/prefect-server/src/prefect_server/api/runs.py", line 74, in create_flow_run
            run_config=run_config,
          File "/prefect-server/src/prefect_server/api/runs.py", line 215, in _create_flow_run
            raise ValueError(f"Required parameters were not supplied: {missing}")
        ValueError: Required parameters were not supplied: {'numbers'}
    a
    14 replies · 2 participants
  • e

    Eric Sawler

    01/12/2021, 2:49 PM
    Can user tokens be restricted by project? Can runner tokens be restricted by project?
    j
    2 replies · 2 participants
  • j

    Jordan Stewart

    01/12/2021, 3:48 PM
    hey guys, question about using a dockerfile with prefect
    FROM python:3
    ADD hello_world/hello_world.py /~
    RUN pip3 install prefect
    RUN pip3 install numpy
    RUN pip3 install pandas
    CMD python3 /hello_world/hello_world.py
    this is an example, but do the dockerfile's need a CMD line since Prefect Agent is what handles running the code? it seems like we would just need ADD and RUN statements, and then Prefect agent would handle execution
    j
    2 replies · 2 participants
  • r

    Rob Fowler

    01/13/2021, 2:43 AM
    I have timeouts on all my tasks now but if a command in a mapped flow gets locked the process is never stopped and locks up my flow when it tries to finish waiting on all the processes
    j
    m
    2 replies · 3 participants
  • j

    JC Garcia

    01/13/2021, 10:26 PM
    Hey guys! Trying to set up a schedule for one of my flows:
    now = datetime.datetime.utcnow()
    
    dev_clock = clocks.IntervalClock(start_date=now, 
                                    interval=datetime.timedelta(minutes=10), 
                                    parameter_defaults={"some_prop": "someval"})
    schedule = Schedule(clocks=[dev_clock])
    
    flow.schedule = schedule
    and it works! the problem is that it only schedules 10 runs, and no more. Any pointers?
    c
    m
    8 replies · 3 participants
  • j

    Jack Sundberg

    01/14/2021, 6:11 PM
    Hey everyone, it looks like you've done a lot with prefect server! Thanks for the awesome tool. I have a few quick questions about the updates to server/cloud: 1. While the free (dev) version of cloud has a limit of 1 concurrent flow run, is there any limit to concurrency in server? I realize the custom concurrency feature isn't available, but I'm shooting for an unlimited concurrency here. 2. Originally, agents for prefect server had to be on the same device. Is that still the case? Or can I now have any number of agents distributed accross devices & clusters -- just like with cloud? Also shoutout to making Agent requests/signaling one directional (from Agent to Prefect API). This helps soooo much with not needing to deal with firewalls and opening ports on HPC clusters. Thank you! 3. Because I'm still early in my codes development, I need to reset my Postgres database often and then start from scratch for testing. This is pretty easy for server, but what is the easiest way for cloud? The graphql interface ("interactive API")? There used to be an option to reset your data in the user settings that I liked to use.
    👀 1
    c
    p
    9 replies · 3 participants
  • r

    Ruslan

    01/15/2021, 10:09 AM
    Hi! I’m using cloud version. Made simple flow with 10 tasks inside. But the starting one by one. It is cloud restrictions or how to make tasks working in parallel inside flow?
    g
    2 replies · 2 participants
Powered by Linen
Title
r

Ruslan

01/15/2021, 10:09 AM
Hi! I’m using cloud version. Made simple flow with 10 tasks inside. But the starting one by one. It is cloud restrictions or how to make tasks working in parallel inside flow?
g

Greg Roche

01/15/2021, 10:22 AM
The default executor (
LocalExecutor
) doesn't run tasks in parallel. Have you set the executor for your flow to a Dask executor? https://docs.prefect.io/orchestration/tutorial/flow_config.html#enable-parallel-execution If you've done that and the tasks still aren't running in parallel then you probably also need to change your flow's logic, most likely by mapping over the tasks: https://docs.prefect.io/core/concepts/mapping.html
🚀 2
:upvote: 1
👀 1
r

Ruslan

01/15/2021, 10:28 AM
LocalDaskExecutor helped, thank you very much!
👍 1
View count: 1