• s

    saml

    9 months ago
    i’m trying to get prefect server running on an ec2 following this tutorial: https://towardsdatascience.com/deploying-prefect-server-with-aws-ecs-fargate-and-docker-storage-36f633226c5f but I can’t seem to get it working. I’ve got server installed, the ecs agent running but when I try to run a task I get this error:
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/ec2-user/'")
    s
    Kevin Kho
    10 replies
    Copy to Clipboard
  • Leon Kozlowski

    Leon Kozlowski

    9 months ago
    I made a one line change to one of my flows and tried to deploy thru my CI job - I’m seeing this message in the
    register
    step
    Registering <flow-name>... Skipped (metadata unchanged)
    What does this check take into account? Must I use the --force command?
    Leon Kozlowski
    1 replies
    Copy to Clipboard
  • alins

    alins

    9 months ago
    hi i want to know how can i run a cron flow parallel ? this is my task
    import datetime
    import os
    import pendulum
    import time
    
    from prefect import Flow, task
    from prefect.schedules import CronSchedule
    from prefect.tasks.airtable import WriteAirtableRow
    from prefect.tasks.github import GetRepoInfo
    from prefect.triggers import any_failed
    
    
    @task
    def my_process():
        time.sleep(70)
    
    daily_schedule = CronSchedule("*/1 * * * *")
    
    
    with Flow("sleep more than cron", schedule=daily_schedule) as flow:
        my_process()
    
    flow.run()
    while i run this with
    python3 flow.py
    this run and wait 70 second till end and the task start again (it takes two minutes run every task) but i want it run every minutes and don't care about the time of the process need to finished
    alins
    Amanda Wee
    4 replies
    Copy to Clipboard
  • Tilak Maddy

    Tilak Maddy

    9 months ago
    Can VertexRun manage to pull an image from private repo on Docker Hub ? If not, how can I accomplish that ?
    flow.run_config = VertexRun(
        image="example/my-custom-private-image:latest",
        machine_type="e2-highmem-8",
    )
    https://docs.prefect.io/orchestration/flow_config/run_configs.html#vertexrun
    Tilak Maddy
    1 replies
    Copy to Clipboard
  • alins

    alins

    9 months ago
    while type
    prefect server start
    it say
    ERROR: Couldn't connect to Docker daemon at <http+docker://localhost> - is it running?
    but the docker is up and running could someone help me please ? the full error
    amir@server:~$ prefect server start
    ERROR: Couldn't connect to Docker daemon at <http+docker://localhost> - is it running?
    
    If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
    Exception caught; killing services (press ctrl-C to force)
    ERROR: Couldn't connect to Docker daemon at <http+docker://localhost> - is it running?
    
    If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
    Traceback (most recent call last):
      File "/home/amir/.local/lib/python3.8/site-packages/prefect/cli/server.py", line 623, in start
        subprocess.check_call(
      File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/amir/.local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
        return self.main(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
        rv = self.invoke(ctx)
      File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
        return callback(*args, **kwargs)
      File "/home/amir/.local/lib/python3.8/site-packages/prefect/cli/server.py", line 664, in start
        subprocess.check_output(
      File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "/usr/lib/python3.8/subprocess.py", line 516, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 1.
    alins
    Anna Geller
    +1
    3 replies
    Copy to Clipboard
  • o

    Oliver Mannion

    9 months ago
    Hiya are there any examples of using the Orion REST API directly? I'm thinking I'd like to use it from non-Python environments to record executions eg: stored procs/Snowflake tasks
    o
    1 replies
    Copy to Clipboard
  • f

    Frank Oplinger

    9 months ago
    Is it possible to access flow parameters from within a state handler? On failure, I’d like to send a notification to a specific user who’s info is passed into the flow as a parameter.
    f
    Tilak Maddy
    +1
    3 replies
    Copy to Clipboard
  • Tilak Maddy

    Tilak Maddy

    9 months ago
    Is run_config, part of the metadata (that get sent when registered to prefect cloud) or is it something that the agent pulls from storage or is it something that must be defined in the executor's environment ?
    Tilak Maddy
    1 replies
    Copy to Clipboard
  • h

    Hugo Shi

    9 months ago
    I seem to be accumulating "concurrent" flows against the concurrency limit I set even though I don't have that many flows running? This seems to happen when an agent dies. Is there a way to clear this out?
    h
    Kevin Kho
    2 replies
    Copy to Clipboard