https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • c

    Carlo

    11/11/2020, 9:01 PM
    Hi. I just upgraded my server 0.13.5, and the UI thinks the url is localhost. { "config_overrides": { "server": { "ui": { "appollo_url": true } } }, "env_vars": [], "system_information": { "platform": "Linux-4.14.193-149.317.amzn2.x86_64-x86_64-with-glibc2.10", "prefect_version": "0.13.5", "python_version": "3.7.3" } }
    n
    m
    12 replies · 3 participants
  • a

    Amanda Wee

    11/11/2020, 10:57 PM
    Hi friends, I'm quite sure I found a discussion on deploying prefect server on ECS (not directly on EC2, but as containers in ECS), including a sample config of some sort, but it seems to have disappeared probably due to the free Slack limit. Would anyone who has successfully deployed prefect server on ECS like to lend a hand? I'm taking reference from the
    docker-compose.yml
    for
    prefect server start
    to create a task definition for
    hasura
    ,
    graphql
    ,
    towel
    , and
    apollo
    containers on a single ECS service, with postgres on an aurora cluster and prefect-ui on another ECS service. Unfortunately, the hasura container keeps stopping and we aren't getting logs, so I'd like to compare with someone who has a working setup.
    ➕ 1
  • s

    Satoru Kadowaki

    11/12/2020, 9:13 AM
    Hello everyone. I've been using Prefect Server for my work for the recent 2 months. So, I'd like to get Prefect-UI operation logs, such as flow run actions. How do I check that these logs? It seems to output the logs in several containers, but I don't get to know specific actions from these logs.
    d
    5 replies · 2 participants
  • j

    Josef Trefil

    11/12/2020, 11:36 AM
    Hi everyone, can anyone please help me with getting loggs from Kubernetes
    RunNamespacedJob
    to TASK RUN window in the UI? So far I'm getting just:
    12:21:32
    INFO
    RunNamespacedJob
    Job test-app has been created.
    12:21:37
    INFO
    RunNamespacedJob
    Job test-app has been completed.
    12:21:37
    INFO
    RunNamespacedJob
    Job test-app has been deleted.
    but I'd also like to see what's going on INSIDE the job itself. Thank you in advance! 🙂👍
    d
    j
    9 replies · 3 participants
  • d

    David Harrington

    11/12/2020, 5:07 PM
    Hi! We are interested in running multiple types of Flows via the Prefect Server using the Kubernetes Agent. We understand this requires using
    Docker
    storage for each job. We construct the images ourselves and may have many flows. We want to avoid constructing a unique image for each specific Flow, and we also want to avoid creating an image that contains every flow. Instead, we want to provide a base image that the Kubernetes Agent can run, and subsequently the launched Job can ask for the necessary Flow at runtime. Ultimately, we want to avoid serializing the Flow within the built image and still use the Kubernetes Agent. A thought we had was to build an image whose entrypoint would query for the Flow, place the serialized Flow in the appropriate file location and start Prefect. Looking for any thoughts on how to best use k8s agent with these constraints! Thanks
    d
    4 replies · 2 participants
  • r

    Ross Rochford

    11/14/2020, 5:37 AM
    Hi everyone, I would like to use Prefect with python-trio and Im willing to develop the necessary integrations. I found some discussion on this topic and according to them, the best option is to add support for AnyIO. I think I can make progress on this with a little support. I guess a first step is to look at adapting the existing code for Twisted and Asyncio?
    m
    j
    +3
    17 replies · 6 participants
  • a

    Aiden Price

    11/17/2020, 9:30 AM
    Hi Prefecters, ran into a spot of bother while trying out the spectacular new helm chart in the server repo. I am deploying it to our Azure Kubernetes Service with an Azure Managed PostgreSQL instance as the backing database. Everything went perfectly fine with the Hasura deployment, but the graphql service wouldn't start. I had a look at the logs and it was the
    db-upgrade
    initcontainer that had failed and stopped the deployment. The error mentioned an invalid character in the
    PREFECT_SERVER__DATABASE__CONNECTION_URL
    and that lead me to the fix.
  • a

    Aiden Price

    11/17/2020, 9:33 AM
    Azure Managed Postgres hands out user names in the form of;
    username@my-unique-instance-name
    , so most often you actually need to URL encode the username to avoid having two
    @
    symbols in your connection string, e.g.
    username%40my-unique-instance-name
    . This is what I did and what the
    db-upgrade
    container was upset about. I hardcoded the full connection string in the helm template with the two
    @
    symbols and it worked straight away.
  • a

    Aiden Price

    11/17/2020, 9:34 AM
    I suspect there may be an issue where Hasura would prefer URL encoding and
    db-upgrade
    prefer there would be an extra
    @
    symbol.
    m
    15 replies · 2 participants
  • a

    Adithya Ramanathan

    11/17/2020, 3:07 PM
    Hi all, Had a quick question trying to better understand the difference/relationship between the object that we store in as a
    .prefect
    file in storage when submitting a flow to the server, versus the serialized variant of a flow that is used to register with the GraphQL API? To motivate the reasoning, we have a need to directly invoke the GraphQL API, and are therefore manually serializing flows ourselves. As per our current understanding, we also need to build the
    .prefect
    file in storage and make sure that is referenced in the serialization, but given the fact that the
    tasks
    are captured in the serialized flow anyway, we were curious as to what the
    .prefect
    file is used for, what other information is captured there? Mainly trying to understand why, rather than perhaps build a work around of any kind. Thanks in advance!
    j
    11 replies · 2 participants
  • e

    Elliot Oram

    11/18/2020, 6:17 PM
    Hi all, Is there a way to ask prefect server what the code in a particular flow is? Currently if I register a flow and then register it again with the exact same code it will increment the version number even though the flow is identical. I can write a system to store a local hashed version of the file and then compare before I re-register, but I wanted to check if you guys had a better solution before I do that! Thanks 🙂
    m
    d
    8 replies · 3 participants
  • j

    Julio Venegas

    11/18/2020, 9:25 PM
    Hi everyone! Trying to get started with prefect 😄 managed to create a
    user-data.sh
    script that gets prefect running in an AWS EC2 instance with amazon linux 2 as the AMI of the machine. Might be useful for someone but please run at your own discretion!
    user-data-prefect.sh
    👀 2
  • a

    ale

    11/19/2020, 12:54 PM
    Hi folks, is there a way to set the expected type for a Parameter? I have a parameter which is a string, but when passing it to tasks, Pycharm raises a warning about type mismatch…
    d
    s
    8 replies · 3 participants
  • a

    Alberto de Santos

    11/20/2020, 12:18 AM
    Hi all, Anyone knows where to find the path to the prefect executable?
    d
    m
    4 replies · 3 participants
  • r

    Roey Brecher

    11/20/2020, 6:37 AM
    Q: Canceling a Flow does not cancel its child Flows that were started with
    StartFlowRun
    . Is this by design? (cancellation was done through the UI).
    n
    1 reply · 2 participants
  • r

    Roey Brecher

    11/20/2020, 7:05 AM
    Wondering if any current Prefect maintainer could chime in on this: https://prefect-community.slack.com/archives/CL09KU1K7/p1592313602391000
    n
    a
    3 replies · 3 participants
  • j

    Joseph Haaga

    11/20/2020, 5:31 PM
    Getting the following error when attempting to run my Flow
    Failed to load and execute Flow's environment: KeyError('my-flow-name')
    The Flow is uses DockerStorage, and I’ve got an image containing only this Flow; are there some
    prefect
    cli invocations I can do inside the container to debug this KeyError?
    n
    9 replies · 2 participants
  • m

    Mike Marinaccio

    11/23/2020, 12:15 AM
    Hi All - I’m trying to debug an issue in my new Prefect prod environment. Basically, I’m able to start all prefect services just fine, including a local agent. Logs are humming along and everything looks to be communicating successfully. However, the UI states that no agents are querying for flow runs. Additionally, if I create a project or register flows from the backend, the UI shows no new project or registered flows, despite successful messaging in the console. I can create a project in the frontend, which persists, but no entry appears in the database. It seems that the UI isn’t reading from / writing to the postgres database at all, which I confirmed by querying the respective tables. TLDR: my Prefect UI is persisting data in GraphQL (must be, right?) but not reading / writing to postgres. My setup: • Single AWS EC2 instance • Prefect server backend • Prefect version 0.13.15 • Latest docker images, as of now (initialize via prefect server start, for now) Any advice on how I should continue my investigation would be greatly appreciated!
    n
    2 replies · 2 participants
  • r

    Riley Hun

    11/23/2020, 5:44 PM
    Hi all - I have a bit of an open-ended question, but would it be acceptable security-wise to store all our secrets into GoogleSecretManager and then have a prefect task read in the secrets and then pass in those secrets to subsequent tasks in the flow?
    k
    1 reply · 2 participants
  • l

    Luke Orland

    11/23/2020, 9:40 PM
    If I have an
    AWS_CREDENTIALS
    secret in Prefect Cloud, should S3Result and S3Download use those credentials automatically, unless I specify something different for the
    boto_kwargs
    argument?
    k
    s
    13 replies · 3 participants
  • r

    Roey Brecher

    11/25/2020, 2:19 PM
    Hello. I’d like to raise the issue of changes in recent prefect versions, that break our deployment. It seems that if Prefect core + server + docker image versions aren’t synchronised to the exact patch version (e.g 0.13.13 might not work with 0.13.17), things might break down. Usually “patch” versions should not include any breaking change. In order for us to make sure our production flows won’t break, we have to make sure that we use specific version across the board. fetching the latest docker image or the latest prefect core version is not something one should do in production, unless one makes sure to also update his Prefect Server (which might not be possible or required in production environments. attached is the latest error we received out of the blue, but this is not the first time we experience a breaking change.
    Unexpected error: ClientError('400 Client Error: Bad Request for url: <http://10.0.4.45:4200/graphql>\n\nThis is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:\n\nquery {\n    mutation($input: create_task_run_artifact_input!) {\n            create_task_run_artifact(input: $input) {\n                id\n        }\n    }\n}\nvariables {\n    {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}\n}')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 856, in get_task_run_state
        value = prefect.utilities.executors.run_task_with_timeout(
      File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 298, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "/usr/local/lib/python3.8/site-packages/prefect/utilities/tasks.py", line 449, in method
        return run_method(self, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 173, in run
        create_link(urlparse(run_link).path)
      File "/usr/local/lib/python3.8/site-packages/prefect/artifacts.py", line 31, in create_link
        return client.create_task_run_artifact(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1689, in create_task_run_artifact
        result = self.graphql(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 296, in graphql
        result = <http://self.post|self.post>(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 211, in post
        response = self._request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 433, in _request
        response = self._send_request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 365, in _send_request
        raise ClientError(msg)
    prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <http://10.0.4.45:4200/graphql>
    
    This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:
    
    query {
        mutation($input: create_task_run_artifact_input!) {
                create_task_run_artifact(input: $input) {
                    id
            }
        }
    }
    variables {
        {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}
    }
    j
    4 replies · 2 participants
  • a

    ale

    11/25/2020, 5:29 PM
    Hi folks, is it possible to update the description of an existing project in Prefect Server using Prefect Client. I see there’s a
    create_project
    API, but nothing related to update
    j
    j
    6 replies · 3 participants
  • m

    Mike Marinaccio

    11/25/2020, 5:45 PM
    Hi All - quick question on restarting failed flows. I’m attempting to restart a flow by 1) setting all tasks to
    Scheduled
    via the UI and 2) rerunning the entire flow. However, the first step fails with the attached UI error. I see no errors in the Chrome console, nor anything in the service logs. This only appears to happen when I select the
    Change the state of all task runs too
    box. I’m running Prefect Server with the latest containers. Any tips? Thanks!
    j
    7 replies · 2 participants
  • i

    Innokenty Tretyakov

    11/26/2020, 11:10 AM
    Hi! I wonder how come that BigQueryTask have no ability to return
    query_job
    object. I mean, if one have an idea to log the results of the query (status, proceeded_bytes, etc.), he or she can not get an object to fetch that info. I mean something like that (tasks/gcp/bigquery.py; class BigQueryTask; def run):
  • v

    Vipul

    11/27/2020, 9:19 AM
    Hi All, quick question, is there any quick way to remove all the metadata and log information from Postgres db after 30 days? Thanks
    j
    4 replies · 2 participants
  • r

    Roey Brecher

    11/27/2020, 4:05 PM
    did anyone encounter issues with Fargate tasks (flows) loosing connection to Prefect Server after a while (running for a few hours)? this is the error we’re seeing.
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
        resp = conn.urlopen(
      File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
        retries = retries.increment(
      File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 446, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.4.45', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0de03bb820>, 'Connection to 10.0.4.45 timed out. (connect timeout=30)'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/flow_runner.py", line 179, in interrupt_if_cancelling
        flow_run_info = self.client.get_flow_run_info(flow_run_id)
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1015, in get_flow_run_info
        result = self.graphql(query).data.flow_run_by_pk  # type: ignore
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 275, in graphql
        result = <http://self.post|self.post>(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 230, in post
        response = self._request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 400, in _request
        response = self._send_request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in _send_request
        response = <http://session.post|session.post>(url, headers=headers, json=params, timeout=30)
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
        return self.request('POST', url, data=data, json=json, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 504, in send
        raise ConnectTimeout(e, request=request)
    requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='10.0.4.45', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0de03bb820>, 'Connection to 10.0.4.45 timed out. (connect timeout=30)'))
  • v

    Vladimir Zoubritsky

    11/29/2020, 11:04 AM
    Hello! I've been trying out the Kubernetes Agent configuration and noticed it was getting killed after some time with failing health checks I've noticed the Agent is listening on
    localhost
    , so it wouldn't be reachable on it's network interface - and changing it to 0.0.0.0 makes health checks pass, are there any security implications for this or it's a good default? https://github.com/PrefectHQ/server/blob/a3626c81f280bf749394c2d00d15d6450b724a44/helm/prefect-server/templates/agent/deployment.yaml#L65
    m
    2 replies · 2 participants
  • j

    JC Garcia

    11/30/2020, 3:30 PM
    Hello! Let's say I wanted to run multiple dbt commands through the dbt task, but will not know the commands until runtime (they will be received as a paramter). What is the recommended way of looping tasks? Would something like this be ok?
    dbt_task = DbtShellTask(
      ...
    )
    
    with Flow("dbt-flow") as flow:
    
        dbt_command_string = Parameter('dbt_command_string', default = 'dbt run', required = True)
    
        dbt_commands = dbt_command_string.split(',')
    
    
        for command in dbt_commands:
          dbt_task(
              command=dbt_command,
              ...
          )
    m
    m
    9 replies · 3 participants
  • j

    JC Garcia

    11/30/2020, 3:40 PM
    Another question, is it possible to have prefect's cli run to exit with an error if any of the tasks error out while watching the state or streaming the logs (-w, -l)?
    j
    3 replies · 2 participants
  • p

    Philip Evans

    12/01/2020, 6:05 AM
    Hi there. I am wanting to run a flow on prefect server with a custom schedule. The schedule is set with an interval clock that runs everyday and generates a clock for roughly 4000 parameters.
    schedule = Schedule(
        clocks=[
            IntervalClock(
                start_date=pendulum.datetime(2020, 12, 1, 16, 20, 0, tz="Australia/Sydney"),
                interval=timedelta(days=1),
                parameter_defaults=param
            )
            for param in parameters
        ]
    )
    However when I deploy the flow to prefect server it runs 10 out of my 4000 odd params and schedules another 10 for tomorrow. Is there a limit to how many Interval clocks I can use with the prefect server schedule or potentially some other reason why it isn't queuing the full set of params to be run? Cheers, Phil
    j
    3 replies · 2 participants
Powered by Linen
Title
p

Philip Evans

12/01/2020, 6:05 AM
Hi there. I am wanting to run a flow on prefect server with a custom schedule. The schedule is set with an interval clock that runs everyday and generates a clock for roughly 4000 parameters.
schedule = Schedule(
    clocks=[
        IntervalClock(
            start_date=pendulum.datetime(2020, 12, 1, 16, 20, 0, tz="Australia/Sydney"),
            interval=timedelta(days=1),
            parameter_defaults=param
        )
        for param in parameters
    ]
)
However when I deploy the flow to prefect server it runs 10 out of my 4000 odd params and schedules another 10 for tomorrow. Is there a limit to how many Interval clocks I can use with the prefect server schedule or potentially some other reason why it isn't queuing the full set of params to be run? Cheers, Phil
j

Jenny

12/01/2020, 3:43 PM
Hi @Philip Evans - Prefect Scheduler runs at a ten minute interval and schedules 10 runs at a time: https://docs.prefect.io/orchestration/concepts/services.html#scheduler For what you're trying to do, you could try mapping through those params in your flow: https://docs.prefect.io/core/concepts/mapping.html#simple-mapping or if you want separate flow runs, you might use the startFlowRunTask: https://docs.prefect.io/api/latest/tasks/prefect.html#startflowrun
p

Philip Evans

12/01/2020, 10:10 PM
Thanks @Jenny! I think your final suggestion will work perfectly. I will give that a go. 👍
j

Jenny

12/01/2020, 10:10 PM
👍
View count: 1