Hi. I just upgraded my server 0.13.5, and the UI thinks the url is localhost. { "config_overrides": { "server": { "ui": { "appollo_url": true } } }, "env_vars": [], "system_information": { "platform": "Linux-4.14.193-149.317.amzn2.x86_64-x86_64-with-glibc2.10", "prefect_version": "0.13.5", "python_version": "3.7.3" } }

Hi friends, I'm quite sure I found a discussion on deploying prefect server on ECS (not directly on EC2, but as containers in ECS), including a sample config of some sort, but it seems to have disappeared probably due to the free Slack limit. Would anyone who has successfully deployed prefect server on ECS like to lend a hand? I'm taking reference from the

docker-compose.yml

for

prefect server start

to create a task definition for

hasura

,

graphql

,

towel

, and

apollo

containers on a single ECS service, with postgres on an aurora cluster and prefect-ui on another ECS service. Unfortunately, the hasura container keeps stopping and we aren't getting logs, so I'd like to compare with someone who has a working setup.

Hello everyone. I've been using Prefect Server for my work for the recent 2 months. So, I'd like to get Prefect-UI operation logs, such as flow run actions. How do I check that these logs? It seems to output the logs in several containers, but I don't get to know specific actions from these logs.

Hi everyone, can anyone please help me with getting loggs from Kubernetes

RunNamespacedJob

to TASK RUN window in the UI? So far I'm getting just:

12:21:32
INFO
RunNamespacedJob
Job test-app has been created.
12:21:37
INFO
RunNamespacedJob
Job test-app has been completed.
12:21:37
INFO
RunNamespacedJob
Job test-app has been deleted.

but I'd also like to see what's going on INSIDE the job itself. Thank you in advance! 🙂👍

Hi! We are interested in running multiple types of Flows via the Prefect Server using the Kubernetes Agent. We understand this requires using

Docker

storage for each job. We construct the images ourselves and may have many flows. We want to avoid constructing a unique image for each specific Flow, and we also want to avoid creating an image that contains every flow. Instead, we want to provide a base image that the Kubernetes Agent can run, and subsequently the launched Job can ask for the necessary Flow at runtime. Ultimately, we want to avoid serializing the Flow within the built image and still use the Kubernetes Agent. A thought we had was to build an image whose entrypoint would query for the Flow, place the serialized Flow in the appropriate file location and start Prefect. Looking for any thoughts on how to best use k8s agent with these constraints! Thanks

Hi everyone, I would like to use Prefect with python-trio and Im willing to develop the necessary integrations. I found some discussion on this topic and according to them, the best option is to add support for AnyIO. I think I can make progress on this with a little support. I guess a first step is to look at adapting the existing code for Twisted and Asyncio?

Hi Prefecters, ran into a spot of bother while trying out the spectacular new helm chart in the server repo. I am deploying it to our Azure Kubernetes Service with an Azure Managed PostgreSQL instance as the backing database. Everything went perfectly fine with the Hasura deployment, but the graphql service wouldn't start. I had a look at the logs and it was the

db-upgrade

initcontainer that had failed and stopped the deployment. The error mentioned an invalid character in the

PREFECT_SERVER__DATABASE__CONNECTION_URL

and that lead me to the fix.

Azure Managed Postgres hands out user names in the form of;

username@my-unique-instance-name

, so most often you actually need to URL encode the username to avoid having two

@

symbols in your connection string, e.g.

username%40my-unique-instance-name

. This is what I did and what the

db-upgrade

container was upset about. I hardcoded the full connection string in the helm template with the two

@

symbols and it worked straight away.

I suspect there may be an issue where Hasura would prefer URL encoding and

db-upgrade

prefer there would be an extra

@

symbol.

Hi all, Had a quick question trying to better understand the difference/relationship between the object that we store in as a

.prefect

file in storage when submitting a flow to the server, versus the serialized variant of a flow that is used to register with the GraphQL API? To motivate the reasoning, we have a need to directly invoke the GraphQL API, and are therefore manually serializing flows ourselves. As per our current understanding, we also need to build the

.prefect

file in storage and make sure that is referenced in the serialization, but given the fact that the

tasks

are captured in the serialized flow anyway, we were curious as to what the

.prefect

file is used for, what other information is captured there? Mainly trying to understand why, rather than perhaps build a work around of any kind. Thanks in advance!

Hi all, Is there a way to ask prefect server what the code in a particular flow is? Currently if I register a flow and then register it again with the exact same code it will increment the version number even though the flow is identical. I can write a system to store a local hashed version of the file and then compare before I re-register, but I wanted to check if you guys had a better solution before I do that! Thanks 🙂

Hi everyone! Trying to get started with prefect 😄 managed to create a

user-data.sh

script that gets prefect running in an AWS EC2 instance with amazon linux 2 as the AMI of the machine. Might be useful for someone but please run at your own discretion!

Hi folks, is there a way to set the expected type for a Parameter? I have a parameter which is a string, but when passing it to tasks, Pycharm raises a warning about type mismatch…

Hi all, Anyone knows where to find the path to the prefect executable?

Q: Canceling a Flow does not cancel its child Flows that were started with

StartFlowRun

. Is this by design? (cancellation was done through the UI).

Wondering if any current Prefect maintainer could chime in on this: https://prefect-community.slack.com/archives/CL09KU1K7/p1592313602391000

Getting the following error when attempting to run my Flow

Failed to load and execute Flow's environment: KeyError('my-flow-name')

The Flow is uses DockerStorage, and I’ve got an image containing only this Flow; are there some

prefect

cli invocations I can do inside the container to debug this KeyError?

Hi All - I’m trying to debug an issue in my new Prefect prod environment. Basically, I’m able to start all prefect services just fine, including a local agent. Logs are humming along and everything looks to be communicating successfully. However, the UI states that no agents are querying for flow runs. Additionally, if I create a project or register flows from the backend, the UI shows no new project or registered flows, despite successful messaging in the console. I can create a project in the frontend, which persists, but no entry appears in the database. It seems that the UI isn’t reading from / writing to the postgres database at all, which I confirmed by querying the respective tables. TLDR: my Prefect UI is persisting data in GraphQL (must be, right?) but not reading / writing to postgres. My setup: • Single AWS EC2 instance • Prefect server backend • Prefect version 0.13.15 • Latest docker images, as of now (initialize via prefect server start, for now) Any advice on how I should continue my investigation would be greatly appreciated!

Hi all - I have a bit of an open-ended question, but would it be acceptable security-wise to store all our secrets into GoogleSecretManager and then have a prefect task read in the secrets and then pass in those secrets to subsequent tasks in the flow?

If I have an

AWS_CREDENTIALS

secret in Prefect Cloud, should S3Result and S3Download use those credentials automatically, unless I specify something different for the

boto_kwargs

argument?

Hello. I’d like to raise the issue of changes in recent prefect versions, that break our deployment. It seems that if Prefect core + server + docker image versions aren’t synchronised to the exact patch version (e.g 0.13.13 might not work with 0.13.17), things might break down. Usually “patch” versions should not include any breaking change. In order for us to make sure our production flows won’t break, we have to make sure that we use specific version across the board. fetching the latest docker image or the latest prefect core version is not something one should do in production, unless one makes sure to also update his Prefect Server (which might not be possible or required in production environments. attached is the latest error we received out of the blue, but this is not the first time we experience a breaking change.

Unexpected error: ClientError('400 Client Error: Bad Request for url: <http://10.0.4.45:4200/graphql>\n\nThis is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:\n\nquery {\n    mutation($input: create_task_run_artifact_input!) {\n            create_task_run_artifact(input: $input) {\n                id\n        }\n    }\n}\nvariables {\n    {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}\n}')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 856, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 298, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/tasks.py", line 449, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 173, in run
    create_link(urlparse(run_link).path)
  File "/usr/local/lib/python3.8/site-packages/prefect/artifacts.py", line 31, in create_link
    return client.create_task_run_artifact(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1689, in create_task_run_artifact
    result = self.graphql(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 296, in graphql
    result = <http://self.post|self.post>(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 211, in post
    response = self._request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 433, in _request
    response = self._send_request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 365, in _send_request
    raise ClientError(msg)
prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <http://10.0.4.45:4200/graphql>

This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
    mutation($input: create_task_run_artifact_input!) {
            create_task_run_artifact(input: $input) {
                id
        }
    }
}
variables {
    {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}
}

Hi folks, is it possible to update the description of an existing project in Prefect Server using Prefect Client. I see there’s a

create_project

API, but nothing related to update

Hi All - quick question on restarting failed flows. I’m attempting to restart a flow by 1) setting all tasks to

Scheduled

via the UI and 2) rerunning the entire flow. However, the first step fails with the attached UI error. I see no errors in the Chrome console, nor anything in the service logs. This only appears to happen when I select the

Change the state of all task runs too

box. I’m running Prefect Server with the latest containers. Any tips? Thanks!

Hi! I wonder how come that BigQueryTask have no ability to return

query_job

object. I mean, if one have an idea to log the results of the query (status, proceeded_bytes, etc.), he or she can not get an object to fetch that info. I mean something like that (tasks/gcp/bigquery.py; class BigQueryTask; def run):

Hi All, quick question, is there any quick way to remove all the metadata and log information from Postgres db after 30 days? Thanks

did anyone encounter issues with Fargate tasks (flows) loosing connection to Prefect Server after a while (running for a few hours)? this is the error we’re seeing.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.4.45', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0de03bb820>, 'Connection to 10.0.4.45 timed out. (connect timeout=30)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/flow_runner.py", line 179, in interrupt_if_cancelling
    flow_run_info = self.client.get_flow_run_info(flow_run_id)
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1015, in get_flow_run_info
    result = self.graphql(query).data.flow_run_by_pk  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 275, in graphql
    result = <http://self.post|self.post>(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 230, in post
    response = self._request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 400, in _request
    response = self._send_request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in _send_request
    response = <http://session.post|session.post>(url, headers=headers, json=params, timeout=30)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='10.0.4.45', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0de03bb820>, 'Connection to 10.0.4.45 timed out. (connect timeout=30)'))

Hello! I've been trying out the Kubernetes Agent configuration and noticed it was getting killed after some time with failing health checks I've noticed the Agent is listening on

localhost

, so it wouldn't be reachable on it's network interface - and changing it to 0.0.0.0 makes health checks pass, are there any security implications for this or it's a good default? https://github.com/PrefectHQ/server/blob/a3626c81f280bf749394c2d00d15d6450b724a44/helm/prefect-server/templates/agent/deployment.yaml#L65

Hello! Let's say I wanted to run multiple dbt commands through the dbt task, but will not know the commands until runtime (they will be received as a paramter). What is the recommended way of looping tasks? Would something like this be ok?

dbt_task = DbtShellTask(
  ...
)

with Flow("dbt-flow") as flow:

    dbt_command_string = Parameter('dbt_command_string', default = 'dbt run', required = True)

    dbt_commands = dbt_command_string.split(',')


    for command in dbt_commands:
      dbt_task(
          command=dbt_command,
          ...
      )

Another question, is it possible to have prefect's cli run to exit with an error if any of the tasks error out while watching the state or streaming the logs (-w, -l)?

Hi there. I am wanting to run a flow on prefect server with a custom schedule. The schedule is set with an interval clock that runs everyday and generates a clock for roughly 4000 parameters.

schedule = Schedule(
    clocks=[
        IntervalClock(
            start_date=pendulum.datetime(2020, 12, 1, 16, 20, 0, tz="Australia/Sydney"),
            interval=timedelta(days=1),
            parameter_defaults=param
        )
        for param in parameters
    ]
)

However when I deploy the flow to prefect server it runs 10 out of my 4000 odd params and schedules another 10 for tomorrow. Is there a limit to how many Interval clocks I can use with the prefect server schedule or potentially some other reason why it isn't queuing the full set of params to be run? Cheers, Phil