Carlo
11/11/2020, 9:01 PMAmanda Wee
11/11/2020, 10:57 PMdocker-compose.yml
for prefect server start
to create a task definition for hasura
, graphql
, towel
, and apollo
containers on a single ECS service, with postgres on an aurora cluster and prefect-ui on another ECS service. Unfortunately, the hasura container keeps stopping and we aren't getting logs, so I'd like to compare with someone who has a working setup.Satoru Kadowaki
11/12/2020, 9:13 AMJosef Trefil
11/12/2020, 11:36 AMRunNamespacedJob
to TASK RUN window in the UI?
So far I'm getting just:
12:21:32
INFO
RunNamespacedJob
Job test-app has been created.
12:21:37
INFO
RunNamespacedJob
Job test-app has been completed.
12:21:37
INFO
RunNamespacedJob
Job test-app has been deleted.
but I'd also like to see what's going on INSIDE the job itself.
Thank you in advance! 🙂👍David Harrington
11/12/2020, 5:07 PMDocker
storage for each job.
We construct the images ourselves and may have many flows. We want to avoid constructing a unique image for each specific Flow, and we also want to avoid creating an image that contains every flow. Instead, we want to provide a base image that the Kubernetes Agent can run, and subsequently the launched Job can ask for the necessary Flow at runtime.
Ultimately, we want to avoid serializing the Flow within the built image and still use the Kubernetes Agent.
A thought we had was to build an image whose entrypoint would query for the Flow, place the serialized Flow in the appropriate file location and start Prefect.
Looking for any thoughts on how to best use k8s agent with these constraints! ThanksRoss Rochford
11/14/2020, 5:37 AMAiden Price
11/17/2020, 9:30 AMdb-upgrade
initcontainer that had failed and stopped the deployment. The error mentioned an invalid character in the PREFECT_SERVER__DATABASE__CONNECTION_URL
and that lead me to the fix.Aiden Price
11/17/2020, 9:33 AMusername@my-unique-instance-name
, so most often you actually need to URL encode the username to avoid having two @
symbols in your connection string, e.g. username%40my-unique-instance-name
. This is what I did and what the db-upgrade
container was upset about. I hardcoded the full connection string in the helm template with the two @
symbols and it worked straight away.Aiden Price
11/17/2020, 9:34 AMdb-upgrade
prefer there would be an extra @
symbol.Adithya Ramanathan
11/17/2020, 3:07 PM.prefect
file in storage when submitting a flow to the server, versus the serialized variant of a flow that is used to register with the GraphQL API?
To motivate the reasoning, we have a need to directly invoke the GraphQL API, and are therefore manually serializing flows ourselves. As per our current understanding, we also need to build the .prefect
file in storage and make sure that is referenced in the serialization, but given the fact that the tasks
are captured in the serialized flow anyway, we were curious as to what the .prefect
file is used for, what other information is captured there? Mainly trying to understand why, rather than perhaps build a work around of any kind.
Thanks in advance!Elliot Oram
11/18/2020, 6:17 PMJulio Venegas
11/18/2020, 9:25 PMuser-data.sh
script that gets prefect running in an AWS EC2 instance with amazon linux 2 as the AMI of the machine. Might be useful for someone but please run at your own discretion!ale
11/19/2020, 12:54 PMAlberto de Santos
11/20/2020, 12:18 AMRoey Brecher
11/20/2020, 6:37 AMStartFlowRun
. Is this by design? (cancellation was done through the UI).Roey Brecher
11/20/2020, 7:05 AMJoseph Haaga
11/20/2020, 5:31 PMFailed to load and execute Flow's environment: KeyError('my-flow-name')
The Flow is uses DockerStorage, and I’ve got an image containing only this Flow; are there some prefect
cli invocations I can do inside the container to debug this KeyError?Mike Marinaccio
11/23/2020, 12:15 AMRiley Hun
11/23/2020, 5:44 PMLuke Orland
11/23/2020, 9:40 PMAWS_CREDENTIALS
secret in Prefect Cloud, should S3Result and S3Download use those credentials automatically, unless I specify something different for the boto_kwargs
argument?Roey Brecher
11/25/2020, 2:19 PMUnexpected error: ClientError('400 Client Error: Bad Request for url: <http://10.0.4.45:4200/graphql>\n\nThis is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:\n\nquery {\n mutation($input: create_task_run_artifact_input!) {\n create_task_run_artifact(input: $input) {\n id\n }\n }\n}\nvariables {\n {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}\n}')
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 856, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 298, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/tasks.py", line 449, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 173, in run
create_link(urlparse(run_link).path)
File "/usr/local/lib/python3.8/site-packages/prefect/artifacts.py", line 31, in create_link
return client.create_task_run_artifact(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1689, in create_task_run_artifact
result = self.graphql(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 296, in graphql
result = <http://self.post|self.post>(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 211, in post
response = self._request(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 433, in _request
response = self._send_request(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 365, in _send_request
raise ClientError(msg)
prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <http://10.0.4.45:4200/graphql>
This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:
query {
mutation($input: create_task_run_artifact_input!) {
create_task_run_artifact(input: $input) {
id
}
}
}
variables {
{"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}
}
ale
11/25/2020, 5:29 PMcreate_project
API, but nothing related to updateMike Marinaccio
11/25/2020, 5:45 PMScheduled
via the UI and 2) rerunning the entire flow. However, the first step fails with the attached UI error. I see no errors in the Chrome console, nor anything in the service logs.
This only appears to happen when I select the Change the state of all task runs too
box.
I’m running Prefect Server with the latest containers.
Any tips? Thanks!Innokenty Tretyakov
11/26/2020, 11:10 AMquery_job
object. I mean, if one have an idea to log the results of the query (status, proceeded_bytes, etc.), he or she can not get an object to fetch that info.
I mean something like that (tasks/gcp/bigquery.py; class BigQueryTask; def run):Vipul
11/27/2020, 9:19 AMRoey Brecher
11/27/2020, 4:05 PMDuring handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 446, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.4.45', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0de03bb820>, 'Connection to 10.0.4.45 timed out. (connect timeout=30)'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/flow_runner.py", line 179, in interrupt_if_cancelling
flow_run_info = self.client.get_flow_run_info(flow_run_id)
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1015, in get_flow_run_info
result = self.graphql(query).data.flow_run_by_pk # type: ignore
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 275, in graphql
result = <http://self.post|self.post>(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 230, in post
response = self._request(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 400, in _request
response = self._send_request(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in _send_request
response = <http://session.post|session.post>(url, headers=headers, json=params, timeout=30)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 504, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='10.0.4.45', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0de03bb820>, 'Connection to 10.0.4.45 timed out. (connect timeout=30)'))
Vladimir Zoubritsky
11/29/2020, 11:04 AMlocalhost
, so it wouldn't be reachable on it's network interface - and changing it to 0.0.0.0 makes health checks pass, are there any security implications for this or it's a good default?
https://github.com/PrefectHQ/server/blob/a3626c81f280bf749394c2d00d15d6450b724a44/helm/prefect-server/templates/agent/deployment.yaml#L65JC Garcia
11/30/2020, 3:30 PMdbt_task = DbtShellTask(
...
)
with Flow("dbt-flow") as flow:
dbt_command_string = Parameter('dbt_command_string', default = 'dbt run', required = True)
dbt_commands = dbt_command_string.split(',')
for command in dbt_commands:
dbt_task(
command=dbt_command,
...
)
JC Garcia
11/30/2020, 3:40 PMPhilip Evans
12/01/2020, 6:05 AMschedule = Schedule(
clocks=[
IntervalClock(
start_date=pendulum.datetime(2020, 12, 1, 16, 20, 0, tz="Australia/Sydney"),
interval=timedelta(days=1),
parameter_defaults=param
)
for param in parameters
]
)
However when I deploy the flow to prefect server it runs 10 out of my 4000 odd params and schedules another 10 for tomorrow. Is there a limit to how many Interval clocks I can use with the prefect server schedule or potentially some other reason why it isn't queuing the full set of params to be run? Cheers, Phil