Hafsa Junaid
08/01/2022, 9:44 PMHTTPSConnectionPool(host='34.66.191.39', port=443): Max retries exceeded with url: /apis/batch/v1/namespaces/default/jobs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f3196f62eb0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
Matt Fysh
08/01/2022, 10:31 PMif __name__ == "__main__": while True:
- are there any examples where flows can be invoked via realtime events, e.g. S3 ObjectCreated
Furthermore, are there any Prefect components in place so that if you change the flow code, it can replay all of the prior events?Stephen Herron
08/02/2022, 1:28 AMimport asyncio
from prefect_snowflake import SnowflakeCredentials
from prefect import flow
creds = SnowflakeCredentials.load("credentials")
from prefect_snowflake.database import snowflake_query
@flow(name="Snowflake Query")
async def select_something():
result = await snowflake_query(
"select 1 as test",
creds,
)
if __name__ == "__main__":
asyncio.run(select_something())
Seems to work locally but when I try to run ad-hoc via a deployment I get the error:
...
prefect_snowflake/database.py", line 69, in snowflake_query
with snowflake_credentials.get_connection(**connect_params) as connection:
AttributeError: 'coroutine' object has no attribute 'get_connection'
Chu
08/02/2022, 2:05 AMChris Hansen
08/02/2022, 3:15 AMHa Pham
08/02/2022, 5:07 AMFlorian Sell
08/02/2022, 6:48 AMRasmus Lindqvist
08/02/2022, 7:43 AMscrapy
. I have configured scrapy to log to stdout, and configured my Prefect task to log_stdout=True
. Should not Prefect then pick up the logs from scrapy or am I missing something?Joshua Greenhalgh
08/02/2022, 8:30 AMDavid
08/02/2022, 8:49 AMKha Nguyen
08/02/2022, 10:12 AM@task
def run(x: int, y: int) -> int:
return x + y
@flow
def my_flow():
results = run.map([1, 2, 3, 4], y=1)
How can I map only on x
, and use the same y
argument?Kha Nguyen
08/02/2022, 11:21 AMOscar Björhn
08/02/2022, 11:23 AMKha Nguyen
08/02/2022, 11:43 AMChu
08/02/2022, 12:13 PMChu
08/02/2022, 2:30 PMTypeError: `fn` must be callable
I’m calling a task inside a With Flow statement, and the Task should return me a list of stringsMateo Merlo
08/02/2022, 2:46 PMRay Shi
08/02/2022, 3:25 PMChristian Nuss
08/02/2022, 5:09 PMRonald Tichauer
08/02/2022, 5:26 PMimage=<my_images_name>
argument to the DockerContainer, but doesn’t work, or at least, it doesn’t run the CMD
specified in the Dockerfile of that imageJacob Goldberg
08/02/2022, 5:44 PM404 Client Error for <http+docker://localhost/v1.41/containers/create?name=elastic-salamander>: Not Found ("No such image: <http://26123096XXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest-165946XXXX%22|26123096XXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest-165946XXXX">)
I can go on AWS ECR, see the container, and see that the URI matches with the one from the error so it does exist…I have changed some settings recently regarding the python version and docker build, but cannot understand why i would get a container not found error
edit: formattingharis khan
08/02/2022, 5:51 PMChu
08/02/2022, 6:36 PMMansour Zayer
08/02/2022, 8:50 PMfrom underlying_flows.flow_a import flow_a
flow_a.register(project...)
With Flow(...):
...
Because when we do that, prefect throws a warning UserWarning: Attempting to call "flow.register" during execution of flow file will lead to unexpected results.
John Mizerany
08/02/2022, 9:19 PM--cli-input-json
tag for cli, should there be an extra argument in the prefect-agent-td.json
file called agentRoleArn
? There are two for executionRoleArn
and taskRoleArn
but if the agent itself needs a Role in AWS should that be specified or can it be included in the other variables?Billy McMonagle
08/02/2022, 9:25 PMHafsa Junaid
08/02/2022, 9:48 PMprefect deployment build ./sample.py:samplefunc --name "ALS Deployment" --storage-block recommenderprefectserver
Error: Deployment.yaml is not created with following exceptionTom Kaszemacher
08/02/2022, 10:02 PM@task()
def get_inputs() -> List:
# This task retrieves a list of inputs to process from a DB and returns it
pass
with Flow(name='dispatcher') as dispatcher:
inputs = get_inputs()
for i in inputs:
create_flow_run('worker', parameters={'input': i})
It doesn’t work as I need to provide either nouts or return a Tuple in order to iterate over my inputs, however I don’t know the size of it.
What would be a correct approach here?
Thanks!André Bonatto
08/02/2022, 10:11 PMHafsa Junaid
08/02/2022, 10:54 PM