Slackbot
08/01/2022, 6:37 PMLeon Kozlowski
08/01/2022, 6:37 PMChu
08/01/2022, 6:42 PMJonathan Mathews
08/01/2022, 7:38 PMrun_task_kwargs_path
for prefect agent ecs start
as it’s not picking up my VPC (not the default one) but the docs don’t say what format options.yml
should be in. Saw this issue here, but hasn’t been an update recently https://prefect-community.slack.com/archives/CL09KU1K7/p1653756981241529Nathan Ballou
08/01/2022, 7:52 PMFile "C:\Users\...\lib\site-packages\comet_ml\connection.py", line 35, in <module>
from comet_ml import config
File "C:\Users\...\lib\site-packages\comet_ml\config.py", line 59, in <module>
from everett.manager import ConfigIniEnv as UpstreamConfigIniEnv
ImportError: cannot import name 'ConfigIniEnv' from 'everett.manager'
Any thoughts?Leo Marvin
08/01/2022, 8:04 PMwith Flow(
"test-flow",
storage=S3(bucket="my-bucket"),
run_config=ECSRun(...)
but getting this error:
Failed to load and execute flow run: NoCredentialsError('Unable to locate credentials')
I added a Secret named AWS_CREDENTIALS
through the prefect cloud UI (https://cloud.prefect.io/team/secrets) with a json of the form
{
"ACCESS_KEY": "...",
"SECRET_ACCESS_KEY": "..."
}
but still getting this error. Curious if anyone one has tips to troubleshoot.Seth Coussens
08/01/2022, 8:48 PM'prefect.engine' is a package and cannot be directly executed
when trying to run a DockerContainer flow in Prefect 2.0?Yupei Chen
08/01/2022, 9:02 PMKyle McChesney
08/01/2022, 9:29 PMSebastián Montoya Tapia
08/01/2022, 9:33 PMHafsa Junaid
08/01/2022, 9:44 PMHTTPSConnectionPool(host='34.66.191.39', port=443): Max retries exceeded with url: /apis/batch/v1/namespaces/default/jobs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f3196f62eb0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
Matt Fysh
08/01/2022, 10:31 PMif __name__ == "__main__": while True:
- are there any examples where flows can be invoked via realtime events, e.g. S3 ObjectCreated
Furthermore, are there any Prefect components in place so that if you change the flow code, it can replay all of the prior events?Stephen Herron
08/02/2022, 1:28 AMimport asyncio
from prefect_snowflake import SnowflakeCredentials
from prefect import flow
creds = SnowflakeCredentials.load("credentials")
from prefect_snowflake.database import snowflake_query
@flow(name="Snowflake Query")
async def select_something():
result = await snowflake_query(
"select 1 as test",
creds,
)
if __name__ == "__main__":
asyncio.run(select_something())
Seems to work locally but when I try to run ad-hoc via a deployment I get the error:
...
prefect_snowflake/database.py", line 69, in snowflake_query
with snowflake_credentials.get_connection(**connect_params) as connection:
AttributeError: 'coroutine' object has no attribute 'get_connection'
Chu
08/02/2022, 2:05 AMChris Hansen
08/02/2022, 3:15 AMHa Pham
08/02/2022, 5:07 AMFlorian Sell
08/02/2022, 6:48 AMRasmus Lindqvist
08/02/2022, 7:43 AMscrapy
. I have configured scrapy to log to stdout, and configured my Prefect task to log_stdout=True
. Should not Prefect then pick up the logs from scrapy or am I missing something?Joshua Greenhalgh
08/02/2022, 8:30 AMDavid
08/02/2022, 8:49 AMKha Nguyen
08/02/2022, 10:12 AM@task
def run(x: int, y: int) -> int:
return x + y
@flow
def my_flow():
results = run.map([1, 2, 3, 4], y=1)
How can I map only on x
, and use the same y
argument?Kha Nguyen
08/02/2022, 11:21 AMOscar Björhn
08/02/2022, 11:23 AMKha Nguyen
08/02/2022, 11:43 AMChu
08/02/2022, 12:13 PMChu
08/02/2022, 2:30 PMTypeError: `fn` must be callable
I’m calling a task inside a With Flow statement, and the Task should return me a list of stringsMateo Merlo
08/02/2022, 2:46 PMRay Shi
08/02/2022, 3:25 PMChristian Nuss
08/02/2022, 5:09 PMRonald Tichauer
08/02/2022, 5:26 PMimage=<my_images_name>
argument to the DockerContainer, but doesn’t work, or at least, it doesn’t run the CMD
specified in the Dockerfile of that imageRonald Tichauer
08/02/2022, 5:26 PMimage=<my_images_name>
argument to the DockerContainer, but doesn’t work, or at least, it doesn’t run the CMD
specified in the Dockerfile of that imageJean Luciano
08/02/2022, 5:28 PMKhuyen Tran
08/02/2022, 5:36 PMRonald Tichauer
08/03/2022, 11:53 PMbuild deployment
but now I saw that even specifying the infrastructure within the deployment, it was not getting it when I built the yaml.
It still wasn’t what I needed to do and couldn’t make it work, that was running a specific docker from a flow. Sadly I think I’ll have to go back to Airflow 😭
I was using prefect 2.0 btwKhuyen Tran
08/04/2022, 2:59 PM