Freddie
08/19/2022, 10:10 AMBigya Man Pradhan
08/19/2022, 10:18 AMfrom prefect.deployments import Deployment
]. I am not sure about how to get infrastructure block
and storage block
(created through the UI) added to the Deployment. Is this feature not included in the object?Iuliia Volkova
08/19/2022, 10:19 AMBen Strange
08/19/2022, 11:08 AMFlorian Kühnlenz
08/19/2022, 1:36 PMJohn Mizerany
08/19/2022, 1:56 PMBigya Man Pradhan
08/19/2022, 2:18 PMValueError: 'api.prefect.cloud' does not appear to be an IPv4 or IPv6 address
Complete error log in comment.Hawkar Mahmod
08/19/2022, 2:28 PMJared Robbins
08/19/2022, 3:06 PMTony Yun
08/19/2022, 3:37 PMwith Flow(
name="ETL",
) as flow:
page = 0
max_page = 999999
while page <= max_page:
page += 1
dummy_task()
Ilya Galperin
08/19/2022, 3:48 PMDeployment
object and specifying a remote S3 flow storage block and Kubernetes infrastructure, we are seeing strange behavior on flow execution. The deployment pushes the flow to S3 storage as expected (which is confirmed by the storage block in the Prefect Cloud UI being referenced in the deployment UI) but errs out with the following: Flow could not be retrieved from deployment...FileNotFoundError: [Errno 2] No such file or directory: '/My/Local/Path/my_project/flow.py'
where the path is the absolute path of the machine that applied the deployment whereas the absolute path in the s3 bucket is just <bucketname://flow.py>
. Here is the code we are using if anyone has any ideas?
from prefect.deployments import Deployment
from prefect.infrastructure import KubernetesJob
from prefect.filesystems import S3
from my_project.flow import entrypoint
infrastructure = KubernetesJob(namespace="prefect2")
deployment = Deployment.build_from_flow(
flow=entrypoint,
name="my_deployment",
work_queue_name="default",
storage=S3.load("default-block"),
infrastructure=infrastructure,
)
deployment.apply()
Owen Cook
08/19/2022, 4:20 PMNeil Natarajan
08/19/2022, 4:43 PMschedule = IntervalSchedule(interval=timedelta(seconds=30))
with Flow("workflow", schedule=schedule) as workflow:
Is there a way to run a flow on an interval schedule using the DaskTaskRunner
in prefect 2.0Ilya Galperin
08/19/2022, 4:49 PMPayam K
08/19/2022, 6:15 PM@task
def make_df(i):
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hi from Prefect %s", prefect.__version__)
<http://logger.info|logger.info>("this is the first step")
data = {'Name':['Tom', 'Brad', 'Kyle', 'Jerry'],
'Age':[20, i**2, 2*i, 18*i],
'Height' : [6.1, 5.9, 6.0, 6.1]
}
df = pd.DataFrame(data)
return df
How should I design my work to run 5 parallel task in ECS cluster?Kal
08/19/2022, 6:39 PMTim Enders
08/19/2022, 6:55 PMkwmiebach
08/19/2022, 7:12 PMChandrashekar Althati
08/19/2022, 7:54 PMPaco Ibañez
08/19/2022, 9:28 PMpackager
or storage
arguments but none of them are working for me.
deployment = Deployment.build_from_flow(
name="docker-example",
flow=my_docker_flow,
packager=FilePackager(filesystem=RemoteFileSystem.load('minio-docker')),
# storage_block=RemoteFileSystem.load('minio-docker'),
infrastructure=DockerContainer(
image = 'prefect-orion:2.1.1',
image_pull_policy = 'IF_NOT_PRESENT',
networks = ['prefect'],
env = {
"USE_SSL": False,
"AWS_ACCESS_KEY_ID": "blablabla",
"AWS_SECRET_ACCESS_KEY": "blablabla",
"ENDPOINT_URL": '<http://minio:9000>',
}
),
)
deployment.apply()
With the above code the deployment is created but the flow is not uploaded to minioAlexander Kloumann
08/19/2022, 9:31 PMsource_ids = ["source_1", "source_2", "source_3"]
with Flow("my_flow") as flow:
for source_id in source_ids:
data = extract(source_id)
data = transform(data)
load(data)
return flow
Thanks in advance!Oscar Björhn
08/20/2022, 1:09 PMMohamed Ayoub Chettouh
08/20/2022, 3:37 PMYaron Levi
08/20/2022, 5:00 PMAmjad Salhab
08/20/2022, 5:09 PMvk
08/20/2022, 8:18 PMsome_task.run(args)
from other tasks and it's very handy, cause very often it's necessary to call existing tasks (especially from prefect task library) deep inside other tasks. in Orion I didn't find how to do that, but pretty sure there should be some way?Michael Z
08/21/2022, 3:47 AMAmjad Salhab
08/21/2022, 11:52 AMLow Kim Hoe
08/21/2022, 11:54 AMFady Khallaf
08/21/2022, 1:44 PMFady Khallaf
08/21/2022, 1:44 PMMohamed Alaa
08/21/2022, 1:49 PMAnna Geller
08/21/2022, 2:26 PM