Lucas Hosoya
06/03/2022, 8:28 PMMaverick Humbert
06/04/2022, 4:34 AMFailed to load and execute flow run: ImportError('Unable to import Github, please ensure you have installed the github extra')
KubernetesRun()
with image="<http://ghcr.io/xxx/custom-prefect:1.0.0|ghcr.io/xxx/custom-prefect:1.0.0>"
Dockerfile of this image looks like:
FROM prefecthq/prefect:core
RUN pip install prefect[postgres] prefect[github] requests
& tried to add env var "EXTRA_PIP_PACKAGES": "prefect[github]"
to KubernetesRun()
Daniel Lomartra
06/05/2022, 12:01 AMwith Flow(
name=FLOW_CONSTANTS["flow_name"],
storage=STORAGE,
run_config = DockerRun(
image=DOCKER_IMAGE,
labels=FLOW_LABELS
)
) as flow:
dbt_job = Parameter(
name='dbt_job',
default=[
{
"name": "Job Step Name 1", "command": r"Arbitrary dbt CLI command #1"
},
{
"name": "Job Step Name 2", "command": r"Arbitrary dbt CLI command #2"
}
]
)
dbt.map(
command = dbt_job['command'],
task_args= {"name": dbt_job['name']}
)
Christopher Haack
06/05/2022, 7:01 AMAndrew
06/05/2022, 8:39 AMKevin Otte
06/05/2022, 1:29 PMRaphael
06/05/2022, 4:57 PMRussell Brooks
06/06/2022, 12:15 AMVolker L
06/06/2022, 12:12 PMMahesh
06/06/2022, 1:02 PMStéphan Taljaard
06/06/2022, 1:15 PMEmma Rizzi
06/06/2022, 1:18 PM--volume
option on the docker agent for my flow to access shared volumes. I tried to configure this in the job_template option of the Kubernetes run (code in thread) but it fails with the following error when i launched the flow. Is there any example to follow to use volumes with kubernetes ?Graham Mattox
06/06/2022, 1:54 PMAlexandru Anghel
06/06/2022, 3:08 PMJai P
06/06/2022, 5:09 PMMikhail Krastelev
06/06/2022, 5:11 PMPaul Stark
06/06/2022, 5:15 PMCarlos Cueto
06/06/2022, 5:16 PMwait_for_flow_run
task after triggering the external flow. I use Git storage, and Prefect Cloud as deployment.
Error:
Error during execution of task: ClientError([{'path': ['flow_run'], 'message': 'request to http://hasura:3000/v1alpha1/graphql failed, reason: read ECONNRESET', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to http://hasura:3000/v1alpha1/graphql failed, reason: read ECONNRESET', 'type': 'system', 'errno': 'ECONNRESET', 'code': 'ECONNRESET'}}}],)Jeff Bradley
06/06/2022, 5:56 PMprofiles.yml
. I understand that for local dev, profiles.yml
from the user’s .dbt folder can be used ( and hopefully not overwritten by setting overwrite_profiles=True
on DbtShellTask
) , but for deployment in K8 it is not clear what the correct thing to do is. Seems like it would be to include a profiles.yml
file along with the project, but I could be wrong about that. I’ve been reading the docs and tutorials but either haven’t found this information yet or just missed it. If anyone can point me in the right direction I would appreciate it.Fina Silva-Santisteban
06/06/2022, 6:35 PMUnexpected error: InvalidIndexError(Int64Index([0], dtype='int64'))
File "/usr/local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 5637, in _check_indexing_error
raise InvalidIndexError(key)
pandas.errors.InvalidIndexError: Int64Index([0], dtype='int64')
What’s even more confusing is that this flow used to run fine on aws fargate cluster for quite a long time and only recently started throwing that error. (We had pushed a change to the codebase, noticed the flow didn’t work anymore, undid the change we pushed but the error remained.) Any help is greatly appreciated! 🙏Ranny Sweis
06/06/2022, 7:01 PMPatrick Tan
06/06/2022, 8:01 PMcreate_flow_run.map(
flow_name=unmapped("LiveLots-ETL"),
project_name=unmapped(f"{prefect_project}"),
run_name=flowname_list,
run_config=unmapped(LocalRun(labels=[prefect_label])),
idempotency_key=idempotency_keys,
parameters=filelist,
)
Matt Alhonte
06/06/2022, 11:01 PMRandomizedSearchCV
)in dask-ml
? map
?Sandip Viradiya
06/06/2022, 11:43 PMDileep Damodaran
06/07/2022, 3:53 AMStephen Lloyd
06/07/2022, 5:14 AMDaskExecutor
might be an option, but I don’t quite understand all the setup I would need to do.Antti Tupamäki
06/07/2022, 8:57 AMSang Young Noh
06/07/2022, 11:05 AMClient error '401 Unauthorized' for url '<https://api-beta.prefect.io/api/accounts/5c1f7ebd-7031-4f69-bc00-6d21a9b17548/workspaces/9a321454-aa9e-4fb8-8f4c-17e327a77a01/blocks/get_default_storage_block>'
For reference, I was trying to run some deployments on prefect cloud, and when that didnt work, I logged out of prefect cloud and activated a local orion server instead.
The deployment command I am running is:
prefect orion database reset && prefect deployment create manager.py
Any help on this would be much appreciated!Ilya Sapunov
06/07/2022, 11:45 AMwith Flow("import") as flow:
for i in range(5):
res = import_task(
command="echo $CREDENTIALS",
env={
"CREDENTIALS": str(i)
},
)
But in that case all tasks will run in parallel, which will cause the request limits to be exceeded
Can I run those tasks sequentially ?flurven
06/07/2022, 11:59 AM