Jonathan Mathews
05/04/2022, 3:43 PMMichelle Brochmann
05/04/2022, 4:23 PMMichelle Brochmann
05/04/2022, 4:56 PM/opt/prefect/healthcheck.py:130: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/storage.html#docker>
flows = cloudpickle_deserialization_check(flow_file_paths)
Is there an obvious reason why this would happen? My tasks are defined in a module that I’m importing into the file where the Flows are defined. I can provide more info if needed!Jonathan Mathews
05/04/2022, 5:07 PM.run()
it complains that it is not JSON serializable:
with Flow(FLOW_NAME, storage=set_storage(FLOW_NAME), run_config=set_run_config(),) as flow:
snowflake_password=EnvVarSecret(name='SNOWFLAKE_PASSWORD', raise_if_missing=True).run()
....
upload_results = SnowflakeQuery(
user=SNOWFLAKE_USER,
password=snowflake_password,
account=SNOWFLAKE_ACCOUNT,
warehouse=SNOWFLAKE_WAREHOUSE,
database=SNOWFLAKE_DATABASE,
role=SNOWFLAKE_ROLE,
schema=SNOWFLAKE_SCHEMA).map(query=queries)
Lon Nix
05/04/2022, 6:32 PMZach Schumacher
05/04/2022, 6:37 PMChris Reuter
05/04/2022, 6:50 PMJonathan Mathews
05/04/2022, 8:33 PMTabari Brannon
05/04/2022, 8:37 PMFile "C:\etls\tractor_hours_analyzer\tractor_hours_analyzer.py", line 89, in <module>
transformed = transform_add_files_to_dataframe(extracted)
raise TypeError('too many positional arguments') from None
TypeError: too many positional argumentswith
Raviraja Ganta
05/04/2022, 8:51 PMSean Leakehe
05/04/2022, 11:01 PMBaris Cekic
05/04/2022, 11:10 PMBaris Cekic
05/04/2022, 11:12 PMBaris Cekic
05/04/2022, 11:15 PMEdmondo Porcu
05/05/2022, 1:20 AMBen Ayers-Glassey
05/05/2022, 3:40 AMDocker
class takes an env_vars
dict, but no build_args
dict, which would be nice.
Here is the place in the Docker
class where ENVs are generated:
https://github.com/PrefectHQ/prefect/blob/8e04ccad071d1127afc7ca3580f1fe6f4e884f27/src/prefect/storage/docker.py#L437-L442
...the ARGs could go right above there.
In any case, ultimately what I'm trying to do is get the Docker
class to install python_dependencies
from our private PyPi server.
Is there a good way to do that?Ben Ayers-Glassey
05/05/2022, 5:27 AMRaviraja Ganta
05/05/2022, 5:50 AMBernard Greyling
05/05/2022, 6:17 AMetl prefect-job-29a6d24a-hwnzx 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-30bd9ad7-zsc6z 0/1 CreateContainerConfigError 0 11h
etl prefect-job-39468921-lt95t 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-3c1aee81-hvbv4 0/1 Completed 0 7h49m
etl prefect-job-3c84d152-wfqw9 0/1 Completed 0 7h53m
etl prefect-job-3fb2397e-7dc5b 0/1 ImagePullBackOff 0 11h
etl prefect-job-5ba165eb-rfxhq 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-62ae93a3-ptbcr 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-6cfd964c-p7xpl 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-86e07407-nmpwb 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-9478c771-bfg26 0/1 ImagePullBackOff 0 11h
etl prefect-job-950ba45b-tv4f2 0/1 CreateContainerConfigError 0 11h
etl prefect-job-a199b1d6-6vwmd 0/1 Completed 0 7h49m
etl prefect-job-a4976ce3-gxqv6 0/1 Completed 0 6h5m
Julian Brendel
05/05/2022, 7:16 AMIevgenii Martynenko
05/05/2022, 7:48 AMrun_config
of type DockerRun
, only KubernetesRun
is supported"
Suppose that Prefect knows it runs in K8s and now all dataflow should use K8s agent, right?Mini Khosla
05/05/2022, 8:51 AMRaviraja Ganta
05/05/2022, 11:04 AMmain-project
|
src
|
| __init__.py
| module1.py
| module2.py
|
configs
|
| config.yaml
|
flows
|
| sample_flow.py
Rasmus Lindqvist
05/05/2022, 1:23 PMprefect server
for local development and prefect cloud
for production which is quite neat in terms of speed. When doing this we set the --external-postgres flag in order to use our development database, which works great!
However we are using Hasura for our other backend service which creates some git-pain as Prefect overwrites our local Hasura metadata files. Have anyone run into the same problem ?Ievgenii Martynenko
05/05/2022, 1:43 PMAdi Gandra
05/05/2022, 2:24 PMEdmondo Porcu
05/05/2022, 3:57 PMGeoffrey Keating
05/05/2022, 4:08 PMprefect_test_harness
provides an environment for flows to be tested, how might I go about testing a task in Prefect 2.0? It seems like calling .run()
on tasks has not made it over from 1.0 yet.Edmondo Porcu
05/05/2022, 5:46 PMwith Flow("mapped-fibonacci") as mapped_flow:
ms = Parameter("ms")
fib_nums = compute_large_fibonacci.map(ms)
flow_state = mapped_flow.run(ms=[10, 100, 1000, 1500])
nice but what about non array parameters? Let's say I have two parameters, one that's an array, and the other which is a constant.
with Flow("mapped-fibonacci") as mapped_flow:
ms = Parameter("ms")
ms2 = Parameter("ms2")
fib_nums = compute_large_fibonacci.map(ms, ms2)
flow_state = mapped_flow.run(ms=[10, 100, 1000, 1500])
André Dias
05/05/2022, 5:47 PMPrefect Agent
and a prefect server
on my local computer, everything residing inside of containers
. My Prefect Server
`container(s)`is based on the docker-compose
file retrieved from prefect server config > server-docker-compose.yaml
. When spinning up a container for an Agent and running prefect agent local start --api <http://host.docker.internal:4200>
, it says that I must be authenticated to communicate with the containerized server living in my local machine.
My first question is: Is authentication
mandatory in this kind of situation , for local run ?
When I authenticate by creating a new tenant using prefect server create-tenant -n default
outside the Agent container, I’m able to spin up the Agent Container
and run everything. The problem is that I wanted to run the create-tenant
inside the container too, but it’s not possible as I’m getting connection refused between containers.