Martin T
05/04/2022, 6:08 AM$ prefect run --path myflow.py --log-level INFO --param myparam=val
Here the development flow becomes rather blackboxed/trial-and-error.
Is it possible to somehow set breakpoints in VSCode, start a prefect flow from CLI, and attach the debugger to the python process?
(Running with the LocalExecutor)Jelle Vegter
05/04/2022, 11:52 AMAbuzar Shaikh
05/04/2022, 12:06 PMIevgenii Martynenko
05/04/2022, 12:45 PMMichail Melonas
05/04/2022, 12:45 PMError downloading Flow from S3: An error occurred (ExpiredToken) when calling the GetObject operation: The provided token has expired.
I’m using AWS SSO credentials to upload the flow to S3:
flow.storage = S3(
bucket=Config.PREFECT_S3_BUCKET,
client_options={
"aws_access_key_id": Config.AWS_ACCESS_KEY_ID,
"aws_secret_access_key": Config.AWS_SECRET_ACCESS_KEY,
"aws_session_token": Config.AWS_SESSION_TOKEN,
},
)
Initially, the flow runs. However, flows scheduled after the token expiry fail with the above error. How do I both use SSO to upload flows to S3 and have them scheduled on Prefect Cloud?Patrick Tan
05/04/2022, 1:47 PMfrom prefect import Flow
from prefect.storage import S3
flow = Flow("s3-flow", storage=S3(bucket="<my-bucket>"))
Birkir Björnsson
05/04/2022, 2:33 PMJason
05/04/2022, 2:50 PM@task
def task_a():
common_class = Common(Secret())
output = common_class.task_a()
return output
@task
def task_b():
common_class = Common(Secret())
output = common_class.task_b()
return output
Jonathan Mathews
05/04/2022, 3:43 PMMichelle Brochmann
05/04/2022, 4:23 PMMichelle Brochmann
05/04/2022, 4:56 PM/opt/prefect/healthcheck.py:130: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/storage.html#docker>
flows = cloudpickle_deserialization_check(flow_file_paths)
Is there an obvious reason why this would happen? My tasks are defined in a module that I’m importing into the file where the Flows are defined. I can provide more info if needed!Jonathan Mathews
05/04/2022, 5:07 PM.run()
it complains that it is not JSON serializable:
with Flow(FLOW_NAME, storage=set_storage(FLOW_NAME), run_config=set_run_config(),) as flow:
snowflake_password=EnvVarSecret(name='SNOWFLAKE_PASSWORD', raise_if_missing=True).run()
....
upload_results = SnowflakeQuery(
user=SNOWFLAKE_USER,
password=snowflake_password,
account=SNOWFLAKE_ACCOUNT,
warehouse=SNOWFLAKE_WAREHOUSE,
database=SNOWFLAKE_DATABASE,
role=SNOWFLAKE_ROLE,
schema=SNOWFLAKE_SCHEMA).map(query=queries)
Lon Nix
05/04/2022, 6:32 PMZach Schumacher
05/04/2022, 6:37 PMChris Reuter
05/04/2022, 6:50 PMJonathan Mathews
05/04/2022, 8:33 PMTabari Brannon
05/04/2022, 8:37 PMFile "C:\etls\tractor_hours_analyzer\tractor_hours_analyzer.py", line 89, in <module>
transformed = transform_add_files_to_dataframe(extracted)
raise TypeError('too many positional arguments') from None
TypeError: too many positional argumentswith
Raviraja Ganta
05/04/2022, 8:51 PMSean Leakehe
05/04/2022, 11:01 PMBaris Cekic
05/04/2022, 11:10 PMBaris Cekic
05/04/2022, 11:12 PMBaris Cekic
05/04/2022, 11:15 PMEdmondo Porcu
05/05/2022, 1:20 AMBen Ayers-Glassey
05/05/2022, 3:40 AMDocker
class takes an env_vars
dict, but no build_args
dict, which would be nice.
Here is the place in the Docker
class where ENVs are generated:
https://github.com/PrefectHQ/prefect/blob/8e04ccad071d1127afc7ca3580f1fe6f4e884f27/src/prefect/storage/docker.py#L437-L442
...the ARGs could go right above there.
In any case, ultimately what I'm trying to do is get the Docker
class to install python_dependencies
from our private PyPi server.
Is there a good way to do that?Ben Ayers-Glassey
05/05/2022, 5:27 AMRaviraja Ganta
05/05/2022, 5:50 AMBernard Greyling
05/05/2022, 6:17 AMetl prefect-job-29a6d24a-hwnzx 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-30bd9ad7-zsc6z 0/1 CreateContainerConfigError 0 11h
etl prefect-job-39468921-lt95t 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-3c1aee81-hvbv4 0/1 Completed 0 7h49m
etl prefect-job-3c84d152-wfqw9 0/1 Completed 0 7h53m
etl prefect-job-3fb2397e-7dc5b 0/1 ImagePullBackOff 0 11h
etl prefect-job-5ba165eb-rfxhq 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-62ae93a3-ptbcr 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-6cfd964c-p7xpl 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-86e07407-nmpwb 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-9478c771-bfg26 0/1 ImagePullBackOff 0 11h
etl prefect-job-950ba45b-tv4f2 0/1 CreateContainerConfigError 0 11h
etl prefect-job-a199b1d6-6vwmd 0/1 Completed 0 7h49m
etl prefect-job-a4976ce3-gxqv6 0/1 Completed 0 6h5m
Julian Brendel
05/05/2022, 7:16 AMIevgenii Martynenko
05/05/2022, 7:48 AMrun_config
of type DockerRun
, only KubernetesRun
is supported"
Suppose that Prefect knows it runs in K8s and now all dataflow should use K8s agent, right?Mini Khosla
05/05/2022, 8:51 AMMini Khosla
05/05/2022, 8:51 AMAnna Geller
05/05/2022, 10:28 AMKevin Kho
05/05/2022, 1:49 PM