Alexey Stoletny
09/13/2022, 4:37 PMChristopher Boyd
09/13/2022, 5:28 PMAlexey Stoletny
09/13/2022, 5:52 PMChristopher Boyd
09/13/2022, 6:05 PMAlexey Stoletny
09/13/2022, 6:07 PMChristopher Boyd
09/13/2022, 6:09 PMAlexey Stoletny
09/13/2022, 6:09 PMChristopher Boyd
09/13/2022, 6:12 PMAlexey Stoletny
09/13/2022, 6:12 PMChristopher Boyd
09/13/2022, 6:13 PMAlexey Stoletny
09/13/2022, 6:13 PMPranit
09/13/2022, 6:14 PMAlexey Stoletny
09/13/2022, 6:15 PMChristopher Boyd
09/13/2022, 6:16 PMAlexey Stoletny
09/13/2022, 6:16 PMPranit
09/13/2022, 6:17 PMAlexey Stoletny
09/13/2022, 6:18 PMAWS_ACCESS_KEY_ID = s3_block.aws_access_key_id.get_secret_value()
AWS_SECRET_KEY_ID = s3_block.aws_secret_access_key.get_secret_value()
CLICKHOUSE_HOST = Secret.load("clickhouse-host").get()
CLICKHOUSE_PORT = Secret.load("clickhouse-port").get()
CLICKHOUSE_PASS = Secret.load("clickhouse-pass").get()
REDACTED_S3_URL = Secret.load("REDACTED-s3-url").get()
REDACTED_FILENAME_REGEXP = String.load("REDACTED-filename-regexp").value
REDACTED_FILENAME_TEMPLATE = String.load("REDACTED-filename-template").value
QUERY_TEMPLATE_PATH = "./sql/REDACTED.sql"
query_template_file = open(QUERY_TEMPLATE_PATH, "r")
query_string_template = query_template_file.read()
query_template_file.close()
query_string = query_string_template.format(
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID,
AWS_SECRET_KEY_ID=AWS_SECRET_KEY_ID,
...)
<http://logger.info|logger.info>(query_string.replace(AWS_ACCESS_KEY_ID, "*****").replace(AWS_SECRET_KEY_ID, "*****"))
query_file_name = "./temp-query-{random_string}.sql"
query_file_name = query_file_name.format(random_string=''.join(random.choice(string.ascii_lowercase) for i in range(10)))
query_file = open(query_file_name, "w")
query_file.write(query_string)
query_file.close()
command_template = "clickhouse-client --host {CLICKHOUSE_HOST} --secure --port {CLICKHOUSE_PORT} --password {CLICKHOUSE_PASS} --queries-file {query_file_name} && rm {query_file_name}"
command_to_execute = command_template.format(CLICKHOUSE_HOST=CLICKHOUSE_HOST, CLICKHOUSE_PORT=CLICKHOUSE_PORT, CLICKHOUSE_PASS=CLICKHOUSE_PASS, query_file_name=query_file_name)
<http://logger.info|logger.info>("About to run: " + command_to_execute.replace(CLICKHOUSE_PASS, "*****"))
<http://logger.info|logger.info>(shell_run_command(command=command_to_execute, return_all=True))
Christopher Boyd
09/13/2022, 6:33 PMsuccess = False
counter = 0
while not success and counter < MAX_RETRY:
try:
secret.load()
success = True
except (ConnectionResetError, TimeoutError) as err:
if counter >= MAX_RETRY:
raise
counter += 1
time.sleep(TIME_BETWEEN_RETRY)
except Exception as e:
logging.warning(repr(e))
raise
tic = time.time()
secret.load()
toc = time.time()
print toc - tic
This won’t solve the issue, but at least maybe get some insight into how long this request is taking to get a responseAlexey Stoletny
09/13/2022, 6:38 PMChristopher Boyd
09/13/2022, 6:39 PMAlexey Stoletny
09/13/2022, 6:39 PMChristopher Boyd
09/13/2022, 6:39 PMAlexey Stoletny
09/13/2022, 6:41 PMRobin Weiß
09/15/2022, 7:38 AMChristopher Boyd
09/15/2022, 12:08 PMCLICKHOUSE_HOST = Secret.load("clickhouse-host").get()
CLICKHOUSE_PORT = Secret.load("clickhouse-port").get()
CLICKHOUSE_PASS = Secret.load("clickhouse-pass").get()
REDACTED_S3_URL = Secret.load("REDACTED-s3-url").get()
Alexey Stoletny
09/17/2022, 4:19 AMRobin Weiß
09/17/2022, 11:19 AMCrashed
state which for some reason means the retries don’t work. For your information: I found that quite a few people seem to be having this problem and it seems possible that it’s an infrastructure or load problem on the Prefect Cloud side. I will probably try deploying our own Orion setup next week to see if it remedies the issues. Will keep you posted!Alexey Stoletny
09/21/2022, 7:01 PMRobin Weiß
09/22/2022, 8:22 AMChristopher Boyd
09/22/2022, 12:48 PMAlexey Stoletny
09/23/2022, 8:45 PMChristopher Boyd
09/23/2022, 8:48 PMAlexey Stoletny
09/23/2022, 8:49 PMChristopher Boyd
09/23/2022, 8:54 PM