Jarvis Stubblefield

12/16/2022, 3:59 PM
I have another issue that may not be related to Prefect, but thought I would post here in the off chance someone has had this issue when using Prefect. I have a task that looks at about 30-50,000 records total if it looks at all of the records in the table. Transactions are to be committed in blocks of 1,000 records checked. For each 1k there are less than that being updated. Previously at times this task would take a few hours when it really had to go over 3-5,000 records. The first three days I started this task it took a while which was to be expected as I stopped manually running the task. However, it is now again taking a LONG time and I’m ending up with MySQL Lock Table Timeouts
MySQLdb.OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')
… any thoughts or ideas on what might be causing additional delay?
So to further explain “LONG” time … the task from yesterday has been running for over 19 hours. These are daily tasks… this lock has so far affected one user being able to log into the system and also affected my running a different one off task to update some records.
This is the long running task flow…
def update_and_alert_about_users_who_need_logs():
    logger = get_run_logger()
    # Update any users who need new auto logs.
    num_updates = 0
    num_last_log_alerts = 0
    num_changes = 0

    active =
    for user in active:
            if LivePerformance.maybe_update(user):
                num_updates += 1
                num_changes += 1

            if user.logs_needed_alert():
                num_last_log_alerts += 1
                num_changes += 1
        except KeyboardInterrupt:

        if num_changes > 1000:
            <|>("Committing progress...")
            num_changes = 0

    <|>("Committing changes.")
    <|>(f"{num_updates} users needed a score update.")
    <|>(f"{num_last_log_alerts} users needed a last log alert.")
So I’m trying to figure out what’s going on … and it seems that the database isn’t doing anything … not sure how to debug this.
mysql> show full processlist;
| Id     | User       | Host               | db              | Command | Time | State    | Info                  |
| 884310 | xxxxxxxxxx | localhost          | NULL            | Sleep   |    3 |          | NULL                  |
| 890453 | xxxxxxxxxx | xxx.xx.xx.75:35920 | tenzinga_django | Sleep   |    0 |          | NULL                  |
| 894731 | xxxxxxxxxx | xxx.xx.xx.31:53302 | tenzinga_django | Sleep   | 1771 |          | NULL                  |
| 894732 | xxxxxxxxxx | xxx.xx.xx.31:53304 | tenzinga_django | Query   |    0 | starting | show full processlist |
4 rows in set (0.00 sec)
I honestly don’t know how to really read the
show engine innodb status;
I’m tempted to kill the process that seems to be sleeping.
Well I checked the agent status … it is still outputting print statements … so it is still running.