hi there - I’m getting an error when trying to run...
# prefect-community
hi there - I’m getting an error when trying to run one of my flows:
Copy code
'413 Client Error: Request Entity Too Large for url: <http://prefect-api.aq.tc/graphql>'
Hi @Kevin Weiler - it looks like you're probably trying to log something that's too large
Do you have any obvious culprits in your code?
looking at that step
lol - my guess is that it’s this:
Copy code
<http://prefect_logger.info|prefect_logger.info>(f'validate prod data stdout {stdout}')
so - what’s the limit on how large this can be? Can it be adjusted?
Ah yes, that could definitely do it 😅
I don't think we expose an explicit way to adjust this, I'm not sure where it's being enforced (my guess would be at the
level). However, I think it's probably not a good idea to increase that cap, since these logs are being stored in your db and regularly storing large entities in single rows of a db is usually not desired behavior.
yeah - that’s fair probably better to explicitly log output like this to a defined place
upvote 1
My advice would be to pare down what you're logging just as a general rule, since it'll help you in the future when you're more explicit
If this sort of large entity is important, you can always send it to a more durable place like an S3/GCS bucket
@nicholas - I was wrong actually, this is the full function:
Copy code
def create_validate_prod_data():
    prefect_logger = prefect.context.get('logger')
    p = subprocess.Popen(['python', '-m', 'pytest', '-vv', '-s', 'config_validation'], cwd='/am/python/')
    stdout, stderr = p.communicate()
    <http://prefect_logger.info|prefect_logger.info>(f'validate prod data stdout {stdout}')
    return_code = p.returncode
    if return_code != 0:
        raise FAIL()
    return datastore.load_state('/am/python/datastore_state.json')
that return line is what it’s choking on
so I think that will get cloudpickled and consumed by whatever downstream task needs it
but will it be stored in the DB?
No, the return won't be stored in the db but you can run into similar entity issues as you're seeing. My advice is pretty much the same except I'd recommend you to pass references to stored data instead of the data itself 🙂
yeah that’s reasonable
ok - definitely works now - thanks for the guidance!