Hey there! I’m trying out the artifact task, and I think it works well. However, I’m trying to get it work with larger artifacts. I submitted earlier a PR to make Apollo’s body size configurable. But I’m now running into another limitation. In the code there is a hard coded limit of 1Mb. I’m willing to submit a PR, but am unfamiliar with this codebase (with graphql in general actually). Can someone point me how to make this limit check configurable?

if data_size > 1000000:  # 1 mb max
    raise ValueError("Artifact data payload exceedes 1Mb limit.")

@Joël Luijmes I'm not 100% sure if this needs to go through the API payload. Artifacts API allows you to configure an arbitrary string that gets rendered in the UI as markdown. Can you share an example flow with artifact that you try to create?

Well I think the payload goes trhough the API for storage, and is later retrieved. Anyways, upon submitting the artifact I get exact exception which is raised:

prefect.exceptions.ClientError: [{'message': 'Artifact data payload exceedes 1Mb limit.', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_task_run_artifact'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Artifact data payload exceedes 1Mb limit.'}}}]

This can be reproduced by generating an artifact which is larger than 1Mb.

Can you estimate how big is your payload? Apollo (which handles all requests) has a limit of 5mb per request. No payload can exceed this size. So if it’s larger than 5MB, then you would need to look at other options. You could also think about splitting your report (or the markdown you try to publish in the UI) into several reports. An artifact gets created per task, but you can have as many tasks with artifacts as you wish. So batching this large report could be an option worth considering. Otherwise, feel free to submit a PR, it can always be discussed with other engineers via Github and they can answer your questions. But I would try the easier option with batching first.

Well I’ve already circumvented the apollo check with this PR https://github.com/PrefectHQ/server/pull/296 🤓 So now I’d like to make this piece also configurable, just need some direction from engineers. I thought I might aswell ask here, but if you prefer I can open up a ticket at GH.

whatever works best for you. This week is Thanksgiving, so you may expect slower replies on Slack. Github is better because if you already have a solution in mind, you can show it there and it’s easier for other engineers to know what you try to do and why, plus Github is more durable.

Submitted an issue 🙂 https://github.com/PrefectHQ/server/issues/319

nice, thank you! 🙌