<@ULVA73B9P> I have a json over 5000 characters th...
# ask-marvin
n
@Marvin I have a json over 5000 characters that I cannot save in Variables on Prefect Cloud, what other options do I have? It is a json map of mongodb collection names to their snake case names for about 15 mongodb databases.
m
When dealing with large JSON objects that exceed the 5000-character limit for Prefect Cloud Variables, you have a few options: 1. Use Cloud Storage: Store your JSON in a cloud storage service like AWS S3, Google Cloud Storage, or Azure Blob Storage. Prefect has integration libraries that allow you to easily interact with these services. You can use storage blocks to save and retrieve your JSON data. More details can be found here. 2. Secrets for Sensitive Data: If your JSON contains sensitive information, consider using Prefect's Secret blocks. While these are meant for sensitive data, they offer a way to securely store and access data. More on secrets can be found here. 3. Disk Storage: If your data isn't sensitive, you can write the JSON to disk within your flow. This approach is useful for large data that doesn't need to be passed between tasks. 4. Compression: Consider compressing your JSON data before storage to save space. This can be done with Prefect's task result persistence options. By utilizing these methods, you can effectively handle large JSON data in Prefect Cloud without relying on Variables.