Sandbox Cache for Scalable Large Scale Dataset Modeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for creating models from database information are not scalable, memory-intensive, and can impact the underlying database or other analysts' models, making it difficult to manipulate data without affecting the original database or sharing models effectively.
Innovation Solution
A system using a sandbox cache that allows users to manipulate data independently of the underlying database, with a user entity cache that stores changes without copying the entire database, using an entity model with primary and foreign keys to manage relationships and dependencies, and tracking sensitive values to ensure data integrity and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a simple spreadsheet is used for modeling, then ease of operation is improved, but scalability deteriorates due to memory and processing limitations
Solution Approach 1:
The system segments the database into an underlying database layer and a sandbox cache layer. The sandbox cache holds only the subset of data needed for modeling operations, while the underlying database preserves the complete historical data. This segmentation allows the modeling operations to be performed on a manageable subset without compromising the scalability or integrity of the overall system.
2Ease of operation
If the entire database is copied to memory for modeling, then ease of operation is improved, but memory consumption worsens significantly
Solution Approach 1:
The system extracts only the necessary subset of data from the underlying database into the sandbox cache for modeling operations. This extraction principle allows the system to work with a manageable amount of data in memory while preserving the complete database on disk, thereby reducing memory consumption while maintaining operational ease.
3Productivity
If changes are made directly in the underlying database, then productivity is improved, but reliability worsens due to potential data loss and impact on other analysts
Solution Approach 1:
The sandbox cache serves as an intermediary layer between the user and the underlying database. All modeling operations and data manipulations are performed within the sandbox cache, which isolates these changes from the underlying database. This intermediary mechanism allows high productivity in modeling operations while preserving the reliability and integrity of the underlying historical data.
4Adaptability or versatility
If the same database is shared among multiple analysts, then adaptability is improved, but object-generated harmful factors worsen due to conflicting manipulations
Solution Approach 1:
The system segments the data access model by providing each analyst with their own sandbox cache instance derived from the shared underlying database. This segmentation allows multiple analysts to work simultaneously on their own isolated copies of the data, enabling adaptability and collaboration while preventing harmful conflicts from affecting the shared database or other analysts' work.
Data Source
AI summary
Users can model changes to entities in large scale sets of data for various portfolio holdings in different respective sandbox caches. A method includes loading sets of data in a database and organizing the loaded data in entity caches according to an entity model, each entity cache corresponding to one or more entities associated with respective portfolio holdings. Further steps include creating an initial report of information drawn from the loaded data for manipulation by a user through a user-interface, storing information in the initial report in a respective sandbox cache having data organized according to the entity model, and enabling the user to manipulate the respective sandbox cache to change values in the data organized according to the entity model in the respective sandbox cache without changing values of data in other sandbox caches or in the database.