Direct Loading of Semistructured Data via Type Implementor Routines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face performance and memory consumption issues when storing semistructured data using conventional path loading, and are unable to effectively utilize direct path loading due to lack of information about opaque types, such as XML types, which have alternative structures and are not defined to the client application.
Innovation Solution
A system where a type implementor registers routines with a client application, allowing the client to create arrays for opaque types, populate columns with values, and store data directly in the database without relying on the SQL engine, using a dispatch table to associate types with their implementors and generate data streams that conform to database data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional path loading is used to store semistructured data, then the SQL engine can process the data, but performance deteriorates and memory consumption increases
Solution Approach 1:
The patent extracts the SQL engine processing step from the data loading path. Instead of routing all data through the SQL engine, it creates a direct path that loads data straight into database blocks, bypassing the SQL engine for the actual insertion operation while still using it for validation and schema enforcement.
Solution Approach 2:
The patent segments the data loading process into distinct phases: validation phase (using SQL engine), transformation phase (converting to database blocks), and loading phase (direct insertion). This segmentation allows each phase to use the most appropriate method, improving overall efficiency.
2Productivity
If direct path loading is used to improve performance, then data can be loaded faster, but the system cannot handle opaque types like XML that have alternative structures not defined to the client application
Solution Approach 1:
The patent introduces an intermediary type implementation layer that sits between the client application and the direct path loading mechanism. This layer provides type information and structure definitions for opaque types, enabling the direct path loader to handle them efficiently without requiring the client application to have complete type knowledge.
Solution Approach 2:
The system performs preliminary actions by pre-defining type structures and creating type implementation modules before data loading occurs. These pre-prepared type definitions enable the direct path loader to efficiently process opaque types without requiring runtime type discovery or dynamic schema generation.
3Productivity
If the client application defines all type structures, then direct path loading can proceed efficiently, but the system lacks flexibility to accommodate new types without modifying the client application
Solution Approach 1:
The patent implements a dynamic type registration system where new types can be registered and their implementation modules loaded at runtime. This allows the system to adapt to new data types without requiring client application modifications, while still maintaining the efficiency benefits of direct path loading through pre-compiled type implementations.
Data Source
AI summary
Techniques and systems are disclosed for directly storing semistructured data in a database. According to one aspect, a client application reads data that comprises instances of a parent type. The client application invokes routines associated with the parent type. An array is created for storing instances of the parent type. These routines invoke routines associated with a child type of the parent type. An array is created for storing instances of the child type. The arrays are populated with values specified in the data. According to one aspect, some columns of the arrays may be populated with other values to be stored in hidden columns of database tables. The client application converts the arrays into a data stream that conforms to the format of the database's data blocks. The client application then streams the data to a database server, which writes the data blocks directly into one or more data blocks in the database.