Cognitive System Processing Embedded Code via NLP Annotation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language processing (NLP) based ingestion mechanisms fail to recognize and discern meaning from embedded computer code in natural language documents, leading to the exclusion of significant technical content, as they use a single parsing strategy for recognizable languages and drop unrecognizable segments like code snippets.
Innovation Solution
The system identifies and classifies embedded computer code segments using recognizable terms and context clues, generating metadata such as code segment annotations that provide natural language descriptions, translations, and correlations, allowing cognitive systems to process and utilize this content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single parsing strategy is used for natural language processing, then processing efficiency is maintained, but embedded computer code segments cannot be recognized or understood
Solution Approach 1:
The document is segmented into recognizable natural language portions and unrecognizable code segments. The system processes natural language segments using standard NLP parsing strategies while separately handling code segments through classification and annotation, thus maintaining processing efficiency for most content while capturing previously excluded code information.
Solution Approach 2:
Code segment annotations serve as an intermediary representation that bridges the gap between raw code segments and the NLP processing pipeline. These annotations provide natural language descriptions of code functionality, allowing the code content to be integrated into the overall document understanding without requiring the NLP engine to directly parse code syntax.
2Device complexity
If unrecognizable segments are dropped during NLP processing, then processing simplicity is maintained, but significant technical content is lost
Solution Approach 1:
The system extracts unrecognizable segments (code portions) from the natural language document stream. These extracted segments are then classified as code, annotated with natural language descriptions, and stored alongside the original document, thereby preserving significant technical content that would otherwise be discarded.
Solution Approach 2:
The system changes the representation parameter of code segments from raw syntax form to annotated natural language description form. This transformation allows code content to be retained and processed by the existing NLP infrastructure without increasing the complexity of the parsing mechanism itself.
3Productivity
If code segments are not processed, then system performance is maintained, but question answering accuracy decreases due to incomplete information
Solution Approach 1:
The system performs preliminary processing of code segments by classifying and annotating them before the main NLP processing completes. This preliminary action ensures that code information is prepared and integrated into the document representation, so that when questions are answered, the system has access to complete information including code content, thereby improving accuracy without significantly impacting overall performance.
Data Source
AI summary
Mechanisms are provided for processing natural language content having a computer code segment. Natural language content is processed using a natural language processing (NLP) engine and a segment of content within the natural language content is identified that is not recognized by the NLP engine. The segment is analyzed to determine whether the segment contains computer code and, if so, a code segment annotation for the computer code is generated that provides a natural language description of functionality of the computer code in the segment. The code segment annotation is stored in association with the natural language content and natural language processing is performed using the NLP engine on the code segment annotation to further process the natural language content.