Hadoop Job Container File Signing Engine Authentication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current Hadoop systems lack file-level authentication during job execution stages, making them vulnerable to security breaches within the intranet, as authentication is only performed at user check-in and not at the file level during data distribution and execution.

Innovation Solution

Implementing a file signing engine across name and data nodes to perform cryptographic digestion and content authentication of job containers, ensuring that files are authenticated at both user and content levels, thereby preventing man-in-the-middle and impersonation risks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If file-level authentication is added during job execution stages, then security is improved, but system complexity increases

Engineering Contradiction:

ImprovesecurityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The authentication process is segmented into multiple stages: initial user authentication at job submission, and subsequent file-level authentication during data distribution and execution. Each stage has specific authentication checks tailored to its requirements, allowing comprehensive security without overwhelming system complexity at any single point.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

User authentication is performed preliminarily at job submission stage using Kerberos, establishing trust before data distribution. This preliminary authentication enables subsequent streamlined file-level verification during execution, reducing the burden at later stages while maintaining overall security.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If multiple authentication stages are implemented, then security is improved, but processing time increases

Engineering Contradiction:

ImprovesecurityVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback mechanisms where authentication results from earlier stages inform later stages. Successful user authentication enables file-level verification, and verified file integrity confirms proper data distribution, creating an efficient multi-stage process that minimizes redundant checks.

Inventive Principle:

Principle #23Feedback

3Reliability

If file integrity verification is performed at each stage, then data integrity is improved, but computational overhead increases

Engineering Contradiction:

Improvedata integrityVSAvoidcomputational overhead

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

Data integrity verification is segmented into distribution-stage verification using CRC checksums and execution-stage verification using cryptographic hashes. Each stage uses verification methods appropriate to its requirements, ensuring data integrity without excessive computational overhead at any single point.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs partial verification at distribution stage using lightweight CRC checksums, then performs more thorough cryptographic verification at execution stage. This graduated approach ensures integrity without applying maximum verification effort at every single step.

Inventive Principle:

Principle #16Partial or excessive action

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

Enhances security by ensuring that files within job containers are authenticated at each stage of the process, preventing tampering and corruption, thus strengthening overall security within the Hadoop file system network.

Implementation Method 1

Implementing a file signing engine across name and data nodes to perform cryptographic digestion and content authentication of job containers

Methodology Applied

Scientific EffectCryptographic digestion:

Data Source

PatentUS11223484B1Enhanced authentication method for Hadoop job containers

Publication Date: 2022.01.11 EMC IP HLDG CO LLC

AI summary

Embodiments for providing content authentication of data in a network having a name node and a data node which may be in a Hadoop Distributed File System (HDFS) network, by associating each data set of the data with a first key identifying a job owner issuing a task for the data set, the first key being a session key that is randomly generated for the task, generating a second identity value for the data set on the data node, and performing the task if the second identity value matches the first key.