Robotics Log Sharding for Parallel Onboard Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotics log formats and processing workflows are inadequate for handling large data volumes and distributed computation, leading to inefficient data storage and processing in the cloud, resulting in delays and high networking costs.
Innovation Solution
Onboard robotics processing of logs into sharded data packages, enabling parallel processing and efficient use of compute resources, with subsequent offloading of metrics and log slices for faster access and reduced data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If robotics logs are offloaded in their entirety into the cloud and processed serially, then data storage is simplified, but processing time increases and networking bandwidth usage increases
Solution Approach 1:
The patent divides robotics logs into discrete time-based segments called shards. Each shard contains log data for a specific time window and can be processed independently. This segmentation enables parallel processing of multiple shards simultaneously, dramatically reducing total processing time while maintaining manageable data sizes for cloud storage.
Solution Approach 2:
The system performs preliminary processing of log data by creating shards and generating metadata (including min/max timestamps and data counts) before offloading to the cloud. This preliminary organization allows the cloud system to quickly identify and process only relevant shards without handling entire log datasets, reducing both processing time and bandwidth usage.
2Volume of stationary object
If robotics logs are offloaded in their entirety into the cloud, then onboard storage requirements are reduced, but networking bandwidth usage increases
Solution Approach 1:
The system extracts only the essential metadata (min timestamp, max timestamp, data count) from each shard and stores this extraction in the cloud index. This allows the cloud system to determine which shards are relevant without transferring the actual log data, significantly reducing bandwidth usage while maintaining reduced onboard storage requirements.
Solution Approach 2:
By pre-processing logs into shards with embedded metadata before cloud transfer, the system enables selective data transfer. Only shards that contain relevant time windows need to be transferred and processed, while others can be ignored based on metadata comparison, reducing overall bandwidth consumption.
3Device complexity
If logs are processed serially in the cloud, then processing simplicity is maintained, but processing speed decreases
Solution Approach 1:
The patent segments logs into independent time-based shards that can be processed in parallel. Each shard is self-contained with its own metadata, allowing multiple cloud workers to process different shards simultaneously without coordination overhead, thereby maintaining processing simplicity while dramatically increasing processing speed through parallelization.
Solution Approach 2:
The system transforms the processing model from a single-dimensional serial process to a multi-dimensional parallel process by introducing time-based shard segmentation. This allows the cloud system to process multiple shards across multiple dimensions (different time windows, different robot instances) simultaneously, increasing productivity without significantly increasing system complexity.
Data Source
AI summary
Processing of robotics log data may be performed onboard the robotics itself and log data may be converted to a form that is more suitable for parallel processing. In one aspect, robotics log data may include multiple channels. These channels are received and are used to generate data packages referred to as shards. Each shard is associated with a time window. For example, each shard may contain the robotics log data from all channels for the relevant time window. Additional data, such as an index and metadata, may be included with the shards so that each shard may be processed independently. Each shard is independently consumable and, as a result, the shards may be processed in parallel to accomplish some task. Furthermore, the creation of shards and the subsequent processing of shards may occur onboard the robotics itself.