Demosaicing Process for HybridEVS Camera: Reconstruction of High-Quality RGB Image from HybridEVS Data Using Deep Learning Model

Event Vision Sensors (EVS) operate at the pixel level to detect temporal contrast changes beyond a predefined threshold. Unlike traditional CMOS image sensors (CIS), EVS inherently provides data compression functionality, enabling high-speed, low-latency data capture while operating at lower power levels. This makes EVS particularly promising for applications in object tracking, 3D detection, and slow-motion imaging.

The Hybrid Event-Based Vision Sensor (HybridEVS) is a notable advancement in this technology. This novel hybrid sensor integrates the Quad Bayer Color Filter Array (CFA) with event-based vision techniques. As illustrated in Figure 1, within a 4x4 block of pixels, two event pixels capture event signals, while the remaining pixels provide color information. However, the challenge lies in the fact that event pixels cannot capture color and texture information, making the demosaicing process for HybridEVS more complex.

Additionally, the manufacturing process of these sensors can introduce pixel flaws, resulting in defect pixels that exhibit significantly divergent values from their unaffected counterparts. The presence of event pixels and defect pixels further complicates the demosaicing task for HybridEVS cameras.

In the realm of image processing, achieving high-quality color reproduction from raw HybridEVS patterns remains a challenging task. To tackle this, we present a cutting-edge approach: Multi-Stage Fusion Demosaicing with Integrated Pixel Attention and Residual Learning. This innovative model combines the latest advancements in computer vision to enhance image fidelity and detail.

The Workflow of Our Demosaicing Model

1. Conversion from HybridEVS to RGB:

Our process begins with transforming the raw HybridEVS pattern image (in RGB) into the standard RGB color space using a HybridEVS2RGB conversion method. This initial step sets the stage for more refined processing.

2. Joint Processing through Fusion:

The next phase involves fusing the converted RGB image with the raw HybridEVS input. This fusion is processed through a novel architecture that integrates Self-Calibrated Convolution with Pixel Attention (SCPA). This technique significantly enhances spatial and spectral coherence, ensuring the details of the image are preserved.

3. Advanced Denoising:

Simultaneously, the raw HybridEVS image is passed through a sophisticated denoising block. This block features four inverse convolutional layers and an advanced attention mechanism designed to suppress noise while maintaining critical image details effectively. The result is a three-channel denoised image that complements the fusion process.

4. Feature Extraction and Downsampling:

The denoised image, along with the output from the SCPA block and the converted RGB image, is then processed through a downsampling layer. This step reduces computational complexity and enhances feature extraction, making the subsequent processing more efficient.

5. Residual Learning for Enhanced Reconstruction:

The downsampled features are fed into a residual learning block, which includes a series of residual group blocks and a residual channel dense attention block. This architecture captures both local and global contextual information, allowing the model to reconstruct images with greater accuracy and detail.

6. Final Image Reconstruction:

The output from the residual learning block is then added back to the input image. This composite is upsampled to produce the final demosaiced image, delivering high-quality color reproduction and clarity.

Key Advantages of Our Approach

- Enhanced Spatial and Spectral Coherence: 

Self-calibrated convolution with Pixel Attention ensures that both spatial and spectral details are preserved.

- Effective Noise Suppression:

Advanced denoising techniques reduce unwanted noise while maintaining essential image features.

-Efficient Feature Extraction:

Downsampling and residual learning improve computational efficiency and detail extraction.

- High-Quality Reconstruction:

The final upsampling step delivers a demosaiced image with enhanced clarity and color fidelity.

We Are Hiring!

クーガーは自律的で大胆なチャレンジを支援し、それぞれの個性を生かした技術と創造性の追求ができる場を目指しています。

ぜひ採用ページからエントリーください。カジュアル面談も実施しています！