Skip to content

KNOWN_ISSUES.md

Latest commit

 

History

History
346 lines (260 loc) · 10.3 KB

KNOWN_ISSUES.md

File metadata and controls

346 lines (260 loc) · 10.3 KB

Known Issues and Limitations

This document provides detailed information about current system limitations, workarounds, and future enhancement paths.

Critical Limitations

1. Custom Packet Fragmentation Protocol

Issue: This implementation uses a custom packet fragmentation protocol to handle H.264 frames larger than the UDP MTU (1500 bytes). Standard video clients like ffplay, vlc, and mpv cannot decode this stream without modification.

Technical Details:

  • H.264 keyframes often exceed 10KB, requiring fragmentation
  • Custom PacketHeader structure includes: frameId, fragmentIndex, totalFragments, fragmentSize, isKeyframe
  • Each large frame is split into MTU-safe 1400-byte fragments
  • Standard clients expect either:
    • Single-packet frames (which would cause corruption)
    • Industry-standard RTP protocol with standardized fragmentation

Impact:

  • Cannot use ffplay udp://... or vlc udp://... directly
  • Server-side code is production-ready and tested
  • Requires custom client implementation for reassembly

Current Workaround: The provided scripts/client_example.sh demonstrates the expected client behavior:

# 1. Register with server
echo -n "HELLO" | nc -u -w1 SERVER_IP 5000

# 2. Receive stream (requires custom reassembly logic)
# Standard clients will show corruption on keyframes

Recommended Solutions:

Option A: RTP Migration (Recommended for Production)

  • Migrate to industry-standard RTP/RTCP protocol
  • Use FFmpeg's RTP muxer: ffmpeg -f rtp ...
  • Clients can then use: ffplay rtp://...
  • Adds ~10-15ms latency but provides:
    • Standardized fragmentation (RFC 6184)
    • Built-in sequence numbers and timing
    • Broad client compatibility
    • Loss detection and reporting

Option B: Custom Client Implementation

  • Implement reassembly logic in client application
  • Pseudocode: 
    std::map<uint32_t, std::vector<Fragment>> frameBuffers;

while (receiving) {
    PacketHeader header = receivePacket();
    frameBuffers[header.frameId][header.fragmentIndex] = data;

    if (allFragmentsReceived(header.frameId)) {
        Frame completeFrame = reassemble(header.frameId);
        decode(completeFrame);
        frameBuffers.erase(header.frameId);
    }
}
  • Requires custom client development (C++/Python/etc.)
  • Full control over buffering and latency

Option C: Disable Fragmentation (Quick Test)

  • Reduce bitrate to ensure keyframes < 1400 bytes
  • Set initial_bitrate=500000 (500 kbps) in config
  • Use encoder_preset=ultrafast for smaller keyframes
  • Significantly reduces quality
  • Only suitable for initial testing

Testing Status: Fragmentation logic implemented and structurally sound, but end-to-end client testing requires hardware setup.

2. Hardware Testing Required

Issue: This system was developed with production-quality code following industry best practices, but has not been tested on actual biomedical microscope camera hardware.

What Has Been Verified:

  • Code compiles successfully with all dependencies
  • Architecture follows proven multi-threaded patterns
  • FFmpeg/V4L2/NVENC APIs used correctly per documentation
  • Error handling and logging comprehensive
  • Docker containerization tested on compatible systems

What Requires Hardware Verification:

  • V4L2 camera capture on actual microscope cameras
  • YUYV to YUV420P color space conversion accuracy
  • NVENC encoding on target GPU hardware
  • End-to-end latency measurements
  • Network streaming to actual client workstations
  • Frame timing and synchronization under load

Risk Assessment:

  • Low Risk: Standard V4L2 API, should work with any compliant camera
  • Medium Risk: Camera-specific quirks (frame formats, timing, buffer counts)
  • Mitigation: Comprehensive error handling and fallback mechanisms in place

Recommended Testing Procedure:

  1. Camera Compatibility Test:

    # Verify camera is detected
v4l2-ctl --list-devices

# Check supported formats
v4l2-ctl -d /dev/video0 --list-formats-ext

# Test capture
v4l2-ctl -d /dev/video0 --stream-mmap --stream-count=100

  2. Build and Configuration:

    # Update streaming.conf for your camera
camera_device="/dev/video0"
camera_width=1920
camera_height=1080
camera_fps=60

# Build and run
./scripts/build.sh
./build/streaming_service streaming.conf

  3. Incremental Validation:

    • Verify camera opens without errors
    • Check frames are captured (look for "Captured frame" logs)
    • Confirm encoding succeeds (check for "Encoded frame" logs)
    • Validate network transmission (use tcpdump)
    • Test client connectivity

  4. Performance Validation:

    • Monitor console output for latency metrics
    • Check for frame drops (should be 0)
    • Verify GPU utilization stays below 80%
    • Test with multiple concurrent clients

Contingency Plans:

  • If camera format is not YUYV: Update src/camera/capture.cpp pixel format
  • If resolution differs: Adjust config file accordingly
  • If NVENC unavailable: System auto-falls back to CPU encoding
  • If timing issues occur: Adjust queue sizes in config

Medium Priority Limitations

3. Configuration Validation

Issue: While comprehensive validation is now implemented (v1.0.2), extreme edge cases may still exist.

What's Validated (as of v1.0.2):

  • Camera dimensions (320-3840 x 240-2160, must be even for H.264)
  • Frame rate (1-240 fps, prevents division by zero)
  • Bitrate ranges (100 kbps - 50 Mbps)
  • Logical constraints (min_bitrate <= max_bitrate)
  • Valid encoder presets
  • Port numbers (1024-65535)
  • Queue sizes (2-100)
  • Boolean value parsing

Potential Edge Cases:

  • Camera hardware may not support all "valid" resolutions
  • GPU memory constraints with high resolutions/bitrates
  • Network interface MTU variations
  • System resource limits (file descriptors, memory)

Recommendation: Always test configuration changes incrementally on target hardware.

4. UDP Packet Loss

Issue: UDP provides no built-in retransmission. Packet loss causes visual artifacts.

Acceptable Loss Rates:

  • 0-1%: Imperceptible (occasional minor glitches)
  • 1-5%: Noticeable artifacts on motion
  • 5%+: Severe degradation

Workarounds:

  • Use wired Gigabit Ethernet (typically < 0.1% loss)
  • Enable jumbo frames (MTU 9000) on local network
  • Implement Forward Error Correction (FEC)
  • Migrate to RTP with RTCP feedback

5. No Authentication or Encryption

Issue: Any client on the network can connect and view the stream.

Security Implications:

  • Sensitive biomedical data transmitted in clear text
  • No access control mechanism
  • Vulnerable to eavesdropping on shared networks

Workarounds:

  • Deploy on isolated VLAN
  • Use VPN for remote access
  • Implement SRTP (Secure RTP) for encryption
  • Add authentication handshake before registration

Minor Limitations

6. Single Camera Support

Current: Hardcoded to one camera device.

Extension Path:

  • Add camera ID parameter to CameraCapture constructor
  • Create multiple capture/encode/network thread groups
  • Assign different UDP ports per camera
  • Minimal code changes required

7. Linux-Only (V4L2 Dependency)

Current: Uses Linux V4L2 API for camera access.

Porting Requirements:

  • Windows: Replace V4L2 with DirectShow or Media Foundation
  • macOS: Replace V4L2 with AVFoundation
  • Core encoding/networking code is platform-agnostic

8. YUYV Color Format Assumption

Current: Assumes camera outputs YUYV422.

Alternative Formats: If camera outputs different format, update src/camera/capture.cpp:

// Change pixel format
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_MJPEG;  // or V4L2_PIX_FMT_RGB24, etc.

// Update swscale conversion
sws_ctx_ = sws_getContext(
    width_, height_, AV_PIX_FMT_YUYV422,  // Change source format here
    width_, height_, AV_PIX_FMT_YUV420P,
    SWS_BILINEAR, nullptr, nullptr, nullptr
);

Future Enhancements

High Priority

  1. RTP/RTCP Implementation

    • Replaces custom fragmentation
    • Standardizes protocol
    • Enables standard client compatibility
    • Estimated effort: 2-3 days

  2. Hardware Testing Suite

    • Automated compatibility checks
    • Performance benchmarking
    • Regression testing
    • Estimated effort: 1 week

  3. Custom Client Application

    • Implements fragment reassembly
    • Provides optimal playback settings
    • Cross-platform (Qt/Electron)
    • Estimated effort: 1-2 weeks

Medium Priority

  1. WebRTC Support

    • Browser-based clients
    • Built-in NAT traversal
    • Lower latency than RTP
    • Estimated effort: 2 weeks

  2. Authentication & Encryption

    • TLS/DTLS for encryption
    • Token-based authentication
    • SRTP for media security
    • Estimated effort: 1 week

  3. Metrics & Monitoring

    • Prometheus exporters
    • Grafana dashboards
    • Real-time performance visualization
    • Estimated effort: 3-4 days

Low Priority

  1. Multi-camera Support
  2. Cross-platform Camera APIs
  3. Hardware-accelerated Color Conversion
  4. Lock-free Queue Implementation

Testing Recommendations

Before Deploying to Production

  • Hardware compatibility verified on target camera
  • End-to-end latency measured and acceptable
  • Frame drops measured (should be 0%)
  • GPU utilization stays below 80%
  • Network tested with expected client count
  • Configuration validation tested with invalid inputs
  • Docker deployment tested with GPU passthrough
  • Client connectivity tested from target workstations
  • Documentation reviewed and updated
  • Backup/recovery procedures established

Acceptance Criteria

  • Latency < 70ms (measured with hardware timestamps)
  • Consistent 60fps (no frame drops over 1 hour)
  • Bitrate adapts correctly to network conditions
  • System recovers gracefully from camera disconnection
  • No crashes with malformed configuration files
  • NVENC fallback to CPU works correctly

Getting Help

If you encounter issues:

  1. Check Logs: All components log detailed information to console
  2. Verify Configuration: Run with valid streaming.conf
  3. Test Incrementally: Isolate camera, encoding, network separately
  4. Review Documentation: See README.md, QUICKSTART.md, TROUBLESHOOTING sections
  5. Open GitHub Issue: Provide logs, configuration, hardware details

Document Version: 1.0.2 Last Updated: 2025-12-16 Status: Pre-production (requires hardware validation)