USB 3D Camera Experimentation

USB 3D Camera Module

USB 3D Camera Module (parallal SBS)

UPDATED: 2021/09/29

For experimentation with 3D video, I purchased a USB Dual Synchronized 3D Camera Webcam Module from Amazon:

https://www.amazon.com/gp/product/B07MJFTYZ3

The camera module is intended for surveillance, but my uses are 3D stereoscopic capture. I'll discuss the different approaches I took to get the camera module to record 3D photos and videos. 

Camera Module

The camera module generates 960p 3D Stereo HD SBS output at 2560x960 pixels for both video and still photos. Video is 60 fps. The resolution is 1280x960 per eye, slightly better than the FujiFilm W3 3D camera video at 1280x720. There is no microphone for audio recording with the Webcam module and the camera lens is fixed focus M9 90 degrees. The image sensor is 1/3 inch CMOS OV9750. 

The camera lenses interaxial distance is 60 mm. This interaxial generates pleasing 3D images when the subject distance ranges from 2 to 6 feet depending on the background distance behind the subject. 

My task was to write or find software that will operate the camera over a wired USB connection. My goal is to capture 3D still images and video.

Write Code

First I wrote a code sketch with the Processing Foundation https://www.processing.org IDE using the Processing Video library to control the camera. Using Video capture sample code I could see the camera in a list available to the code, but could not turn on or display the video stream from the module. The code reports an internal data stream error.  This code does work for other 2D USB Webcams that I experimented on.

Here is the code for you to try with 2D USB cameras:

// USB 3D Camera video capture

// Java mode Processing IDE version 3.5.4

// Processing Video Library 2.0

// Library uses Gstreamer Version 1.16.2

import processing.video.*;

Capture cam;

int index = 1;

void setup() {

  //size(640, 240);

  //size(1280, 480); // for laptop display

  size(2560, 960);  // for USB 3D Camera screen size on a 2160p laptop display

  //size(1920,1080);

  frameRate(30);

  String[] cameras = Capture.list();

  if (cameras.length == 0) {

    println("There are no cameras available for capture.");

    exit();

  } else {

    println("Available cameras:");

    for (int i = 0; i < cameras.length; i++) {

      println("index="+ i + " "+cameras[i]);

    }

    // The camera can be initialized directly using an 

    // element from the array returned by list():

    println("using index="+ index+ " camera "+cameras[index]);;

    println("width="+width + " height="+height);

    cam = new Capture(this, width, height, cameras[index]);

    //cam.frameRate(60);

    cam.start();

  }

}

void draw() { 

  image(cam, 0, 0);

}

void captureEvent(Capture c) {

  cam.read();

}

Write Code Success

I reached out to Processing Video Library Github participants and Lucas Fredericks suggested a code fix in green highlight below, that works with the USB 3D Camera:

// USB 3D Camera video capture

// Java mode Processing IDE version 3.5.4

// Processing Video Library 2.0

// Library uses Gstreamer Version 1.16.2

import processing.video.*;

Capture cam;

int index = 1;

void setup() {

  //size(640, 240);

  size(1280, 480);

  //size(2560, 960);  // screen size that will fit 2160p laptop display

  //size(1920,1080);

  //frameRate(30);

  String[] cameras = Capture.list();

  if (cameras.length == 0) {

    println("There are no cameras available for capture.");

    exit();

  } else {

    println("Available cameras:");

    for (int i = 0; i < cameras.length; i++) {

      println("index="+ i + " "+cameras[i]);

    }

    // The camera can be initialized directly using an 

    // element from the array returned by list():

    println("using index="+ index+ " camera "+cameras[index]);;

    println("width="+width + " height="+height);

    //cam = new Capture(this, width, height, cameras[index]);

    cam = new Capture(this, width, height, "pipeline: ksvideosrc device-index=1 ! image/jpeg, width=2560, height=960, framerate=60/1 ! jpegdec ! videoconvert");

    cam.start();

  }

}

void draw() { 

  image(cam, 0, 0);

}

void captureEvent(Capture c) {

  cam.read();

}

Zoom

I tried the Zoom app on Windows 10 and with the settings menu found the 3D camera and saw video. I did not try it in a Zoom session. The camera module generates a SBS video stream.

VLC Media Player

Next I used open source VLC Media Player (version 3.0.16) running on Windows 10. In the main menu I initialized with menu item Media, Open Capture Device and made the settings captured in the screen shot below.

A mouse right click starts a recording (red record button) and again to stop recording (gray stop button). I played back the video full screen SBS and viewed with a stereoscope. Now I have a way to record videos in 3D using the Webcam.

Android Apps

I want to use my Android phone with the camera module. After trying several apps without success, I discovered UsbWebCamera app by Serenegiant on the Google play store. It has a subscription payment featue that may put off users, but I tried this version anyway, before discovering the UsbWebCameraPro version that I recommend to use instead.

With the paid versions you can set the camera resolution which you cannot do in the free version. This feature makes it possible to use this camera.

To connect the USB 3D Camera to my phone (Samsung S20) I used an OTG cable micro-USB male to USC-C male (Host end) manufactured by Cable Creation on Amazon. It is about 8 inches long, just the right size for use with my phone. 

Later I tried a less powerful Samsung S8 phone, but video was not captured successfully using the subscription version of the app. When I switched to the Pro version on the S8 I could record video successfully.

The UsbWebCamera settings are the following:

• Resolution 2560x960 at 60 fps
• Audio settings: Audio source Disabled (Disabled, mic, or USB Audio Input). Set to mic to use the phone's microphone during video recording.
• Max video capture duration - using default 30 seconds
• Still capture mode - PNG for uncompressed still image
• Play shutter sound for photos

Camera Mount on a Phone Case

To use the camera with my phone I mounted in with rubber bands on the phone case. Using the volume up button I can capture either still photos or videos in 3D when in photo or video mode respectively.

Camera Mount on the OWL Viewer

I mounted the camera module on the back side of London Stereoscopic Company OWL Viewer: Virtual Reality Kit magnetic holding plate. https://shop.londonstereo.com/VRKIT.html

Looking through the lens you can see the camera view in stereo.

OWL Viewer with mounted Camera module 

OWL Viewer with camera module and S8 phone

Photo Example

Photo from Camera unedited

Aligned and Edited Photo

Anaglyph Aligned Photo

Stereo Photo Maker Screenshot during Alignment

YouTube Example

Resting Cat

https://youtu.be/z4_WwwX1fKs  (no 3D metadata applied, you need a stereoscope to view)

https://youtu.be/s7NEPUrfZrY (added 3D metadata for viewing in anaglyph and 3D TVs)

To add 3D metadata I used the following code in a Windows batch file invoking ffmpeg.  See  https://www.ffmpeg.org/

ffmpeg -i %1 -map 0 -c copy -metadata:s:v stereo_mode=left_right "%~n1.mkv"

ffmpeg -i "%~n1.mkv" -aspect 1280:960 -map 0 -c copy -strict -1 "%~n1_3D.mp4"

pause

Advantages

1. The cameras are synced and run at 60 fps. This solves synchronization and wiring issues using separate dual cameras in a 3D camera rig.
2. No microphone - use a higher quality external microphone for audio recording or use the phone microphone
3. Near human eye separation distance

Disadvantages

1. Low resolution, small sensor
2. Fixed lens, not interchangeable
3. With so many Android phone variations, your phone may not work with this camera module

Improvements

• Build a small enclosure to house the camera module and clip on a phone case.

Conclusion

For me this was worthwhile experimentation because I learned more about 3D video and cameras. I'll be looking for a camera module like this in 4K and better lenses.

3D Photo Reel

I have been looking for alternative ways to show and distribute my 3D photos. The View-Master 3D reel inspired me to create an Android app that includes a stereo viewer and 3D media for presentation. I wanted an app that displays my media content on my glasses-free Rokit 3D Pro phone display. 

I wrote "3D Photo Reel Demo" Android app to explore the features most useful in the app viewer. One feature I wanted immediately was to be able to zoom (for very high resolution photo detail) and pan photos (for landscapes). Two finger expand and pinch perform zoom and shrink. Moving a finger in contact with the screen pans the image.

I noticed the 3D Photo app that came with the Rokit could not handle very high resolution photos because these photos were scaled down to a lower resolution for display. I want to display high resolution 3D photos and see the detail clearly. The disadvantage of working with high resolution photos is that the viewer is slow to load and process images. And you need a high end device with at least 4GB of RAM and fast processing hardware.

I thought the app might have some commercial applications too, for example, Real Estate listing 3D photos, product advertising, and virtual tours, to think of a few. Publishing photos as an app is a lower cost method than creating a View-Master reel.

I chose to include the 3D photo media with the app instead of downloading over the Internet.  With this design there is only a one time install setup without any additional steps for loading content. 

After getting the first version to work on the Rokit 3D Pro, I decided to expand the possible display devices. I added a mode selection feature for SBS, cross-eye SBS, half-width SBS, Above/Below, half-height Above/Below, Column (Rokit 3D) and Row Interlace, and Anaglyph variations. The SBS modes are for those skilled/lucky individuals who can free-view (left/right) or cross-eye view (right/left) side by side images.

Changing the mode for the display allows viewing on an external 3D TV or monitor (half-width SBS for example). To accomplish this I mirror the screen to my TV or connect via USB C. I also use Google ChromeCast Ultra connected to a 3D TV with good results to mirror a phone screen.

I have tested the app on various devices including the Sony XZ Premium (model G8142 modified to always display 4K with any app), Samsung S8 phone, and Samsung Chromebook Plus (configured to run Android apps). 

The app permits keyboard input either from a direct USB connection or a Bluetooth device/controller. A set of single key commands control various functions. I use this to help with debugging and it's also handy for remote control. The keyboard control features in the app are a super-set expanding what the GUI (graphical user interface) offers. The GUI is for novice first-time users while keyboard control is for advanced users. 

Function Key Mapping Tables

      Keyboard Input (USB or Bluetooth)

• First Photo - MOVE_HOME, MEDIA_REWIND, "F"
• Last Photo - MOVE_END, MEDIA_FAST_FORWARD, "L"
• Next Photo - VOLUME_UP, Space Bar, TAB,  ENTER, MEDIA_NEXT
• Previous Photo - VOLUME_DOWN, DEL, MEDIA_PREVIOUS, ESC
• Slideshow (start/stop) - MEDIA_PLAY_PAUSE, "S"
• Start Slideshow - MEDIA_PLAY
• Stop Slideshow - MEDIA_STOP
• Photo Information - "I", PERIOD
• Zoom Cycle - "Z" 
• Zoom In - "]"
• Zoom Out - "["
• Page In - PAGE_UP, ">"
• Page Out - PAGE_DOWN, "<"
• Reset Photo Position, Zoom, Parallax - "R", "0"
• Swap Left/Right or Right/Left - "X"
• Background Color Toggle- "B"
• Menu - "M"
• Mode Single Left - "1"
• Mode Single Right - "2"
• Mode Parallel SBS - "3"
• Mode Parallel SBS Cross-Eye - "4"
• Mode Parallel Half-Width SBS - "5"
• Mode Above/Below - "6"
• Mode Above/Below Half-Height - "7"
• Mode Column Interlace - "8"
• Mode Row Interlace - "9"
• Mode Red/Cyan Anaglyph - "A"
• Mode Color Anaglyph - "C"
• Mode Dubois Anaglyph - "D"
• Mode SPM Anaglyph - "J"
• Parallax Screen Mode - "P"
• Parallax Positive Step - "E"
• Parallax Negative Step - "W"
• Screenshot - "#"
• Title Screen - "N"
• Introduction Screen - "T"
• Exit - BACK

Mouse Input (Wired or Bluetooth)

• Left Button - Select On Screen Key or Move Photo
• Right Button - 
• Scroll Wheel - Zoom (not implemented)

Android TV Remote Control

• Next Photo - CHANNEL_UP
• Previous Photo - CHANNEL_DOWN
• First Photo - MEDIA_REWIND
• Last Photo - MEDIA_FAST_FORWARD
• Zoom Cycle In and Out  - DPAD_CENTER
• Move Photo Left - DPAD_LEFT
• Move Photo Right - DPAD_RIGHT
• Move Photo Up - DPAD_UP
• Move Photo Down - DPAD_DOWN
• Reset Photo Position, Zoom, Parallax - "0"
• Mode Single Left - "1"
• Mode Single Right - "2"
• Mode Parallel SBS - "3"
• Mode Parallel SBS Cross-Eye - "4"
• Mode Parallel Half-Width SBS - "5"
• Mode Above/Below - "6"
• Mode Above/Below Half-Height - "7"
• Mode Column Interlace - "8"
• Mode Row Interlace - "9"
• Mode Red/Cyan Anaglyph - PROG_YELLOW
• Mode Color Anaglyph - PROG_BLUE
• Mode Dubois Anaglyph - PROG_RED
• Mode SPM Anaglyph - PROG_GREEN
• Menu - CAPTIONS (subtitle)
• Slideshow (start/stop) - MEDIA_PLAY_PAUSE
• Slideshow Start - MEDIA_PLAY
• Slideshow Stop - MEDIA_STOP
• Photo Information - PERIOD
• Exit - BACK