Friday, October 11, 2019

3D Camera With Bluetooth Remote Control

Twin Smartphone 3D Camera, Bluetooth Remote Controller, and OWL viewer

Twin Smartphone 3D Camera Lens in OWL viewer

Twin Smartphone 3D Camera in OWL viewer

During the summer 2019, I went to my first National Stereoscopic Association convention, 3D-Con, in Akron, Ohio, USA. It was a great experience. I learned a lot about 3D stereo photography, saw wonderful 3D Art, photos, and videos. I met expert 3D enthusiasts who graciously shared their knowledge and Art. 


3D Camera Phone Rig


Before attending the convention I was experimenting with making a 3D camera rig using twin smartphones mounted with magnets. During the convention my ideas for this 3D camera project came together and I was able to complete a working system upon my return home.

Convention vendors sold all kinds of 3D related gear and Art. At the London Stereoscopic Company booth they sold the "OWL - Stereoscopic Viewer" for viewing stereo cards. I bought my OWL viewer on-line a couple of years ago, but I found out they now offer a Virtual Reality kit that uses a magnetic board for securing a phone to the viewer. I bought the magnet board holding plate from LSC to hold my twin phones in the OWL.

My 3D camera rig consists of two identical Samsung Galaxy S6 smartphones mounted in portrait mode, side by side, in the OWL viewer and held in place by the magnet. I placed a adhesive metal sheet bracket on each phone case to cling to the magnet. One of the S6 phones was used before I upgraded and the other I bought used on EBay. I cut two openings in my OWL viewer for charging cables and a hole for a 1/4 inch tripod mount.

In the OWL the phone camera lenses appear above the OWL frame unobstructed. Looking through the OWL lenses you can see in stereo the camera screen images. This turns out to be a good way to preview the depth of the scene you want to capture. The preview helps you determine if you are too close or too far from your subject besides helping with composition framing. The OWL rig mounted smart phone camera lens separation (inter-axial) is 75 mm in portrait mode. By coincidence the FujiFilm FinePix Real 3D W3 camera has the same lens separation distance. See the rig photos above.


Camera App

Open Camera Remote App Icon

I could have used the Samsung camera app that came with the phone, but I wanted a wireless trigger for both focus and shutter controls.

At the convention there was a discussion group about using smartphones to capture 3D. The Hyper3DPhone app was identified as a possible app for twin Android smartphone cameras. It works by pairing two phones over a Bluetooth connection and triggering one phone sends a message to the other to also take a picture. Although it worked well for triggering two phones nearly simultaneously, its weakness was that I had to touch the camera screen to take a photo and there was a lack of camera control features I use in other camera apps.

For several years I have been extending features of the open source Android Open Camera app to include wireless remote control. My modified app is named Open Camera Remote and you can install it from the Google Play Store . My app uses WiFi broadcast messages to trigger focus and shutter remotely on multiple phones connected to a local area network. Newer versions of the camera app permit control using wired and wireless Bluetooth keyboards, mice, and other game controllers. In addition I added a tiny HTTP Web Server in the app retrieve photos and videos taken by the camera and stored on the phone. This was my choice for a camera app in my 3D OWL rig.


Wireless Focus/Shutter Trigger

3D Camera in Owl Viewer with Bluetooth controller work in progress.
With this 3D camera rig I use two Bluetooth camera controllers. The Open Camera Remote app also allows shutter control over WiFi for which I would need access to a local private WiFi network, and a third phone to use as a remote control trigger. WiFi has a longer range advantage over Bluetooth. 


Bluetooth Camera Remote Shutter


I used two Bluetooth camera shutter remotes as pictured above for my DIY controller. I needed two, one for each smartphone, because two phones cannot share the same Bluetooth remote connection. I took the circuit boards out of the remote and wired them to switches and battery power in a case shown below. 


Home-made Bluetooth Remote for Twin Cameras


The remote control circuit boards were wired to trigger simultaneously with the push of a single button switch. Two diode OR circuits permit a single switch for focus and another OR circuit for shutter release to bypass the built-in controller switches. The controller's large button does focus (volume up, Android key code 24) and the smaller button does shutter release (enter, Android key code 66). The camera app settings have to be set to use the volume up switch for focus.

Synchronization of the cameras is just OK with variable results. Good synchronization of the two cameras depends mostly on how long it takes for the cameras to focus before taking a picture. To make sure focus does not interfere with synchronization, first I touch the screen to select my subject that the camera will focus on. When both cameras are in focus, then I press the shutter button to take a picture. I use the focus switch to focus repeatedly on the subject to make sure focus is achieved before pressing the shutter control switch.

I got better synchronization when I placed the camera apps in manual focus mode. But I found it too cumbersome to make sure the focus distance was the same for each camera.

Here is a video demonstrating the 3D Camera in operation.





Viewing 3D Using the OWL

After taking a picture you can view it in 3D with the OWL by tapping the right bottom corner preview icon on each phone to launch your photo viewer of choice. I use the app I wrote 3D/VR Stereo Photo Viewer in 2D mode for viewing my photos. Other photo viewing apps work just as well. The Back button on the phone returns to the camera app.


3D Smartphone Camera Tripod Mount in OWL 3D Viewer

Since writing this blog last October, it's now mid May, 2020, I have used the rig with twin Samsung Galaxy S8 phones that have a much better camera with improved results. I also experimented with the 3D Stereo Photo Viewer app running on a Rokit 3D Pro phone to retrieve photos from the twin phones for display on its glasses free 3D display screen. That's another future blog posting.

13 comments:

  1. Very interesting, Thanks for taking the time to do this blog

    ReplyDelete
  2. Thanks for the details on the making of the rig. It is a clever design. Have you used it for people pictures? What is the interaxial of the rig?

    ReplyDelete
  3. Thanks Dylan,
    The interaxial is 75 mm. I did use the rig at an indoor Cosplay portrait meetup shoot last October, got some good shots, but the lighting was not very good, with slow shutter speeds and high ISOs. I used two Samsung S8. Next time I would be better off adding a constant light off camera LED light stand.

    ReplyDelete
  4. I recently came across your very interesting write-up and took the plunge and ended up with a stereo rig made of 2x Huawei Mate 8 phones. (I have been a long time user of SDM, but decided to give phones also a shot.) So far, very happy with Open Camera Remote, but not so much with triggering and synchronization. Regarding triggering, I personally find wired to have yet another device in my hands, so I've been thinking about other options. One idea I had was to pair the phones over bluetooth and mirror touch events from one phone to another. Would this be possible given the current bluetooth triggering implementation? Also, any ideas what to do to improve sync? I came across a project called sotfwaresync, btu so far no luck with that. Also, it's very CPU intensive... Waiting for your reaction. Thanks!

    ReplyDelete
  5. Thanks Andrei, I find Bluetooth has poor sync compared to wire triggers over the USB phone connection (using two wired mice for example). I think Bluetooth introduces unpredictable delays receiving wireless signals. If both phone camera can be focus locked then any delays acquiring focus should not figure into camera synchronization. I did try Google github softwaresync with the Samsung S8 but it did not sync as well as claimed by the author's testing using Google Pixel phones. As for your idea about touch events, maybe two Bluetooth or wired mice joined with common switches might work, to get near simultaneous touches to focus and trigger the shutter. An Arduino board may be able to generate commands over a pair of USB wired connections using common mouse switch and cursor positioning hardware.

    ReplyDelete
    Replies
    1. Thanks! Regarding sync:

      - I experimented a bit with Wifi remote capture. The "focus hold" seems to introduce unpredictable (and unexpected?) delays. Btw., how is this option supposed to work? I've got better results without focus hold, but with focus lock on the camera. So, the steps are: enable focus lock in open camera, touch screen to focus on each camera, then capture using wifi remote. Sync improved quite a bit, still not very happy, though. What is your process for getting best capturing sync?

      - regarding touch to sync. I did not yet experiment with BLE remotes, since I have none. What I would like to try is to pair phones over bluetooth and then send any GUI element touch event from one phone (server) to the other (client). Unfortunately, I cannot pair two phones using the BLE option in open camera remote. Any idea why? Would this be difficult to achieve starting from your current code? I have seen for instance such options: https://github.com/itanbp/android-ble-peripheral-central or https://www.bignerdranch.com/blog/bluetooth-low-energy-on-android-part-1/

      I would like to get rid of any additional HW device remotes, wires, etc.

      I would also like to help implementing the above option, but I have no Android programming experience. I do have C/C++, arduino, low-level HW programming experience, though ;).

      Maybe we could take this over email? My address is jalba.andrei AT gmail.

      Thanks!

      Delete
    2. - I think you found the best way to sync with WiFi remote capture, similar to what I experienced.

      - BLE is a feature in the base Open Camera app and I did not experiment with it.

      - Have fun experimenting!

      Delete
  6. Andrei, do you know about the Gosky cable release? It, with a splitter cable, allow you to trigger both phones at the same time. The quality of sync is acceptable in most cases, IMO. It may be good enough to allow flash sync.

    ReplyDelete
    Replies
    1. Thanks. Yes, I did know about this option, but I would really like to not use any additional device, if possible ;).

      Delete
  7. For what is worth, I have a first version of bluetooth-paired open camera remote:
    https://www.dropbox.com/s/dhksz8plmvfi66c/app-remote-debug.apk?dl=0

    This is work in progress, but if anyone wants to test, do the following:
    -install app on both phones;
    -Set one phone as server, that is, it will capture events and apss them via BLE to the other phone, the client. To set as server, start app, go in menu->more camera controls->Bluetooth LE remote control and enable option "Enable BLE server". Exit menu.
    -On the server, proceed as usual to focus then touch camera capture. If all is good, a capture event should be fired on the client phone. For now, also zoom events are transmitted to client.

    I would be very curious if anyone can test and compare sync quality. Please report back here. Thanks!
    - On the other phone, start app, go in menu->more camera controls->Bluetooth LE remote control and set option Enable BLE remote control. Then, click "select remote device" and scan for the other phone. Once found, pair it and exit menu.

    ReplyDelete
  8. Hey Andrei,
    Nice work!

    I was able to get your BLE remote control working on two Samsung S6 phones. I had to redo client SCANs whenever either app went into the background, i.e. it was no longer active. You could add code to restart the BLE for the selected device when the app returns to active state (onResume).

    I invoked zoom simultaneously, but a focus request on the server was not sent to the client phone (a very desirable feature for sync). What other server touch requests can be passed to the client, besides zoom and shutter release?

    I did not get good sync. I got better sync, but not ideal, when I first touch focused both phones, and then did a shutter release.

    I assume only two phones can be paired together, since the connection is via BLE.

    For a future build, please change the apps package name and app name in the manifest etc. so I don't have to delete existing Open Camera Remote apps.

    There is an app, https://play.google.com/store/apps/details?id=fr.stereoscopie.Hyper3DPhone that pairs two phones for 3D using Bluetooth, but I do not know if it uses BLE. It has a delay parameter to improve sync.

    ReplyDelete
  9. Hi Andy,

    I updated the app, making an actual attempt at synchronizing the cameras. Idea is that now, when a shutter event is fired by touching the photo button on the server the following happens:
    - client starts taking a photo
    - when the execution reaches deep in the CameraController class, just before actually capturing the photo, it enters an infinite sync loop waiting for a server-side message.
    - in the meantime, the server also starts taking a photo
    - server executes a delay (500 ms currently, adjustable from GUI)
    - when the execution reaches deep in the CameraController class, just before actually capturing the photo, send a message to client to break the sync loop.
    - server waits for 2ms (adjustable from GUI, an estimate including BLE transmission delay + additional overhead on the client-side).
    - both cameras capture a photo.

    It is possible that the client never breaks the sync loop; if this happens increase the server delay to more than 500 ms.

    Additionally, if the client is set to manual focus mode, the focus distance is passed from server.

    ====

    To your comments:

    > I had to redo client SCANs whenever either app went into the background, i.e. it was no longer active. You could add code to restart the BLE for the selected device when the app returns to active state (onResume).

    Strange, I did not notice this. Will have a look next time.

    > I invoked zoom simultaneously, but a focus request on the server was not sent to the client phone (a very desirable feature for sync).

    Pinch zoom and slider-based zooming events should get transfered from server to client. Please let me know if this still does not work.

    > What other server touch requests can be passed to the client, besides zoom and shutter release?

    Focusing. I also intend on mirroring exposure values in the next update.

    > I did not get good sync.

    Please test the new version. My impression, without actually mesuring, is that results are much more consistant/repeatable and seem to by in better sync :). This is just judging from the sound and animation...

    > I got better sync, but not ideal, when I first touch focused both phones, and then did a shutter release.

    Indeed, this is the alternative way of manipulating the focus.

    > I assume only two phones can be paired together, since the connection is via BLE.

    Correct.

    > For a future build, please change the apps package name and app name in the manifest etc. so I don't have to delete existing Open Camera Remote apps.

    Will do that ;).

    > There is an app, https://play.google.com/store/apps/details?id=fr.stereoscopie.Hyper3DPhone...

    I knew about it, but is by far not as feature-rich as open camera...

    ===

    If you have any means of testing sync, would be great to hear.


    Also, if you have suggestions, I'd like to hear them.

    ReplyDelete