PocketStudio for mobile

Recording Scenes using Augmented Reality

PocketStudio native mobile app allows you to access all your projects’ 3D scenes and record them (create virtual cameras) using augmented reality. It is fun, easy and useful!

By trade, filmmakers master the art of cinematography which includes choosing camera angles to record scenes. However most of the time directors have to rely on animators to set virtual cameras which because 3D software are too complex for them to use which leads to frustration. Using a mobile phone to record 3D scenes, a device everybody uses constantly to shoot videos, removes this limitation.

How does it work?

All PocketStudio’s projects data are centralized on the cloud as a result of its collaborative nature. This is how any PocketStudio’s project can be accessed and edited at anytime from anywhere in the world but also from any device whether it be a computer or a mobile phone. All that is required is an application specifically built for the device and using PocketStudio’s proprietary collaborative engine in order to access and edit the project’s data seamlessly.

PocketStudio’s cloud infrastructure can thus be leveraged on different devices to take advantage of their own specific features, such as augmented reality on mobile devices.

All cameras recorded in AR are stored on the cloud as well and are thus instantly available for you to use in your projects. The process is entirely transparent to users thanks to PocketStudio’s built-in real-time collaborative editing technology.

To install the app see Installing PocketStudio for Mobiles.

To record a camera using PocketStudio’s augmented reality app, do as follows:

Edits made to the project from the phone will be visible in the desktop application in real-time and vice versa. The two applications are connected to the server and the project’s data are synchronized automatically and instanteously between all devices. See real-Time Collaborative editing on mobile devices for further details.

  1. Tap on the icon to launch the application.

  2. Sign in on the app using your usual username and password. If you can sign in on the mobile app while being already signed it on the desktop application. This works fine.

  3. Once you have signed in, the list of your projects will be listed to the screen. To open the project you want to record the scene of, tap on the project’s card.

  4. You will then be displayed with a list of 3D scenes contained in that project.

  5. Tap on the scene you wish to create a camera for. It is recommanded to not draw scenes on the phone that contain more than 1M triangles. See tips for preparing your 3D scenes for AR for further details.

    About data synchronization: if the data for the scene you wish to work with is not yet on the phone, the app will first download the scene’s data from the server. This can take some time depending on your internet connection speed and scene complexity. Once up to date, the scene will be automatically updated if a users changes it remotely. Data synchronisation happens automatically for you.

  6. The ARCamera view will show up. Track a horizontal surface by pointing the phone to a horizontal surface while moving it around. See tips for tracking real-world surfaces for further details. As soon as the app has detected a stable surface, a wireframe will be overlaid on top of that real-world surface in the camera view. If you now move the camera around, the grid should feel like it is sticking to the real-world surface. Does it work?

  7. Tap on the grid to drop the 3D scene into the view. The scene origin will be placed at the touch point. The 3D scene will be rendered on top of the camersa view without textures and lights (on mobiles devices, these features are not supported at the moment).

    If you open the sceen in PocketStudio for desktop, you will be able to see your AR camera moving live in the viewport. For further details see Watching your AR camera live in PocketStudio for desktop

    You can rotate, scale and move using gesture:

    • Dollying in/out: press your thumb and a finger together on the screen and pull them apart to dolly in (get closer to the scene - make the scene larger). Pinch the two fingers together to dolly out (get further away from the scene, make the scene smaller).
    • Rotate: swip a finger across the screen to rotate the 3D scene around.
    • Translate: press your thumb and a finger together on the screen and move them around to move the scene. XX DOES THIS WORK?? XXSee Tips for shooting in AR for further details.

    The 3D scene is not visible? The data of the scene need to be entirely downloaded before being drawn to the screen. If you tap on the screen to drop the 3D scene and that nothing happens, check that the data of the scene have been downloaded in full first. The second reason can be the size of the scene. It might not be visible at first if it is too big or too small.

  8. If your scene contains animations (see below to see what type of animation is supported by the app) you can play this animation before recording the camera. See animation support in AR for further details. Tap on the play button  in the middle bottom of the screen. The animation will loop.

  9. To record the camera, tap on the record button  located to the left of the screen. When the cameras is being recorded, the record button turns green and blinks . To stop recording tap on the record button again. When the camera is rolling, the animation will loop. You can thus create takes that are longer than the animation itself. As soon as you stop recording, the camera is stored on the cloud an can be used in your project right away.

  10. Go the PocketStudio’s desktop application. Open the scene you have created a camera for. Open the camera popover window. Cameras created with the phone have their names starting with AR and have a small icon  in the lower left corner on the camera card, indicating the camera is animated.

Tips for shooting in AR

Stabilizing the camera: cameras made with “hand-held” phones are shakier than professional cameras which are heavier. You can either use a phone/camera stabilizer while shooting and/or smooth the camera after you have recorded the scene, using the Smoothing camera property. See editing and Exporting your AR cameras in PocketStudio for Desktop.

AR tracking instability: when getting too close on an object or when making the scene very large in order to be able to be close to an object, the phone tracking might become instable: the camera makes random jumps and as result the scene jitters. The problem is somehow similar to the floating point precision error rendering artifacts you eventually get when dealing with scenes that are extremely large (in terms of distances between objects). Scaling up the scene (if you need to get close to a small object for example) can help solve the problen partially: by scaling the scene up you increase the scale in which the phone will be able to record and track small movements.

Animation support in AR

At this point in time, the mobile app only supports skinned animations (bones + skinning weight).

Tips for tracking real-world surfaces

If you find it difficult to track a real-world surface, try one of the following:

  • Make the real-world scene as bright as you can. When the scene is dark, it is much harder for ARCore to spot and track surfaces. Add more light to the scene.
  • ARCore will have a very hard time tracking uniform surfaces. For example it will be hard for ARCore to track the floor of a room that is completely white. ARCore needs variations of colors and shapes in order to reconstruct its environement in 3D and from there, compute the position of the phone in 3D space. While working in a clean bright environement is a good idea, be sure that your working environement is populated with a variety of objects or patterns the phone can track.
  • The shinier the surfaces the harder the track. Choose matte surface preferentially.

Tips for preparing your 3D scenes for AR

If you experience some stuttering while shooting a scene, the scene is probably too heavy to be rendered in real-time. If this happens, go back to the desktop application, decimate half of the geometry and load the scene on the phone. If the rendering is smooth great, if not repeat the decimation process. On high-end Android phones, the limit is around 1 million triangles.

Watching your AR camera live in PocketStudio for desktop

To watch AR cameras live from the desktop application do as follows:

  • In PocketStudio for desktop, open the scene that you are looking at on your mobile’s screen.
  • One of the cameras in the scene should be moving as you move the phone around. This is the live AR camera.
  • Open the camera popover by clicking on the camera icon on the toobar. In the camera list, look for the camera named Android_AR. To look through the camera, double click on its thumbnail.

Real-time collaborative editing on mobile devices

The same collaborative engine runs on PocketStudio for desktop and for mobiles. As a result when edits are made in PocketStudio for desktop, these changes are visible on the phone instantantly. You can add objects or move them around and they will pop up and change position on the phone in real-time. This also works the other around, however manipulation of objects is currently disable on mobile devices.

Reviewing AR cameras and using them in a project

To review and use your AR cameras in the project do as follows:

  • From PocketStudio for desktop, open the scene for which you created AR cameras in the Scene Editor.
  • Click on the camera icon on the toobar to display the camera popever.
  • Cameras created with the mobile app using augmented reality have their names starting with the prefix AR; to indicate that they are animated, a timer icon  is displayed in the bottom left corner of their card.
  • To look through an AR camera, double-click on its thumbnail as for any camera.

Smooth your AR camera

If the raw AR camera motion is too shaky you can smooth it out using the Smoothing camera property. The higher the values the smoother the camera path. Set the value to 1000 as a starting point and adjust the value until you get a result you like.

Exporting camras

To export cameras produced with the mobile phone see cameras: exporting cameras. When AR cameras are exported, PocketStudio doesn’t export the raw data but the filtered, smoothed data. To export the raw data set the AR camera Smoothing property back to 0.