In our Art and Machine Learning final project, we wanted to explore how computers visualize and perceive the world.
Embracing naïvety, we imagine what a neural net might dream on some random night during its training to become more aware of its surroundings.
In the process of creating and exploring this VR experience, we asked ourself how this computer perception relates to our own, and whether or not there is something universal about the nature of perception.
From a conceptual standpoint, we explored Polyfauna and the interactive experience it affords.
From a machine learning standpoint, we researched autoencoders and looked specifically at architecures that work at representing pointclouds.
Basic architecture of an autoencoder. The purpose is to efficiently learn a representation of point clouds.
Polyfauna inspired this project. As the user propels forward in space, the experience evolves based on the objects they approach.
The product is a VR experience where the user interacts with point cloud. These point clouds spawn and evolve. The nature of how they transform, and at what speed they transform is affected by the user’s camera orientation.
Producing this involved several pipelines and points of coordination among the team. Moving parts involved training an autoencoder on custom point clouds, and then finding an appropriate method to interpolate from the latent space.
A normal autoencoder was applied on a large point cloud set of a dancing figure. We chose a computer screen as an ideal object to interpolate against, particularly to compliment our concept of computer vision.
An autoencoder was trained on custom pointclouds. Here are some results from interpolating the latent space produced. Dimensionality reduction was minimalized but still visible.
We used the PCX plugin to load point clouds into Unity. Formatting the pointclouds to size properly on Android was a challenge.
A script ensures random terrain prefabs spawn in front of the user, thus infinite terrain. In an early iteration we wanted more of a Tron-like aesthetic.
On the Unity side, the team used PCX plugin to load point clouds into the engine. Interpolation frames were precomputed and loaded into the scene to prevent any lag issues with loading pointclouds in real time as new objects are spawned.
Generative terrain was used, ensuring that new terrain spawns as the user propels forward. Additionally, old terrain is recycled as the user moves away from it. A spawner was made ensuring that point clouds randomly appear in front of the user's camera. These also recycle based on distance.
We had many participants on exhibition day, and it was interesting to see all the different reactions to the VR experience. Many people told us it was their first time using Cardboard for VR. In that sense, people were splitting their engagement between the technology and the art. Regardless,our biggest bottleneck of the day was getting people to take off the cardboard so that we could give everyone in line a chance to participate.
Some motion sickness issues seem probable as users spend extended time in the experience, or if the phone in use overheats. This issue is preventing us from publishing the app on Google Play. This is likely not easy to resolve without upgrading platforms to something that has a robust graphics card. Although we made the graphical experience as simple as possible, point clouds inevitably require some intense calculations to render properly.