Dimensional Soundscape

Dimensional Soundscape

by Christy, Esther, Iris, and Mavis

1) Overview of the game

This Apple Vision Pro MR project is an immersive experience designed for open-ended musical exploration and relaxation. 

Inspired by niche instruments and natural forms, each instrument is abstracted into simple, intuitive visual shapes. Instead of traditional musical training, users interact through effortless eye gaze and pinch gestures, allowing them to “play” these forms as if they were living sound objects. Users are free to wander, experiment, and create their own soundscape at their own pace. 

2) Development process

After coming up with our concept, our first course of action was to decide on what kind of visuals we wanted for our instruments. With Christy’s different idea sketches, we voted and ultimately decided on 4 different visual ways we could represent instruments for our immersive music system. Then, each person took on a specific instrumental sound (ambient sound, mbira, etc.) and paired it with a visual and developed that combination, with the spatial layout mimicking the visual sketch and the individual notes containing sounds from the chosen instrument. After some individual progress, we combined our individual files into one project, initially in separate immersive view files that could only be opened one at a time. But, considering the initial ideation and purpose of the app, we wanted to allow all of the instruments to play at the same time, so we ultimately combined all of our different files into one immersive view file. 

3) Successes

We unified multiple distinct musical experiences into a single spatial environment, giving users individual toggles to control each instrument's visibility. Every instrument occupies its own themed layer around the user's perspective — designed to be both immersive and accessible. Each visual layout was informed by the instrument's character and tonal texture, and users can pinch to play any combination simultaneously, building up layered soundscapes in real time.

4) Challenges

During the brainstorming stage, figuring out how to abstract existing instruments into forms that would be easier and more intuitive for people to understand was challenging. During development, the biggest challenge was that some team members doesn't have Xcode or Swift experiences, and we adapted through guidance from teammates & AI-assisted learning. Later when each person was developing the visuals for their own instrument,  Swift’s 3D capabilities were not as powerful as Unity’s, so we had a lot of discussion around visual presentation and rendering styles. In the end, we decided to use simple, nature-inspired color palettes that better matched the experience. In terms of interaction, each instrument was triggered based on the direction the user was looking, which made it difficult to play multiple notes at once. For the final integration process, combining everyone’s instruments into one shared was challenging both from a design and technical perspective and technically. We had to carefully consider and implement the spatial positioning and distance between different instruments in the environment. Another small hardware-related challenge was that the Apple Vision Pro could not comfortably accommodate glasses, so some team members couldn't see clearly while developing and testing the project.

5) What we learned
Christy - My main takeaways from this project is t he experience with Apple development system and understanding how this system allows for smoother interactions. Another major takeaway was learning more about music visualization and immersive musical experiences. Throughout the design process, we narrow down broad concepts while also explore how to abstract existing instruments into simplified visual forms. It helped me rethink how immersive technology can make music feel more approachable and engaging for a wider audience.

Esther - Interaction management and detecting in Native Swift is a lot easier than Unity for developing AVP projects! It was also much simpler to navigate project management and merging with Swift as well as we just had our individual files and could reference easily with an AppModel file.

Mavis - this is my second time developing for Apple Vision Pro with Swift’s visionOS libraries. It’s been fun and rewarding, especially when having everyone’s creation combined into one space. I learned that you can create organic 3D shapes with code only without needing an editor and that is very helpful when you don’t want to use default 3D assets like cubes or spheres.

Iris - I got way more comfortable with Xcode, RealityKit, and SwiftUI — coming in i had basically zero experience. 

6) Possible future revisions

We’d love to make the four instruments feel more connected, so users could layer mbira, glockenspiel, gu-zheng, and ambient sounds together to compose their own piece. Adding a recording and playback feature would let people save and share what they create. We also want to push the visuals further with richer particle effects, smoother transitions between scenes, and more responsive animations tied to the music. Eventually it’d be cool to make it collaborative, where multiple users in the same space could play instruments together in real time. Beyond that, we’d love to expand the instrument library to include more niche cultural instruments from around the world.