Concept For Enhanced Sonic Realism

LinearLineman

I read this idea on the PianoWorld Pianoteq7 thread. I thought the tech stars here might care to chew this over a bit. Thanks to Propianist on the PianoWorld forum for permission to repost.

“The Sound Recording mode (on Pianoteq7) allows almost infinite variable positioning of the virtual mics, any position in virtual 3D space (X,Y,Z coordinates and aiming any orientation angle.)
More importantly, the same 3D freedom is allowed for the virtual "headphones" in binaural model.

Now, those of you who've ever tried a Sony PlayStation PSVR or Oculus Rift virtual reality headset, will know that you experience a much more convincing illusion of tangible 3D "reality" from the live head motion-tracking virtual reality experience, than you ever feel before from simply looking at flat "fixed" 3D stereoscopic photographs, or holograms, or even a "moving but still fixed" 3D stereoscopic video (like watching 3D Blu Ray) although it's motion video with a changing viewpoint perspective, your own natural human head movements (the way you instinctively interact with the world around you) have no effect on the "fixed" perspective the director is showing you in the 3D movie. Without that sensory feedback loop you still remain disconnected from the truly convincing immersive experience.

Now, suppose there was a way of combining live head-tracking for your stereo headphones as you play Pianoteq in binaural mode, whereby the movements of your own head could be mapped and applied in real time to reposition the virtual "headphones" of Pianoteq's binaural sound recording 3D modelling of the acoustic environment. Then, I believe, the grand piano would "appear" before you as an almost tangible 3D sound source. As you lean in closer, things get louder, as you bend to the right or left, you hear the treble or bass keys more closely. You could even stand up from your stool and lean forward putting you head inside the lid, over the strings and hammers, like you can with a real grand piano (lid open) and hear their vivid brightness, feel their proximity.

I think the same user experience quantum leap up going from "fixed" stereoscopic 3D photos and video, to totally immersive motion-tracked 3D stereoscopic virtual reality, could be an equally impressive quantum leap forward in the "sonic virtual reality" and our holy grail quest of trying to reproduce the feeling and sonic experience of sitting in front of a Steinway D concert grand piano.
I now think, lack of head motion tracking, is probably the stumbling block why playing digital pianos always feels fake, whereas real acoustic pianos actually exist in front of you.
Pianists play with huge expression and body language, rolling their torso and shoulders around, wild head movements, etc. - we're hardly ever sitting still, static and locked in one place. Our ears therefore perceive an ever changing stereoscopic sound perspective, which even with binaural HRTF recordings is not captured by one stationary fixed mic position.

This idea is not easy to implement and test, because obviously the required hardware is not commonplace.
Beyerdynamic Headzone Pro was on manufacturer's attempt at head-tracking headphones, but I can't think of many others... (off the top of my head, haha)
...although millions of people (including pianists like me) already do own VR headsets like PSVR and the 3D stereo Playstation cameras which connect via USB. Maybe we can use or adapt the existing hardware somehow? Perhaps the host computer could use a 3D camera like that and track motion if people stuck a few high-vis stickers onto their headphones(!) or perhaps you could just use your PSVR or Oculus headset while you play Pianoteq, and to that end, it would be great to have a 3D rendered Steinway D image in front of you too!

I'm sure this is the way forward, folks.
There are people out there who do know a lot about VR and head tracking and the cameras and technology for VR. It wouldn't take much genius to apply the same tech to a stereo audio only experience, with motion tracking of the virtual binaural piano model that Pianoteq already has. The question is, can the Pianoteq 3D coordinate parameters be manipulated live in real-time without causing audible glitches? Does Pianoteq need to pause and recalculate a new mathematical model for each new 3D position? Or could it just track the motion (perhaps using some 3rd party hardware that could turn the motion tracking into a stream of MIDI data which feeds Pianoteq's parameters, and maybe update the position 25 times a second (every 40ms) like video 25fps can fool the eye, maybe 25fps positional data update can fool the ear too, and Pianoteq could somehow update the maths model every 40ms with new positional data, or smoothly morph between the data values?

I'm sure this is the way forward, somebody needs to look into it, please... it's gotta be worth someone building a prototype and seeing how it feels to play a virtual piano with Virtual Binural Reality.”

NeonSilicon

I have the Abbey Road Studio 3 plugin from Waves and put it in the monitoring slot in Ardour. I have the head tracking bluetooth thing attached to my headphones. The effect is pretty damn terrific. The main thing is that it does actually make working with the audio more realistic and immersive. It feels more natural.

Playing even very non-realistic games in a VR headset with good tracking is mind-blowing. Silly stick figure characters feel somehow real.

CracklePot

If you can Midi map those parameters for at least head orientation, you could probably do a test setup using an iPhone or iPod.

You could use the device orientation tracking to drive the virtual head orientation. You just stick the device to the top of the headphones with tape or Velcro or whatever, and put the headphones on and look around.

It probably can’t track 3D position, unless you can use location tracking on the device, but I doubt it is even accurate enough.