Sony Files Shocking Patent Application for Monitoring and Counteracting Motion Sickness

Nausea.  Retching.  Headaches.  All-too-familiar symptoms for those unfortunate individuals who have experienced “virtual reality sickness” (and also, for those who have been to a Kenny G concert).  Often compared to motion sickness, or kinetosis, the condition can arise in a user when there is a disagreement between “visually perceived” movement and the vestibular system’s sense of movement.  In short, it is a major challenge for VR content developers, and also one of the reasons why several HMD companies have recommended that games run at a minimum of 90 FPS.

To deal with this problem, Sony has filed a patent application entitled132 “Motion Sickness Monitoring and Application of Supplemental Sound to Counteract Sickness.”  The patent application was originally filed on February 5, 2015 and published today.  To detect motion sickness, the application describes monitoring a user’s body motion, pupil motion, gaze, head motion, balance, or “by tracking facial expressions of the user that may indicate motion sickness (e.g., a gagging motion, sticking out the tongue, etc.).”

When motion sickneSony AMS FIG 2ss is detected, the system takes action by delivering “supplemental sound,” reducing the intensity of the game or vibrating the headset.  Sony’s application claims that supplemental sound has a therapeutic effect, and in particular, describes a system to deliver “bone conduction audio” by transmitting audible or ultrasonic sound waves through a portion of a user’s skull close to the ear.  These vibrations can be generated by a device incorporated into the strap of the HMD.

Sony’s application goes on to describe further embodiments for battling motion sickness including transmitting microwaves that are “safe for the inner ear,” infrared stimulation and “small electric shocks.”

The use of vestibular stimulation is a known technique to battle motion sickness in virtual reality environments (over fifteen years old), and some parties have already obtained patents for it. (See here and here.)  However, it will be interesting, to say the least, to see if Sony can eventually introduce this technology into its commercial PSVR headsets.  Until then, VR users will just have to keep a case of ginger ale handy.

 

SUPER SMASH BROS VR EDITION: An Update on ZeniMax v. Oculus

zenimax v. oculus title

The legal battle between two heavyweights over the Oculus Rift continues between ZeniMax Media and id Software, in one corner, and Oculus and Facebook, in the other.  In a court filing this week, the two sides reportedly failed to reach a resolution in a recent court-ordered mediation.

The lawsuit also made headlines in December 2015 when Facebook’s CEO, Mark Zuckerberg, was ordered to sit for a deposition in the case.

If you’ve been living under a rock for the past two years, I’ve provided a summary below.  (I also recommend reading the complaint, a lengthy, but fascinating read for those interested in the VR industry.)

In May 2014, ZeniMax Media, parent company of id Software and Bethesda Softworks, filed suit in a federal court in Texas, asserting claims of trade secret misappropriation, copyright infringement, breach of contract, unfair competition, unjust enrichment, trademark infringement and false designation against Oculus and Palmer Luckey.  (Zenimax has since amended its complaint to add Facebook and John Carmack as defendants.)

In its complaint, ZeniMax alleged that, in 2012, Luckey approached John Carmack (then, an iD software employee) about a Rift prototype, which Carmack offered to help improve.  Subsequently, Carmack and other ZeniMax employees allegedly “transformed” the Rift prototype by adding positional tracking, reducing latency and making other significant improvements.  (According to Facebook, the improvements to the Rift amounted to “hot-gluing” a motion sensor previously purchased from Hillcrest, “placing tape on the outside edges of the lenses, attaching a strap, and plugging the headset into an external power source.”)

Before revealing the improved Rift prototype — and at ZeniMax’s request — Luckey executed an NDA, which stated that ZeniMax would retain exclusive ownership of the IP disclosed under the agreement.  (Facebook has since denied that the NDA is valid or enforceable.)  Carmack later demonstrated the enhanced Rift prototype at E3 in 2012, which resulted in significant attention and acclaim regarding the HMD.

carmack

After a successful E3, Luckey and Oculus allegedly continued to seek out Carmack’s expertise, but ignored ZeniMax’s overtures regarding compensation for its IP. Moreover, Oculus and Luckey allegedly demonstrated the Rift prototype using ZeniMax’s properties (Doom 3: BFG Edition and RAGE) against ZeniMax’s wishes.  In addition, Oculus later solicited and hired away several ZeniMax employees, including Carmack, who became CTO of Oculus.  According to ZeniMax, Oculus has never paid a dime to Zenimax.

In March 2014, Facebook announced that it would acquire Oculus for $2 billion in cash and stock.  Two months later, Zenimax filed its lawsuit.

Perhaps on a related note, earlier this year at E3, Bethesda, ZeniMax’s subsidiary, announced that it was developing virtual reality versions of its blockbuster AAA titles, Fallout 4 and Doom, which, unsurprisingly, would be released on the HTC Vive.

The jury trial in the ZeniMax v. Oculus case is scheduled to begin on Monday, January 9, 2017.

BMW v. TurboSquid: Virtual Patent Infringement?

The blog, Protecting Designs, reported an interesting case filed last month that may have legal ramifications to the burgeoning VR industry:  BMW v. TurboSquid, Inc.

The issue:  Can selling a digital 3D model of a car, which exists only in virtual reality, infringe a design patent?

According to Wikipedia,

TurboSquidTurboSquid is a digital media company that sells stock 3D models used in 3D graphics to a variety of industries, including computer games, architecture, and interactive training. The company, headquartered in New Orleans, Louisiana in the United States, is most known for brokering the sale of 3D models in return for a percentage of the sales. As of 2014, TurboSquid had over 370,000 3D models in its library, making it the largest library of 3D models for sale in the world.

TurboSquid’s website lists Sony Pictures, Activision and Pixar as a few of their clients.

On May 3, 2016, BMW filed a complaint in a federal district court in New Jersey alleging, among other claims, that TurboSquid’s 3D models of certain vehicles infringe BMW’s design patents.  In particular, BMW alleges that TurboSquid is “marketing 3-D virtual models of vehicles that infringe the BMW trademarks, trade dress, and design patents.  [TurboSquid] markets and tags BMW-trademarked 3-D virtual models of BMW vehicles as suitable for games.”

mini
Image from U.S. Patent No. D664,896

BMW’s design patents (D473,165, D639,209, D664,896, D714,190, D714,687, and D724,495) are directed to BMW, Mini, and Rolls-Royce branded vehicles produced by BMW Group. The ’165 patent is titled “Surface configuration of a vehicle, toy and miscellaneous consumer products incorporating the design,” and the other patents are each titled “Vehicle, toy, and/or replicas thereof.” BMW have also asserted trademark infringement claims against TurboSquid and is seeking injunctive relief, disgorgement of all of TurboSquid’s profits under 35 U.S.C. Section 289, attorneys’ fees and other remedies.

 

It remains to be seen whether “virtual” objects can infringe a design patent (or perhaps in a future case, utility patents).

For more analysis on the trademark claims, Patently-O had an interesting write-up which can be accessed here.

Sony’s HMD Patent Application

Earlier this year, at the Game Developers Conference in San Francisco, Sony announced that its virtual reality headset for the PS4 would launch in October 2016.  At GDC, Sony also demonstrated the social aspect of its VR systems, in which multiple players interacted with each other in a virtual reality environment.

Sony 163 Appl FIG 11

Last week, on May 12, the USPTO published Sony’s patent application, Sony Computer Entertainment’s U.S. Patent Appl. No. 14/996,163 (the ‘163 application).  The ‘163 application is simply entitled “Head Mounted Display,” claims priority to a provision filed in June 9, 2013.  The “inventive concept” of the claims here appears to be rendering a player’s hands into the virtual reality scene, as shown below in claim 17 (emphasis added).

17.    A method for operating a head mounted display (HMD), comprising,

providing the HMD having a head attachment portion and a viewing module coupled to the head attachment portion, the viewing module including an inner side having a view port into a screen configuring for rendering a virtual reality scene, and an exterior housing;

providing a communications module for exchanging data with a computer system, the computer system is configured to generate the virtual reality scene for the screen;

providing a depth camera integrated into the viewing module and oriented to capture depth data of an envrionment in front of the exterior housing; and

processing, by the computer system, the depth data captured by the depth camera to identify hands of a user wearing the HMD in the environment, wherein the hands are rendered into the virtual reality scene, the hands being tracked such that movements of the hands appear as movements of virtual hands extending into the virtual reality scene.

As evident from the claims, the rendering of the virtual hands relies upon data captured by a depth camera located on the HMD.  This is interesting because the depth camera appears to be technology that Sony developed as early as 2011 in response to Microsoft’s Kinect technology.  More information about Sony’s depth camera can be found here and here.

 

Valve’s Chaperone Patent Application

A key distinguishing “feature” of the Vive, a virtual reality system jointly developed by HTC and Valve, is room scale technology.  Room scale allows a VR system to track a user’s position within a physical space through the use of “lighthouses” placed in the corners of a room.  Room scale purports to create a more immersive experience, whereby a user can physically move about within her virtual environment.

The downside to all of this is that Vive users may run into walls … or worse.

Enter: Chaperone.  Chaperone is a feature of Vive, wherein graphical representations of physical boundaries appear when the user is about to collide with a wall. A recently published patent application reveals Valve’s attempt to patent the aptly-named feature to prevent such accidents (and perhaps avoid lawsuits like this).

Vive Chaperone Patent FIG 9

U.S. Patent Application No. 14/933,955, published on May 5, 2016, is entitled “Sensory Feedback Systems and Methods for Guiding Users in Virtual Reality Environments.”  The application is assigned to Valve, and claims priority to a provisional application filed in November 2014.  The application presently includes a single independent claim which recites, in part, “a method for warning a user of a head-mounted display of potential collisions with real-world obstacles.”  (The USPTO has yet to issue its first office action.)

The specification of the ‘955 application is an interesting read for VR enthusiasts (i.e., geeks) for other reasons.  First, for historical buffs, the application includes a photograph of an early HMD, Vive Chaperone Patent FIG 2which could be described as “less-than-flattering.”

Second, to define a user’s physical “play” space, the Vive setup currently requires a user to physically walk around the edge of the room with a controller in hand.  The specification of the ‘955 application discloses defining the boundaries of the user’s physical space through the use of “lasers or ultrasound.”  Could this be a feature of Vive 2.0?

Third, the specification also describes the “teleportation” game mechanic that is presently used in two room-scale games: the Lab and the Budget Cuts demo.  This game mechanic addresses the problem of locomotion within a “virtual space” (e.g., spacecraft hangar) that is larger than the real-world space in which the Vive system is setup (e.g., your home office).  It will be interesting to see if Valve attempts to file a continuation on this game mechanic, especially in light of Alice and the Federal Circuit’s recent decision in In re Smith.

Finally, it will also be interesting to see how the claims of the ‘955 application will change, if at all, throughout prosecution, especially in light of expectations that Oculus will be implementing its own room scale technology in the near future.