Apple Receives Patent for Head-Mounted Display for Use with Smartphones

apple

Apple has been in the news a lot lately  — from the $14.5 billion EU tax fine to the anticipated announcement of the iPhone 7 just a few days away.

However, Apple has been conspicuously silent about its plans with regard to virtual and augmented reality. Although Apple has not made any formal announcements (apart from recent statements by Tim Cook, Apple’s CEO, regarding VR and AR), it’s patent filings may provide a hint about the direction they are going.

On Tuesday, the USPTO issued a new patent to Apple (USP No. 9,429,579), entitled “Head-Mounted Display Apparatus for Retaining a Portable Electronic Device with Display.” This marks the seventh U.S. patent that Apple has received relating to HMDs. (Samsung has fourteen U.S. patents relating to HMDs, with five of them being design patents.) Although recently issued, the ‘579 patent is a member in a family of patents, whose parent was filed in September 2008.  In other words, Apple has been toying with the idea of an HMD for at least the last eight years.

Claim 1 of the ‘579  patent recites:

A head-mounted device that is worn on a user’s head and configured to integrate with a cellular telephone that is removable, the head-mounted device comprising:

a frame that is configured to physically receive and carry the cellular telephone, wherein the frame places a display screen of the cellular telephone in front of the user’s eyes; and

an optical subassembly configured to receive at least one image frame from the display screen of the cellular telephone, wherein the optical subassembly is interposed between the display screen and the user’s eyes.

As described in the specification, the optical subassembly (shown below at 604) may include “lenses, light guides, light sources, mirrors, diffusers, and the like.” Specifically, optical sub assemblies may include “aspherical and diffractive optical arrangements.”  In addition, the optical subassembly may also include “one or more optical modules that may be operative to adjust or modify the displayed media based on any suitable criteria.” For example, the “optical modules” may offset left and right images so that the user is given the illusion of viewing media in three dimensions.

As we’ve said before, a company’s patent filings are certainly no guarantee that a commercial product will ever be released.  However, based on recent rumors surrounding Apple’s acquisitions, hirings and iPhone 7 technology, it is a good bet that Apple has something in the works.

Magic Leap Violates Laws Prohibiting Restraints on Employee Mobility Says Defendants

magic leapAs reported back in June, augmented reality startup, Magic Leap, filed a trade secret misappropriation lawsuit in the Northern District of California against two former high-level employees, Dr. Gary Bradski and Dr. Adrian Kaehler.  Magic Leap’s complaint alleges, among other things, that Bradski and Kaehler worked together for at least a year, while employed by Magic Leap, to create and develop plans for a new company using “Magic Leap’s intellectual property and Proprietary Technologies.”

Earlier this month, Bradski and Kaehler struck back against Magic Leap with a flurry of responsive filings.

First, in their Motion to Dismiss, the defendants allege that Magic Leap’s lawsuit “violates Dr. Bradski and Dr. Kaehler’s freedom to pursue independent consulting; in addition, California law prohibits restraints on employee mobility and ‘protects Californians and ensures that every citizen shall retain the right to pursue any lawful employment an enterprise of their choice.'”  The motion alleges that both Bradski and Kaehler were considered experts in computer vision and machine learning, and discussed starting a robotics company long before joining Magic Leap.  Moreover, according to the motion, both Bradski’s and Kaehler’s employment agreements with Magic Leap each provided that “he will be free to pursue independent consulting work.”  After their hiring, Magic Leap allegedly failed to allocate budget to Kaehler’s project, and “for Dr. Bradski’s part, he had been increasingly ostracized.”  In March and April 2016, Kaehler disclosed his plans to leave Magic Leap to work on a “noncompetitive project,” and both Kaehler and Bradski subsequently returned their computers and research to Magic Leap.

In terms of legal argument, the motion asserts that “Dr. Bradski’s and Dr. Kaehler’s skills and expertise, pre-dating employment, and their prior work combining machine vision, deep nets, and robotics cannot be any Magic Leap trade secret.”  In an interesting legal argument, Defendants also assert that the Defense of Trade Secrets Act claims should be barred because “all acts complained of occurred, were disclosed, and ceased prior to the [DTSA] going into effect on May 22, 2016.”

In addition to its Motion to Dismiss, Defendants each filed a Motion for Sanctions of Dismissal (“Rule 11 Motions”) against Magic Leap.  The Rule 11 Motions forcefully assert that Magic Leap filed a “baseless suit without reasonable inquiry; and for the improper purpose of harassing the defendants, staging a fishing expedition and smear campaign designed to paralyze and prevent defendants from any gainful employment.”  In addition to dismissal, defendants seek an award of attorneys’ fees which they claim are “justified and necessary for effective deterrence of future violations by Magic Leap and similarly-minded employers.”

Magic Leap has until September 12 to respond to Kaehler’s and Bradski’s Motions to Dismiss and Rule 11 Motions.  A hearing on the motions is scheduled for Wednesday, October 12, 2016.

BMW Dismisses Design Patent Lawsuit Against TurboSquid

TurboSquidTwo months ago, we reported on a lawsuit filed in New Jersey by BMW against digital media company, TurboSquid, Inc.  According to the complaint, TurboSquid was “marketing 3-D virtual models of vehicles that infringe the BMW trademarks, trade dress, and design patents.”  The case presented interesting legal questions with respect to IP and virtual reality.  For example, can selling a digital 3D model of a car, which exists only in virtual reality, infringe a design patent?

On August 10, BMW filed a Notice of Voluntary Dismissal of the case — without prejudice — leaving open the possibility of BMW “rebooting” its lawsuit against TurboSquid in the future.  Thus, at least for now, it appears that the interesting legal questions are left to be resolved in a different or future dispute.  Interestingly, we note that TurboSquid no longer sells 3-D models of BMW or Mini Cooper vehicles on their website — although they do have models of other vehicle manufacturers, such as Tesla, Audi and Toyota.

Another Update on ZeniMax v. Oculus

As reported here earlier, the two year-old legal battle over the Oculus Rift between ZeniMax  Media and Oculus VR continues to rage on in a federal court in Texas, with trial set for January 9, 2017.  Less than two weeks ago, the court granted ZeniMax’s motion to unseal its Second Amended Complaint, which raises new allegations against Oculus and Palmer Luckey, and now adds Brendan Iribe and John Carmack as defendants.

ZeniMax’s pleading remains substantially the same as its previous complaint, but adds some interesting new allegations.  For example, the new complaintZvO - SAC images alleges that Luckey “lacked the expertise, knowledge, training or resources” to develop VR technology and did not have the “expertise or knowledge to create a viable SDK for the Rift.”  ZeniMax also alleges that Carmack admitted that “without ZeniMax, Oculus ‘wouldn’t exist as a funded company.'”  (an admission most likely obtained during deposition.)

The most relevant new allegations, however, may have to do with the Non-Disclosure Agreement signed between Luckey and ZeniMax in May 2012.  As reported earlier, the NDA allegedly prohibited Luckey from using or disclosing ZeniMax’s Proprietary Information (including its trade secrets), as shown below:

ZvO - SAC images-2

According to the new complaint, Luckey “informed Iribe and others about the NDA he had signed which … confirmed that ZeniMax alone owned the critical VR technology being used by Oculus.”  Despite this, “Iribe privately directed Luckey and other Oculus employees to continue to obtain ZeniMax’s hardware and software technology from Carmack, and Iribe directed Oculus’s employees to use that technology to create the Oculus SDK and to develop, modify, and tune the Oculus Rift hardware.”  Further, the complaint alleges that throughout several rounds of financing and continuing through Facebook’s acquisition of Oculus, Luckey and Iribe “concealed from potential investors and business partners the existence of the NDA that Luckey had executed.”

ZvO - SAC images-3ZeniMax’s complaint also levels some new allegations at Carmack, who was previously employed by iD Software (a ZeniMax sub) before becoming the Chief Technology Officer at Oculus in August 2013.  Allegedly, “before leaving ZeniMax, Carmack secretly and illegally copied thousands of documents containing ZeniMax’s intellectual property from his computer at ZeniMax to a USB storage device which he wrongfully took with him to Oculus.  After he had joined Oculus, Carmack returned to ZeniMax’s premises and took without permission a customized tool that Carmack and other ZeniMax personnel had developed for work on virtual reality.”  According to ZeniMax, “Carmack has refused to return the tool to ZeniMax, and he has used this tool for developing virtual reality technology for Oculus.”

Oculus recently issued a response stating that the “complaint filed by ZeniMax is one-sided and conveys only ZeniMax’s interpretation of the story. We continue to believe this case has no merit, and we will address all of ZeniMax’s allegations in court.”

With the trial less than six months away, the parties continue to engage in a pitched legal battle, with recent filings including motions to exclude expert reports, motions for sanctions and motions to compel the production of documents (all of which are sealed).  According to the court’s scheduling order, motions for summary judgment are due in one week on September 2, 2016, and pre-trial filings due on December 2, 2016.

Finally, the court has ordered the parties to hold a face-to-face meeting to discuss settlement in mid-December.  Will the parties eventually settle?  If the recent failed mediation is any indication, the answer may be ‘no.’  However, the cost of trial (and subsequent appeals) could play a large factor.  Perhaps the only sure take-away from this story is that there will be some unhappy associates working over Thanksgiving this year.

Is Sony Going to Launch Its Own ‘Powerglove’ for the PSVR?

PSVRWith the launch of Sony’s Playstation VR less than two months away, Sony continues to be busy on the patent front with respect to virtual reality.  Earlier today, two patent applications assigned to Sony Computer Entertainment, each entitled “Magnetic Tracking of Glove Fingertips,” were published by the USPTO.  The full patent application can be seen here.

While Sony has filed patents for a similar technology in the past, these new patent publications (originally filed in February 2015) relate to a “magnetic tracking system to track fingertips and knuckles … that capture hand/finger pose.”

As seen in FIG. 4A (right), Sony’s application describes a glove Sony Magnetic Gloveinterface object having a plurality of emitters and proximity sensors.  The proximity sensors (404a-e) are located at the fingertip portions/areas of the glove, and are configured to “generate data indicating distance/proximity” to the emitters (422a-c) located on the wrist.  The application goes on to describe the wrist portion as a “bracelet that surrounds the user’s wrist when the glove … is worn.”

“The emitters are defined by electromagnets, and the proximity sensors are defined by magnetic sensors such as Hall effect sensors.”  A Hall effect sensor is a type of magnetic sensor used to detect a magnetic field. (In an alternative embodiment, Sony’s application also describes a glove system in which the emitters are defined by ultrasonic emitters and the proximity sensors are defined by microphones capable of detecting ultrasonic frequencies.)

As further described in the application, the glove will include a controller for powering and operating the sensors and emitters and communicating with the gaming console.  The controller can be configured to control the activation of the emitters and the reading of the proximity sensors in a time division multiplexed arrangement.

Obviously, Sony’s published patent applications do not necessarily mean that a commercial product will ever see the light of day.  However, it can be a good indicator of where Sony is spending its R&D dollars when it comes to virtual reality.  And with the timing of these multiple patent application filings, as well as the length and specificity of each application, it is not completely far-fetched that Sony has its own version of a ‘Powerglove’ in the works for the PSVR.

NextVR: Building a Patent Portfolio

bg-dark-nextvr-all

In the past year, virtual reality broadcasting startup, NextVR, has been a frequent topic in the media for its impressive media deals and ability to raise funds.  Earlier this month, NextVR closed an $80
million
Series B round of funding, with an eye towards expanding into Asian markets where demand for virtual reality content is growing.  Continuing on with its global push, it was also recently announced that, in conjunction with FOX Sports, NextVR would present the opening football match of the 2016/2017 Bundesliga season in virtual reality.

Less reported, however, is the fact that NextVR is quietly building a patent portfolio for encoding, streaming and displaying 360 video.  Last week, six new patent applications were published having NextVR as the assignee.  According to our search of FPO, NextVR has 32 worldwide patents and patent applications.  (By comparison, Oculus has 42.)

NextVR patent fig 14The majority of NextVR’s patent applications, published on August 18, are directed to methods and apparatus for using “selective resolution reduction on images to be transmitted and/or used by a playback device.”  These applications appear to cover methods and devices for downsampling video content using texture maps and mesh models before presenting the video content on a HMD.  As the application explains, “reducing the resolution of images which are less likely to be viewed while maintaining the resolution of portions of images corresponding to an environment which are likely to be viewed, it is possible to make efficient use of limited bandwidth available for streaming image data to a playback device.”

While downsampling video is certainly not a new technology, the novelty of NextVR’s invention appears to relate to the use of texture (or “UV maps”), which are stored on the server and sent to the (HMD-connected) client ahead of the image data.  The application describes “different UV maps in combination with selective resolution reduction can be used to allocate different amounts of resolution to different portions of an image of an environment depending on which portion of the environment is considered important at a given point in time while the same environmental model is used despite the different allocations of resolution.”  Another interesting aspect of the technology, as described in the application, is that “the data rate used for transmitting images can be held relatively constant since the number of pixels in the images can remain the same with the UV map controlling the allocation of pixels to portions of the environment.”

NextVR’s recently published patent applications can be viewed here.

DirecTV Files Patent Application for Viewing “Set Top Box Content” on a Virtual Reality Device

The_DirecTV_logoLast week, the USPTO published DirecTV’s patent application for delivering “set top box content” to a virtual reality display device.  This appears to be one of the first published patent applications by a major MVPD (“multichannel video programming distributor”), and could signal that the industry is ready to push their VR chips onto the table.

DirecTV’s patent application describes transmitting a live linear television signal to a user receiving device (e.g., a set top box), which then renders the signal and transmits it to a client device (e.g., smart phone).  The client device includes a virtual reality application, which then scales the signal for display on a virtual reality device (e.g., Oculus Rift).

DirecTV patent app fig 1

Thus far, the big players in traditional (cable, satellite) video content distribution, or MVPDs (“multichannel video programming distributors”), have been relatively quiet with respect to virtual reality technology.  As seen below, most VR-related announcements by MVPDs have pertained to investments or acquisitions in studios, like NextVR.  Moreover, a recent survey of 628 entertainment and media companies showed that at least 30% of the industry considered virtual reality “a non-starter, like 3DTV.”

Non-traditional content distributors have been somewhat more forward thinking.  For example, in March, Hulu announced it would begin streaming content to Samsung Gear devices. Netflix also released a 360 promotional video last week in conjunction with its popular series, “Stranger Things.”

Top Ten MVPDs VR