We continue to salute Brunswick for their efforts to detect people in the water near a recreational boat.

Basics of the Patent

U.S. Patent 10,372,976 Person Detection in a Marine Environment invented by Troy Kollmann and Jon Nowick, assigned to Brunswick issued on 6 August 2019.

The patent’s first claim describes detecting people in the water near vessels (1) using at least one image sensor, (2) that “at least one image sensor” detects visible light, (3) that “at least one image sensor” is at at angle and distance to the water at the rear of the vessel, (4) the images are scanned to detect the presence or absence of a person, (5) the scanning module includes an Artificial Neural Network (ANN), (6) that ANN being trained to detect patterns within the image associated with a person in the water behind the vessel, (7) the system outputs detective information regarding the presence or absence of a person in the image, (7) wherein the ANN module is trained using positive and negative images of objects in the water from a similar viewpoint and distance at the stern.

U.S. Patent 10,372,976 Person Detection in a Marine Environment

The patent includes a second independent claim focused on using a similar neural network approach to detecting the presence of the operator at the helm. This feature would basically be a virtual kill switch to prevent Circle of Death propeller injuries and fatalities.

A third independent claim focuses on training the neural network (ANN) based on the predefined object (like a person), determining the range and fields of view associated with where the camera is mounted with respect to the water, and training ANN using images of the predefined object (such as a person) on water from those view angles and fields of view.

Our Prior Art

I began collecting information on potential methods of detecting people in the water near the propeller, including CCD cameras, at MerCruiser / Mercury Marine, a Brunswick Company, over 25 years ago.

Back in January 2018 we (PropellerSafety.com) covered the publishing of Brunswick U.S. Patent Application 2017/0323154 A1 and some of our prior art in this field.

At that time we suggested the claims in Brunswick’s patent application would likely have problems with our (Gary Polson / Polson Enterprises / PropellerSafety.com) prior art.

Neither Brunswick nor USPTO cited our work. USPTO likes to stick with citing patents, but can sometimes be drawn out by obvious prior art outside the patent databases.

Since then, Brunswick made many changes in the patent claims in response to the objections raised by the U.S. Patent and Trademark Office (USPTO).

We suggest their first claim may still have some problems with our prior art.

Brunswick Had Problems Getting This Patent Issued

Unusually, back on 11 April 2017, Brunswick rescinded their previous Non Publication Request. Meaning, originally Brunswick paid a hefty fee to avoid publication of the patent application prior to the patent being issued. In April 2017, Brunswick changed its mind.

One patent that caused Brunswick a lot of grief was Volvo Penta’s patent by Lennart Arvidsson, U.S Patent 8,195,381 “Safety System for Marine Vessels”.

Just a glance at the Volvo Penta patent drawing below tells the tale.

Volvo Penta U.S. Patent 8,195,381 sketch

Brunswick also had considerable challenges with a broader Neural Network image processing patent application, U.S. Patent Application 2014/0355871 “System and Method for Structuring a Large Scale Object Recognition Engine to Maximize Recognition Accuracy and Emulate Human Visual Cortex” invented by Mohamed of VuFind.

Brunswick’s Woes Continued

The patent application was filed 5 May 2016.

Initially (19 January 2018) USPTO objected to all of Brunswick’s claims because the claims “do not amount to significantly more than an abstract idea.” USPTO says the steps could be applied mentally and are similar “to collecting information, analyzing it, and displaying certain results of the collection and analysis.” The claims were “rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.” In addition, the examiner specifically pointed out how claims 1 through 12 also had problems with prior art.

A few months later (10 April 2018) Brunswick submitted a new round of claims. Some were new versions of previous claims crafted to avoid objections raised by USPTO, others were new.

About six months later (16 November 2018) after a special request (and fee) for Continue Examination, USPTO sent another Non Final Rejection letter. Brunswick had failed to win USPTO over on most of the independent claims so the application collapsed like a house of cards.

About 3 months later Brunswick tried again with another round of claims. These claims were more detailed and much narrower in scope than they started with.

This time Brunswick was successful.

Changes in Brunswick’s Patent Claims Along the Way

Several elements were added to Brunswick’s first patent claim on 10 April 2018. These additions significantly narrowed the protection provided by the patent (the claim is much narrower in scope).

Among elements added were:

• The camera being a visual light image sensor (vs infrared, sonar, etc). (vs other types of image sensors)
• The claim only applies to the area behind the vessel. (vs any location around the vessel)
• the scanning module outputs information regarding a presence or absence of a person in the marine environment. (vs detecting floating or submerged debris, pending collisions, docking aids, etc)
• The ANN is trained using images of objects on a water surface having a similar range of view angles and fields of view to the sensor on the boat. (more on this below)

These additional elements severely restrict the breadth of the patent.

Ten claims present in the second round of patent claims were cancelled, some remaining claims were amended, and some new claims were added.

Thoughts on Brunswick’s ANN Training Methods

The patent focuses on training ANN using photos of people on the water at angles and distances similar to those seen from the camera on the boat. They also note the use of Negative photos in training (photos without people in the image).

While all this sounds logical and like a good idea, Artificial Neural Networks and Convolutional Neural Networks sometimes function best with broader training. Meaning just showing them some “person” and some “not person” images or other configurations considerably distant from more exact images on the water similar to those being scanned might work best.

This patent seems to have locked them into training ANN using very specific images.

Encouragement

With Brunswick making great efforts to get this patent issued in the U.S. and filing a similar patent application in multiple countries in Europe and in Japan, Brunswick may at least at one time thought this device might “be the one”. Maybe it is still coming?

We encourage Brunswick and others to continue to develop Virtual Propeller Guard technologies to make boating safer. We look forward to seeing these products in the marketplace.

We also thank Brunswick for spending the dollars and effort necessary to prosecute this patent through the system. It’s issue will likely stimulate further efforts in this field.