Artificial Intelligence (AI) for Marine Rescue

(Photo: HTX)

HTX’s Sense-making & Surveillance Centre of Expertise (S&S CoE), in collaboration with the Singapore Civil Defence Force (SCDF), has developed the Marine Video Analytics for Rescue and Recovery at Sea. This is a breakthrough technology in the market for Video Analytics (VA) solutions for maritime search and rescue.

Existing solutions predominantly focus on airborne platforms, such as Unmanned Aerial Vehicles (UAVs) and drones. There is a scarcity of VA systems which incorporate specialised deep learning capabilities tailored to meet SCDF's need for fast and accurate human detection—both on the ocean surface and the underlying seabed.

Our solution comprises a custom-built video analytics, Seaborne Electro-Optic (SEO) sensors, underwater sonar technology, and an array of onboard sensory instruments to deliver accuracy in man-overboard detection. This cutting-edge technology could serve as a force multiplier for the SCDF, significantly enhancing SCDF’s operational effectiveness and efficiency to maritime emergency response.


Problem Statements

Three specific problem statements were identified:

  1. Mitigating challenges that hinder effective sea rescues
    1. Harsh sea conditions1
    2. Need for fast response and accurate detection of humans
    3. Objects can drift very far within a short span of time2
    4. Minimal window to retrieve body during rescue missions for capsize incidents
  2. Scarce manpower limits sea rescues
    1. Crew takes on a multifaceted role between manning the ship, navigational safety, and search-rescue-recovery operations
    2. Use of AI to augment search and rescue work
  3. Human fatigue arising from extensive hours of rescue operations
    1. Search-rescue-recovery operations entail long hours with elevating risk of fatigue that could compromise rescue success


Enhancing Visual Clarity Amidst Maritime Challenges

Captured imagery and video footage at sea often suffer from suboptimal quality, influenced by fluctuating light conditions, variable sea states, and the vessel's motion-induced blur. To mitigate these complexities, we've implemented an array of advanced image enhancement techniques:

1. Distant & Small Object Detection

  1. A computer vision task involving detection and localisation of small objects within images or videos
  2. Used to overcome challenges involving a combination of Dim Target Detection (low signal to clutter ratio) and Small Target Detection (typically <10 x 10 pixels)
  3. Uses Neural Networks for feature extraction

2. Use of Processing Pipeline for Maritime Environment

  1. Used to create more robust and accurate VA while tackling challenges in variable lighting and weather conditions
  2. Depending on the weather condition, overcast and camera’s capability, the colours out at sea might not be perceived accurately by the equipment. Hence, there is a need to employ colour and contrast enhancement to improve image qualities to detect specific colours (e.g. red and yellow) which are generally unnatural and distinct colours out at sea)
  3. Involves the use of automated colour and contrast enhancements to video footages, followed by a fusion of multi-models including Deep Learning Detection and Colour Detection
Pre-Processed Image Processed Image 
 AI 1a AI 1b

(Photo: HTX)


Overcoming Data Scarcity via Synthetic Data Generation

In the development stage, the team encountered significant obstacles in identifying readily available datasets for Medium Wavelength Infrared (MWIR), Long Wavelength Infrared (LWIR), and maritime sonar in the maritime context. Recognising that data paucity would constrain our AI model's training efficacy and hinder overall project advancement, we leveraged a set of different innovative techniques to circumvent these limitations:

1. Data Scarcity for the training of Man-In-Water Detection Models

To address this challenge, we employed an integrated approach combining Deep Learning Models with specialised computer vision pre-processing techniques. This enabled us to generate synthetic thermal infrared imagery for effective model training (refer to the image below). Subsequently, AI models were rigorously trained on these synthetic datasets and fine-tuned to improve the accuracy in the detection both humans and vessels.

AI 2
(Photo: HTX)

2. Data Scarcity for the training of Underwater Sonar Detection Models

Collecting datasets from sonar is resource-intensive, owing to the inherent difficulties of underwater mannequin placement and retrieval. Moreover, automated solutions for sonar image preparation and cleansing are scarce—largely due to the limited volume of readily available sonar data and a paucity of ongoing research in this specialised area. To circumvent these obstacles, our development team employed a blend of superimposition and data augmentation strategies, thereby generating enriched datasets for robust model training.

AI 3
(Photo: HTX)


Coming Next

Some of the enhancements to look out for in our development pipeline:

1. Analytics on the Edge

The team is working on the Research & Development works to enable models to run on the edge to miniaturise the overall set up and tackle space scarcity issues onboard. We are also looking to provide the options for users to view the display on a tablet for greater usability and portability.

2. Over the Air (OTA) Model Updates

Model updates shall be sent via 4G network to the end device, eliminating the need for engineers to plug in and upload the models physically onboard during each model refresh/update.

3. Data Integration with other system onboard such as GIS

Mapping functions through triangulation mapping will be made available for the Man-in-Water model. This helps to retrieve the exact GPS location of the object of interest for visualisation and action.


Interested to find out more or discuss further?
Do connect with us:



  1. How Long Can You Survive in the Sea?
  2. Surprising drifting of bodies along the coast of Portugal and Spain


Related Link