CONNECTING TO CONSERVATION
Imagine if you could take anyone on a virtual, time-traveling SCUBA dive to convey climate impacts on a coral reef.
Scientists at the Scripps Institution of Oceanography (SIO) are studying reef change over time in Maui using large-area imaging (LAI), a form of photogrammetry that creates 3D models of reefs using overlapping underwater photos. While these models are currently only used for research, they also have immense untapped potential to inform conservation through interactive data visualization. With the help of my brother Orion, a PhD graduate from Scripps, I’m developing a data visualization platform using Unreal Engine 5 that will allow the general public in Maui to virtually explore their reefs. Utilizing SIO’s time series of 3D models, the platform will illustrate ecological changes & place-based conservation needs & include a user-friendly interface, dynamic animated environments, & video interviews with community members to share local stories about conservation & environmental change.
We’ve built a prototype & are currently seeking funding to build the full visualization platform.
Divers in the field collecting images to be used in Large Area Imaging products
A well established pipeline exists for the creation and analysis of Large-area imaging point clouds. After thousands of component images of the reef are captured in the field by a diver, the photos are used to create a 3d reconstitution in Agisoft, computed using a process known as structure from motion. Researchers use these point clouds to conduct virtual field work, as these 3d reconstitutions act as digital archives of benthic ecosystems. Ecological data extracted from LAI may include species density and abundance, habitat structural complexity metrics, and the size and location of sessile organisms (McCarthy et al. 2023).
In order to extract and analyze this data, researchers at SIO developed a state of the art software program called Viscore, a point-based analytics engine which has the ability to run complex comparative algorithms on point clouds to analyze change over time. Despite high levels of accuracy, Viscore lacks the capacity for real-time interactivity; its purpose is specifically for data analysis rather than visualization. The ability to create or view atmospheric effects, visualization options, and contextual media are features not included in Viscore. Therefore, its potential use as a communication and education tool would necessitate the expansion of the aforementioned pipeline to include a forward facing, real-time, interactive platform that can host large quantities of data, such as a game engine. This video shows an example of visualizations created with Viscore; although the results are compelling and the visual fidelity is high, there is a lack of intuitive legibility as to how reef ecosystems are affected comprehensively by the change over time being highlighted.
The first iteration of this interactive will impact communities and policy in the Maui Nui region by leveraging the graphic fidelity and visual tools offered by a game engine such as Unreal, mirroring the experience of diving on a reef as closely as possible to cultivate pathos and encourage marine stewardship. The platform will host multiple types of media in real-time, including 3D environments, animated marine life, high density point clouds, user interface systems, and contextual media such as videos, graphs, and oral histories. This diverse range of media will illustrate complex research in a format that is approachable for the general public and resonates with the experiences of the local population. Each site represents a unique story of change over time that will be told visually by toggling between 3D models collected in different years.
The long-term vision is for this project to achieve broader impacts beyond Hawaii by including other monitoring sites and islands across the tropics, culminating in a much larger network of data visualizations that would be user-driven and tailored to local context. Hosting these visualizations in Unreal Engine would improve on existing LAI pipelines that are less accessible to non-academic audiences. Reducing barriers to accessing LAI is an important fixture of this work. Our goal is to enable researchers and citizen scientists to easily share their research, 3D visualizations, and conservation stories. In addition, as an interactive educational platform, our eventual audience is museums, science centers, aquaria, classrooms, and conservation institutions worldwide.
PROTOTYPE
In our prototype, we have implemented a simple yet comprehensive interface with unique visualization features. The central hub for the experience is an interactive 3D map where users can gain an overview of reef health across the region. Users have the option to navigate between islands in the archipelago and explore trends across sites through multiple data visualization options. To provide further context, we will highlight impactful stories observed through years of research, embedding media from SIO field work in an interactive field guide. When developed to completion, the field guide will act as an invaluable tool to provide deeper context and perspective on how to view the research in a clear and accessible format. Using the interactive map, users will be able to travel to specific monitoring sites to view 3D point clouds in a highly detailed ocean scene that will closely reflect the actual ecology of each site. Our prototype transforms a singular site, "Maalaea" into a distinct environment level with simulated realistic ocean conditions, complete with animated reef fish, algal blooms, sedimentation events, and other site specific impacts to highlight reef changes over time. Our intentions are to develop comparable site levels for all 40+ sites in the long term monitoring program.
In order to view our prototype, we’ve provided a video walkthrough, as well as link to a pixel-streamed version of the interactive on the web. For those interested, they can try out the interactive here; it is recommended to do so with a powerful computer or laptop connected to power, and a strong internet connection. Due to the nature of pixel-streaming, some latency is to be expected.
FIELD REFERENCES
A primary goal of our interactive (which influenced our choice to develop with Unreal Engine) is to convey realistic ocean conditions and their susceptibility to change. The ability to replicate these conditions with high fidelity enables a virtual “time traveling” that has the power to highlight impacts from coral bleaching, algal blooms, wave damage, poor water quality, overfishing, outbreaks of coral predators, and other disturbance events that would not be obvious if viewed by a snorkeler or diver at a single point in time. Using both of our underwater experience (Orion as an expert and myself as a novice diver) as well as photos and videos taken in the region is essential to replicating accurate conditions and constructing robust particle systems to convey realistic fish behaviors.
Ocean conditions such as algae, sediment, and surge
Both of the above images show examples of how Rugosity or “Habitat Structural Complexity” can create homes for different sized organisms on the reef. Healthy corals and specific species in particular (such as branching corals) can help contribute to structural complexity, while disturbances such as coral bleaching and strong surge can reduce structural complexity. It is important to measure rugosity at different scales to determine it’s main drivers and impacts on the reef; for example, at small scales, rugosity is typically caused by biotic processes, such as coral growth- at large scales, it is usually due to abiotic processes, such as ocean topography.
VISUAL DEVELOPMENT
RESEARCH
My brother Orion is the driving force behind the research featured in this interactive. Orion earned his doctorate from Scripps in February after spending six years documenting the resilience of coral reefs in Hawai‘i. He has produced a sizable body of research that is currently available as scientific papers (see https://www.researchgate.net/profile/Orion-Mccarthy/research) and discussed in depth in his PhD defense which can be viewed below for those who desire more context about Large Area Imaging, reefs in the Maui Nui region, and how reef resilience is quantified and implemented into policy.
Skip to 10:25 for the beginning of the presentation
(Note, the photograph of Orion and his “sister” playing with ocean creatures in the tub is actually our cousin, not me)