Taking Facial Recognition to the Ocean - Automatic Identification of Tiny Arctic Animals
Last year, I conducted a crowd-funding campaign to fund my research on science blogs. As a part of that, I offered to blog about upcoming research papers for researchers who backed my project. I got requests to blog about a number of fascinating research studies - so here's one for you!
Moritz Schmid and several other researchers from Canada developed a model for automatic detection of zooplankton in the Arctic Ocean. Think facial recognition - but for identification of tiny Arctic animals!
I interview study author Moritz Schmid about this new model below.
Me: You use the Lightframe On-sight Keyspecies Investigation (LOKI) system to identify zooplankton in the Arctic ocean. What is this system and what is special about it for detecting life underwater?
Moritz: The LOKI is an underwater camera system that is towed vertically though the water. As it is towed upward, zooplankton (animals drifting with ocean currents) in the water enter the camera of LOKI and pictures are taken. At the same time we also record the environmental conditions in the water, like temperature and salinity, using several sensors. That information can then be linked back to the images.
LOKI is special because it is one of the few systems that has exceptional image quality. This makes good and detailed identification of the little critters possible, which is important when we want to study different species in the ecosystem.
Here we have for example an image of a Mysis shrimp where we can see a lot of details, like internal structure and swimming legs.
Photo via Moritz Schmid.
What is special about underwater imaging systems in general, which are still used rather rarely, is that we know exactly where in the water pictures were taken, with a vertical resolution of less than 1 meter. That is a huge improvement over net systems that have for example 9 opening and closing nets which then have to be spread over let’s say 900 meters of water or even thousands. That is not much vertical resolution! So the system is very good in that respect.
Me: What are zooplankton, and what is the importance of investigating / detecting them in the Arctic ocean?
Moritz: Zooplankton are animals that have very limited swimming abilities and therefore drift mostly with the ocean currents. Some zooplankton are very very small, like miniature versions of the shrimps we eat, but jellyfish of several meters also belong to the zooplankton. With the LOKI system we focus on copepods, which can be seen with the naked eye. They are in the range of 0.2 – 20 millimeters.
A typical copepod as seen by the LOKI system.
Copepods are very important in marine ecosystems because a multitude of other animals feed on them, including fish, whales and seabirds. The reason they are such well-liked prey is that they are full of fat, high energy lipids that can sustain many other animals including top predators.
Me: Your paper describes a model for the automatic detection of zooplankton taxa. What is a model anyway? A computer model? How does the model automatically detect zooplankton taxa?
Moritz: Well, in general a model is just a way of recreating something that was observed, often in a simplified way. In our case we want our computer model to be able to identify zooplankton species just as if a person would be looking at the images.
For that to work one has to train the computer model. That is done by showing many images to the computer. For these images the computer knows which taxa or species the images represent, because we tell the computer that information. What the computer really sees when looking at an image is many different variables that describe the pixels of the image (for example a number describing how round the object is).
What the computer model then learns is the relationship between these many numbers describing the pixels in the image and the species that these numbers represent. That way, when we give the computer new images for which we do not know which species are seen on them, the computer can apply the rules it learned earlier and identify the new images. The process I described here is known as machine learning, a subarea of artificial intelligence.
The easiest way to imagine this is thinking about any Hollywood spy movie where the good guys use facial recognition to identify pictures of the bad guys. Other areas where this kind of technique is used all the time are for example personalized ads, where the model is trained on your past web activity, and online retailers where you get product recommendations.
Me: How could this model contribute to our understanding of current species as well as discovery of new species in the Arctic ocean? How can other researchers use it?
Moritz: The model itself can only identify what it was trained on/what it knows. Therefore the model can’t identify new species. However, it would be possible to find a new species when looking through all the images LOKI took, without using the model. That is however not what the system was designed to do. The system and model is really for better understanding the species we know are there and abundant.
Underwater imaging systems like LOKI produce an immense amount of images (hundred thousands to millions on a research cruise), making up several terabytes. An automatic way to analyze all these images really is necessary. On top of that, the automatic identification model we were able to implement is the taxonomically most detailed model to date. Where other models can only distinguish very broad groups of zooplankton, our model can identify developmental stages of species.
The model in combination with the rest of the system (e.g. the environmental sensors) then allows us to study the connections of the zooplankton with the physical properties of the ocean as well as other living organisms on unprecedented vertical detail.
We showed a few new things in the paper regarding the way we built the model, therefore we hope that other researchers will build on that for their own approaches. The model is specific to the area in which we used LOKI and also to the system itself. There are however methods that could allow the model to be adjusted to images from other systems (keyword: domain adaption).
Me: How can your model of automatic detection of zooplankton contribute toward our understanding of climatic / temperature changes in the arctic ocean?
Moritz: Now that is a great final question.
Researchers have shown that changing conditions in the Arctic such as receding sea ice affect the phytoplankton. Phytoplankton, similar to zooplankton, drift through the oceans with the help of currents. Phyto- comes from ancient greek phutón and means plant. But more accurate than saying plant would be to say microalgae. Phytoplankton are microalgae that are autotrophs, meaning they make energy just from sunlight and nutrients and are the very beginning of the Arctic food web. In the Arctic there is almost no sunlight in winter, and the ice cover is thick, therefore there is no light and phytoplankton can’t grow. But as soon as the ice cover recedes in spring, and the sunlight comes back, phytoplankton can grow again and even bloom (massive growth of phytoplankton).
Phytoplankton - the foundation of the oceanic food chain. Image credit: NOAA MESA Project.
The zooplankton are also highly adapted to the Arctic environment, and especially to feeding on this phytoplankton bloom, but to do so they have to be in the right place at the right time. Life in the Arctic Ocean is all about timing! Unfortunately, studies have shown that Arctic phytoplankton blooms are happening earlier in the year with advancing climate change. If zooplankton or copepods now come too late to feed on the phytoplankton bloom they will face severe consequences.
That is why with the LOKI and the automatic identification model that we developed we aim at better understanding the links between the phyto- and zooplankton, in order to then better understand and predict future changes. That is where the environmental sensors of LOKI come into play again because they tell us about physical properties of the ocean, such as temperature, but also about phytoplankton activity.
Interested in the automatic detection of zooplankton via computer models? Ask Mortiz a question on Twitter!