Arena for hunters and hunted: Finding principles of intelligence with SCIoI’s loop method

By: Press Team

Researchers study the struggle for survival in the sea and simulate it in an interactive augmented reality application to uncover principles of collective intelligence.

Science of Intelligence (SCIoI) shows a large fish jumping out of the river Spree and hunting a school of robotic fish as the motif for the Berlin University Alliance’s (BUA) campaign “The Open Knowledge Lab – For the Great Transformations of Our Time.” What does this entail and how does it connect to general principles of intelligence? The three SCIoI members Alicia Burns, Palina Bartashevich and David Mezey explain their interdisciplinary research:

Alicia: At first, this connection seems to be a big leap, pun intended (laughs). But at SCIoI one of the types of intelligence we’re interested in is collective intelligence: the idea that some groups are better problem solvers than any one individual. Animals are very good at this! So we start by studying intelligent behavior in biological systems – in our case, fish. We go “into the wild” to study them and try to understand what makes them tick.

Palina: Yes, and to gain a deeper understanding of their behavior, we want to “build” it by developing computer models of the observed phenomena and further testing them in the real world on robotic platforms.

David: This gives us a great advantage, as we can test things that we would not be able to test just by observing the biological system. We can change, for example, how many fish or predators are in the group, how fast they swim, how they interact, and we can study how such changes affect the overall collective behavior we observe.

Palina: What is exciting in such an approach across disciplines – biology, computer science, and robotics – is that it allows us to gain a new perspective from different sides. Computer scientists and roboticists use their models to check biologists’ ideas. This often leads to new hypotheses that biologists can verify in nature, creating a constant exchange of ideas, like in a loop.

How does that play out hands-on in your research?

Alicia: In the motif, you see a striped marlin, one of the fastest and largest fish in the ocean. Marlin have to work together to herd the sardines, but they work against each other when competing for food, and if two of them get the idea of attacking at the same time, they could seriously injure each other. So they have to find a way to collectively solve this problem of cooperating and competing at the same time.

What intelligent strategies do the hunters pursue, then?

Alicia: From drones, we have observed a fascinating phenomenon! Marlin light up when they attack by exposing certain skin cells that reflect bright blue light – resulting in the stripes that give striped marlin its name. These stripes not only potentially confuse the sardines, but may also signal their intention to the other marlins. Without knowing it, they’re basically saying “Now it’s my turn, get out of the way” – an elegant (and intelligent) solution to a collective hunting problem.

Marlin hunting sardines under water ©SCIoI/Alicia Burns; Alicia Burns ©BUA

What about the sardines? Do they have similar strategies?

Palina: Yes! The sardines evade an attacking marlin by splitting into two groups in front of the billfish and swimming in an arc behind the attacker. This is called the fountain effect. In order for a shoal to perform it successfully, the individual fish only need to follow a few simple rules. The important thing is to avoid the attacker by swimming at an angle of around 30 degrees.

In other words, this clever fountain effect works even though the fish don’t actually know what they are doing?

Palina: Exactly. This is collective intelligence. It’s not about how the individual behaves per se, but how the group acts intelligently as one entity without having the big picture. The observed escape angle also correlates with the known angle range at which fish, like sardines, can see backwards. In other words, the fish want to get as far away from their pursuer as possible but keep at the same time an eye on them, while also staying close to their conspecifics. Computer simulations allowed us to identify the best hunting strategies for predators and the most effective escape tactics for prey in this scenario, which shed light on why certain attack directions are more frequent than others and why we observe the fountain effect overall. I was also lucky to witness this powerful arm race myself on the fieldwork with Alicia. The dynamics are breathtaking!

Palina Bartashevich ©BUA; Computer Model of Marlin hunting sardines ©SCIoI/Alicia Burns and Palina Bartashevich

The computer science expert was on the boat as well?

Palina: And in the water! That was me! I have to admit, it is quite an out-of-the-ordinary working routine for a computer scientist, so I did a snorkeling course to prepare myself. There were so many details that only became clear to me during the boat trips off the Pacific coast of Mexico. For example, how do marine biologists know where to look in the vast ocean for the battle between sardines and marlins?

Alicia: The birds circling above, hoping to get some prey…

Palina: … and sometimes through the bubbles on the water surface, which indicate that a life-and-death struggle is raging just below the surface.

Birds circling prey and on the boat during field research ©SCIoI, IGB/ Alicia Burns, Matt Hansen, Max Licht

And how does the understanding of such intelligent behavior and the principles help us in general?

Alicia: These behaviors are shaped by evolution – so a classic evolutionary arms race between predator and prey, where both the sardines and the marlin are constantly trying to one-up each other. By breaking these phenomena down to their component parts we can start to unravel the building blocks of intelligent behavior that can then inform other systems.

Palina: And by understanding group dynamics in its fundamentals, we can contribute to future construction of robots and artificial systems that can fulfill tasks together in areas that are dangerous for humans. Or, to give more informed answers on how to deal with, for example, panic at mass events by simple means. At Science of Intelligence, we investigate all of these phenomena holistically.

So, after you watched and analyzed different fish collectives in nature, you then came back to the computer to translate the observations into computer models and analyze them. What came next?

Palina: Next, we aimed to bring our models to life. We reached out to our colleague David, who specializes in robot swarms, and together developed a plan to enhance our models and uncover new insights through robotics.

CoBe – Collective Behavior (augmented reality interactive projection for research and science transfer) ©SCIoI; David Mezey ©BUA

David: A crucial factor in testing and finding new phenomena from biological data is to also bring it to the real world and into physical systems with their own limitations similar to animals. Fish, for instance, can only move or turn as fast as the water and their bodies allow them in contrast to simulated fish. Validating findings not only in simulations but also in real-world scenarios using robots gives us a better understanding about the observed behavior. It allows us to introduce the complexities and unpredictabilities of the real world that simulations cannot fully capture. These include variations in lighting, surface textures and friction, obstacles or a limited perception and it is also commonly known as the simulation-to-reality gap. As a result, real-world applications, in contrast to simulations, can uncover unforeseen challenges that an embodied system must always face, be it animals or robots. This way, studying behavior on robots might also facilitate understanding behavior in animals.

Palina: To bring into reality two robot swarms interacting with each other is a real engineering challenge. One has to get right not only the interactions inside each collective but also between the collectives. Here, the simulation-to-reality gap can become a bottleneck or rather an inspiration, as it was in our case.

David: To tackle this challenge, we came up with an exciting system using augmented reality. We built a big arena where the simulated fish are projected on the ground, and based on Palina’s algorithms they react similarly to what is observed in nature. This is the “augmented” part. The “reality” part is where physical agents, for example, humans or robots can interact with the projected fish swarm as predators and try to catch the fish, which reacts to them in real time and in some cases results in the fountain effect. Successful patterns that are shared across these systems might uncover general principles of collective intelligence.

Palina: Also, it is a fantastic ground for proof-testing the models. It was hard to replicate a predator’s behavior separating a single fish from the group in the simulation until the robot successfully did it while using the same strategy!

What have you already learnt?

David: For example, how much easier it is to catch the projected fish, if you’re hunting them in a group of two. No wonder marlin hunt in groups! I will soon be supervising a course at the Technical University of Berlin that allows students to learn the basics of modeling collective behavior and they can also implement their models in our interactive arena, among other things. In the meantime, our system serves not only our experiments, but also as an important science communication tool. Our interactive arena allows our visitors to interact closely with our simulations and dive deep into our research. Quite literally!

Alicia: And that can be very instructive! I thought that my knowledge of how marlins hunt would make me an excellent sardine hunter. But quite the opposite is true: I was worse than many people who were dealing with the topic for the first time (laughs).