Research

Object-level scene descriptions and attention in visual search

Exploring visual scene analysis in humans and assessing its potential for use in computer vision systems

Research Unit: 1

Project Number: 1

Example Behavior:
Individual Intelligence

Disciplines:
Computer Vision
Machine Learning
Psychology

 

Principal Investigators:
Martin Rolfs
Klaus Obermayer
Marianne Maertens
Olaf Hellwich

Doctoral Researchers:
Lynn Schmittwilken
Olga Shurygina
Nicolas Roth

 

Expected Project Duration
2019 - 2024

← Projects Overview

Object-level scene descriptions and attention in visual search

Exploring visual scene analysis in humans and assessing its potential for use in computer vision systems

Photo: Unsplash.com/Freddy-Marschall

This project focuses on the potential of object-level search processes for understanding visual scene analysis in humans and assesses their potential for real-world scene interpretation and search tasks by computer vision systems. In order to perform this research, we will study an ecologically valid setting where free-viewing mode, real-world dynamic scenes, and visual exploration tasks that have to be performed simultaneously. This will provide novel insights into the question of how human subjects balance attentional processing under – potentially conflicting – task demands. To this end, we will combine eye-tracking experiments in humans with the computational modelling of fixation sequences, with algorithm development, and with the design of computer vision systems using foveated attention and camera movements akin to eye movements.

Wirth, L., Shurygina, O., Rolfs, M., & Ohl, S. (2022). Target-location rather than target-object specific saccadic selection in visual working memory. PERCEPTION / 44th European Conference on Visual Perception (ECVP) 2022. https://journals.sagepub.com/doi/10.1177/03010066221141167
Shurygina, O., Pooresmaeili, A., & Rolfs, M. (2021). Pre-saccadic attention spreads to stimuli forming a perceptual group with the saccade target. Cortex, 140, 179–198. https://doi.org/10.1016/j.cortex.2021.03.020
Shurygina, O., & Rolfs, M. (2022). Saccade kinematics reflect object-based attention in realistic but not in simplified stimuli. PERCEPTION / 44th European Conference on Visual Perception (ECVP) 2022.
Shurygina, O., & Rolfs, M. (2022). Eye movement characteristics reflect object-based attention. Journal of Vision / VSS 2022, 3481–3481. https://doi.org/10.1167/jov.22.14.3481
Shurygina, O., & Rolfs, M. (2021). Visual sensitivity and reaction time measures show no evidence for purely exogenous object-based attention. Journal of Vision / VSS 2021, 2571–2571. https://doi.org/10.1167/jov.21.9.2571
Schmittwilken, L., Wichmann, Felix. A., & Maertens, M. (2024). Fixational eye movements enable robust edge detection. Journal of Vision, 222, 108450. https://doi.org/10.1016/j.visres.2024.108450
Schmittwilken, L., Matic, M., Maertens, M., & Vincent, J. (2021). BRENCH: An open-source framework for b(r)enchmarking brightness models. Journal of Vision. https://doi.org/10.1167/jov.22.3.36
Schmittwilken, L., & Maertens, M. (2022). Fixational eye movements enable robust edge detection. Journal of Vision. https://doi.org/10.1167/jov.22.8.5
Schmittwilken, L., & Maertens, M. (2022). Medium spatial frequencies mask edges most effectively. Journal of Vision. https://doi.org/10.1167/jov.22.14.4041
Roth, N., Rolfs, M., Hellwich, O., & Obermayer, K. (2023). Objects guide human gaze behavior in dynamic real-world scenes. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1011512
Roth, N., McLaughlin, J., Obermayer, K., & Rolfs, M. (2024). Gaze Behavior Reveals Expectations of Potential Scene Changes. Psychological Science, 35(12), 1350–1363. https://doi.org/10.1177/09567976241279198
Roth, N., Shurygina, O., Muscinelli, F. M., Obermayer, K., & Rolfs, M. (2023). Modeling the spread of object-based attention during free viewing. MODVIS 2023. https://docs.lib.purdue.edu/modvis/2023/session02/3/
Roth, N., Rolfs, M., & Obermayer, K. (2022). ScanDy: Simulating Realistic Human Scanpaths in Dynamic Real-World Scenes. MODVIS 2022. https://docs.lib.purdue.edu/modvis/2022/session01/6/
Roth, N., Bideau, P., Hellwich, O., Rolfs, M., & Obermayer, K. (2021). Modeling the influence of objects on saccadic decisions in dynamic real-world scenes. PERCEPTION / 43rd European Conference on Visual Perception (ECVP) 2021. https://journals.sagepub.com/doi/full/10.1177/03010066211059887
Roth, N., Bideau, P., Hellwich, O., Rolfs, M., & Obermayer, K. (2021). A modular framework for object-based saccadic decisions in dynamic scenes. CVPR EPIC Workshop / ArXiv:2106.06073. https://doi.org/10.48550/arXiv.2106.06073
Roth, N., Rolfs, M., & Obermayer, K. (2022). Scanpath prediction in dynamic real-world scenes based on object-based selection. Journal of Vision / VSS 2022. https://doi.org/10.1167/jov.22.14.4217
Roth, N., McLaughlin, J., Obermayer, K., & Rolfs, M. (2023). Looking for potential action: Differences in exploration behavior of static and (potentially) dynamic scenes. Journal of Vision. https://doi.org/10.1167/jov.23.9.5293
Muscinelli, F., Roth, N., Shurygina, O., Obermayer, K., & Rolfs, M. (2022). Object-based Spread of Attention Affects Fixation Duration During Free Viewing. PERCEPTION / 44th European Conference on Visual Perception (ECVP) 2022. https://journals.sagepub.com/doi/10.1177/03010066221141167

Research

An overview of our scientific work

See our Research Projects