Material Design in Augmented Reality with In-Situ Visual Feedback

Material design is the process by which artists or designers set the appearance properties of virtual surface to achieve a desired look. This process is often conducted in a virtual synthetic environment however, advances in computer vision tracking and interactive rendering now makes it possible to design materials in augmented reality (AR), rather than purely virtual synthetic, environments. However, how designing in an AR environment affects user behavior is unknown. To evaluate how work in a real environment influences the material design process, we propose a novel material design interface that allows designers
to interact with a tangible object as they specify appearance properties. The setup gives designers the opportunity to view the real-time rendering of appearance properties through a virtual reality setup as they manipulate the object. Our setup uses a camera to capture the physical surroundings of the designer to create subtle but realistic reflection effects on the virtual view superimposed on the tangible object. The effects are based on the physical lighting conditions of the actual design space. We describe a user study that compares the efficacy of our method to that of a traditional 3D virtual synthetic material design system. Both subjective feedback and quantitative analysis from our study suggest that the in-situ experience provided by our setup allows the creation of higher quality material properties and supports the sense of interaction and immersion.

Authors: W. Shi, Z. Wang, T. M. Sezgin, J Dorsey, H. Rushmeier.

Read the full paper.

Sketch-Based Articulated 3D Shape Retrieval

Sketch-based queries are a suitable and superior alternative to traditional text- and example-based queries for 3D shape retrieval. The authors developed an articulated 3D shape retrieval method that uses easy-toobtain 2D sketches. It does not require 3D example models to initiate queries but achieves accuracy comparable to a state-of-the-art example-based 3D shape retrieval method.

Authors: Y. Sahillioglu, T. M. Sezgin.

                                                    Read the full paper.

What Auto Completion Tells Us About Sketch Recognition

Auto completion is generally considered to be a difficult problem in sketch recognition as it requires a decision to be made with fewer strokes. Therefore, it is generally assumed that the classification of fully completed object sketches should yield higher accuracy rates. In this paper, we report results from a comprehensive study demonstrating that the first few strokes of an object are more important than the lastly drawn ones. Once the first few critical strokes of a symbol are observed, recognition accuracies reach a plateau and may even decline. This indicates that less is more in sketch recognition. Our results are supported by carefully designed computational experiments using Tirkaz et. al.’s sketch auto completion framework on the dataset of everyday object sketches collected by Eitz et. al

Authors: O. C. Altıok, K. T. Yesilbek, T. M. Sezgin.

Read the full paper.

Building a Gold Standard for Perceptual Sketch Similarity

Similarity is among the most basic concepts studied in psychology. Yet, there is no unique way of assessing similarity of two objects. In the sketch recognition domain, many tasks such as classification, detection or clustering require measuring the level of similarity between sketches. In this paper, we propose a carefully designed experiment setup to construct a gold standard for measuring the similarity of sketches. Our setup is based on table scaling, and allows efficient construction of a measure of similarity for large datasets containing hundreds of sketches in reasonable time scales. We report the results of an experiment involving a total of 9 unique assessors, and 8 groups of sketches, each containing 300 drawings. The results show high interrater agreement between the assessors, which makes the constructed gold standard trustworthy.

Authors: Serike Cakmak, T. Metin Sezgin.

Read the full paper.

Gaze-Based Biometric Authentication: Hand-Eye Coordination Patterns as a Biometric Trait

We propose a biometric authentication system for pointer-based systems including, but not limited to, increasingly prominent pen-based mobile devices. To unlock a mobile device equipped with our biometric authentication system, all the user needs to do is manipulate a virtual object presented on the device display. The user can select among a range of familiar manipulation tasks, namely drag, connect, maximize, minimize, and scroll. These simple tasks take around 2 seconds each and do not require any prior education or training [ÇS15]. More importantly, we have discovered that each user has a characteristic way of performing these tasks. Features that express these characteristics are hidden in the user’s accompanying hand-eye coordination, gaze, and pointer behaviors. For this reason, as the user performs any selected task, we collect his/her eye gaze and pointer movement data using an eye gaze tracker and a pointer-based input device (e.g. a pen, stylus, finger, mouse, joystick etc.), respectively. Then, we extract meaningful and distinguishing features from this ultimodal data to summarize the user’s characteristic way of performing the selected task. Finally, we authenticate the user through three layers of security: (1) user must have performed the manipulation task correctly (e.g. by drawing the correct pattern), (2) user’s hand-eye coordination and gaze behaviors while performing this task should confirm with his/her hand-eye coordination and gaze behavior model in the database, and (3) user’s pointer behavior while performing this task should confirm with his/her pointer behavior model in the database.

Authors: Cagla Cig, T. Metin Sezgin.

Read the full paper.

Semantic Sketch-Based Video Retrieval with Autocompletion

The IMOTION system is a content-based video search engine that provides fast and intuitive known item search in large video collections. User interaction consists mainly of sketching, which the system recognizes in real-time and makes suggestions based on both visual appearance of the sketch (what does the sketch look like in terms of colors, edge distribution, etc.) and semantic content (what object is the user sketching). The latter is enabled by a predictive sketch-based UI that identifies likely candidates for the sketched object via state-of-the-art sketch recognition techniques and offers on-screen completion suggestions. In this demo, we show how the sketch-based video retrieval of the IMOTION system is used in a collection of roughly 30,000 video shots. The system indexes collection data with over 30 visual features describing color, edge, motion, and semantic information. Resulting feature data is stored in ADAM, an efficient database system optimized for fast retrieval.

Authors: C. Tanase, I. Giangreco, L. Rossetto, H. Schuldt, O. Seddati, S. Dupont, O. C. Altıok, T. M. Sezgin

Read the full paper.

IMOTION – Searching for Video Sequences using Multi-Shot Sketch Queries

This paper presents the second version of the IMOTION system, a sketch-based video retrieval engine supporting multiple query paradigms. Ever since, IMOTION has supported the search for video sequences on the basis of still images, user-provided sketches, or the specification of motion via flow fields. For the second version, the functionality and the usability of the system have been improved. It now supports multiple input images (such as sketches or still frames) per query, as well as the specification of objects to be present within the target sequence. The results are either grouped by video or by sequence and the support for selective and collaborative retrieval has been improved. Special features have been added to encapsulate semantic similarity.

Authors: L. Rossetto, I. Giangreco, S. Heller, C. Tanase, H. Schuldt, O. Seddati, S. Dupont. T. M. Sezgin, O. C. Altıok, Y. Sahillioglu.

Read the full paper.

iAutoMotion – an Autonomous Content-based Video Retrieval Engine

This paper introduces iAutoMotion, an autonomous video retrieval system that requires only minimal user input. It is based on the video retrieval engine IMOTION. iAutoMotion uses a camera to capture the input for both visual and textual queries and performs query composition, retrieval, and result submission autonomously. For the visual tasks, it uses various visual features applied to the captured query images; for the textual tasks, it applies OCR and some basic natural language processing, combined with object recognition. As the iAutoMotion system does not conform to the VBS 2016 rules, it will participate as unofficial competitor and serve as a benchmark for the manually operated systems.

Authors: L. Rossetto, I. Giangreco, C. Tanase, H. Schuldt, O. Seddati, S. Dupont. T. M. Sezgin, Y. Sahillioglu.

Read the full paper.

Multimodal Data Collection of Human-Robot Humorous Interactions in the JOKER Project

Thanks to a remarkably great ability to show amusement and engagement,  aughter is one of the most important social markers in human interactions. Laughing together can actually help to set up a positive atmosphere and favors the creation of new relationships. This paper presents a data collection of social interaction dialogs involving humor between a human participant and a robot. In this work, interaction scenarios have been designed in order to study social markers such as laughter. They have been implemented within two automatic systems developed in the JOKER project: a social dialog system using paralinguistic cues and a task-based dialog system using linguistic content. One of the major contributions of this work is to provide a context to study human laughter produced during a human-robot interaction. The collected data will be used to build a generic intelligent user interface which provides a multimodal dialog system with social communication skills including humor and other informal socially oriented behaviors. This system will emphasize the fusion of verbal and non-verbal channels for emotional and social behavior perception, interaction and generation capabilities.

Read the full paper.

Keywords: Multimodal Data, Human-Robot Interaction, Humorous Robot

Authors: L. Devillers, S. Rosset, G. Dubuisson Duplessis, M. A. Sehili, L. Béchade, A. Delaborde, C. Gossart, V. Letard, F. Yang, Y. Yemez, B. B. Türker, M. Sezgin, K. El Haddad, S. Dupont, D. Luzzati, Y. Estève, E. Gilmartin, N. Campbell.