• Tangible Intelligent Interfaces for Teaching Computational Thinking Skills, Scientific & Technological Research Council of Turkey, High Priority Areas R&D Program, (Principal Investigator, Koç University), 2019 – 2021, 1003S
• Backchannel Feedback Modeling for Human-Computer Interaction ( E.Erzin, Y. Yemez, T.M. Sezgin). Funded by Scientific & Technological Research Council of Turkey, 2018-2020
• JOKER
  European Commission ERA-NET Program, 2013-2017 The JOKER project will build and develop a generic intelligent user interface providing a multimodal dialogue system with social communication skills including humor, empathy, compassion, charm, and other informal socially-oriented behavior.
• iMotion
  European Commission ERA-NET Program, 2013-2017 The IMOTION project will develop and evaluate innovative multi-modal user interfaces for interacting with augmented videos. Starting with an extension of existing query paradigms (keyword search in manual annotations), image search (query by example in key frames), IMOTION will consider novel sketch- and speech-based user interfaces.

• ASC-Inclusion (Sponsored by the European Commission) 2011-2014.

The main goal of this project is to develop a computer software program that will assist children with Autism Spectrum Conditions (ASC) to understand and express emotions through facial expressions, tone-of-voice and body gestures.This software will assist them to understand and interact with other people, and as a result, will increase their inclusion in society. Academic partners include University of Cambridge, United Kingdom, Technische Universität München, Germany, Bar Ilan University, Israel, Koç University, Turkey, and Università degli Studi di Genova, Italy.

• Intelligent Interfaces for eLearning
  Scientific & Technological Research Council of Turkey, 2013-2016.

The goal of this project is to build the pen-based interfaces for the classroom of the future, and it is funded under the National Priority Areas R&D Program of the Research Council of Turkey (TUBITAK). The scope of the project is not public at the moment. Contact Dr. Sezgin for details.

• Semi-supervised Intelligent Multimodal Content Translator for Smart TVs
  SANTEZ Programme, Ministry of Science, Industry, and Technology, Turkey 2012-2014

TVs are slowly morphing into powerful set-top computers with internet connections. As such, they slowly assume role of take over roles and functions that were traditionally associated with desktop computers. TV users, for example, can use their TV for browsing the internet. Unfortunately, the vast majority of the content in the internet has been designed for desktop viewing, hence they have to be adapted for viewing on a TV. In this Project, we aim to develop a semi-automatic content retargeting system, which is expected to work with minimal intervention of an expert.

• Gesture-Based Interfaces
  Koç Sistem R&D Programme, 2011-2013
• Pen-based Multimodal Intelligent User Interfaces
  Career Grant, Scientific and Technological Research Council of Turkey, 2011-2014
• Educational Sketch-Based Intelligent Interfaces
  Turk Telekom R&D Programme, 2010-2013
• Interactive Intelligent Sketching Board
  KOLT Teaching Innovation Grant, 2010-2011
• Deep Green: Commander’s Associate
  DARPA/BAE/SIFT (British Aerospace/Smart Information Flow Technologies), 2008-2009
• Deep Green: Commander’s Associate
  DARPA/SAIC (Science Applications International Corporation), 2008-2009

• Early Processing for Sketch Recognition

Freehand sketching is a natural and crucial part of everyday human interaction, especially important in design, yet is unsupported by current design automation software. We are working to combine the flexibility and ease of use of paper and pencil with the processing power of a computer, to produce a design environment that feels as natural as paper, yet is considerably smarter. One of the most basic steps in accomplishing this is converting the original digitized pen strokes in the sketch into the intended geometric objects. We have implemented a system that combines multiple sources of knowledge to provide robust early processing for freehand sketching.

Selected Publications

• Tevfik Metin Sezgin. Feature Point Detection and Curve Approximation for Early Processing of Free-Hand Sketches.Master’s Thesis. May 2001. Department of EECS, MIT.
• Tevfik Metin Sezgin and Randall Davis. Handling Overtraced Strokes in Hand-Drawn Sketches. In Making Pen-Based Interaction Intelligent and Natural . 2004.
  [ BibTeX ][ PDF ][ PS ]
• Tevfik Metin Sezgin and Randall Davis. Scale-space Based Feature Point Detection for Digital Ink. In Making Pen-Based Interaction Intelligent and Natural . 2004.
  [ BibTeX ][ PDF ][ PS ]
• Tevfik Metin Sezgin, Thomas Stahovich, and Randall Davis. Sketch Based Interfaces: Early Processing for Sketch Understanding.Workshop on Perceptive User Interfaces, Orlando FL . 2001.
  [ BibTeX ][ PDF ][ PS ]
• Tevfik Metin Sezgin and Randall Davis. Early Sketch Processing with Application in HMM Based Sketch Recognition. In MIT Computer Science and Artificial Intelligence Laboratory Technical Report AIM-2004-016, July 2004.
  [ PDF ][ PS ]

 

Sketch Recognition

A major portion of pen-centric research has revolved around the goal of enabling natural human-computer interaction. We believe progress in recognition techniques is critical to achieving the goal of natural sketch-based interfaces. We need to improve over the existing recognition algorithms in terms of efficiency and recognition accuracy. Our work in recognizing sketches using temporal patterns that naturally appear in online sketching contributes toward addressing these algorithmic issues.

Our analysis of real sketch examples from target user groups has revealed that individuals have personal sketching styles manifested in the form of patterns in temporal stroke orderings (i.e., people tend to use predictable stroke orderings during sketching). Based on this finding, we have developed two algorithms that use ensembles of Hidden Markov Models (HMMs) and Dynamic Bayesian Networks (DBNs) to learn temporal patterns in stroke orderings and perform efficient recognition.

Selected Publications

• Tevfik Metin Sezgin and Randall Davis.Temporal Sketch Recognition in Interspersed Drawings. Fourth Eurographics Workshop on Sketch-Based Interfaces and Modeling, University of California, Riverside, CA, August 2-3, (2007).
• Tevfik Metin Sezgin Overview of Recent Work in Pen-Centric Computing: Vision and Research Summary. In Invited Workshop on Pen-Centric Computing Research, Brown University, March 26-28 2007.
• Tevfik Metin Sezgin and Randall Davis. Sketch Interpretation Using Multiscale Models of Temporal Patterns. In IEEE Journal of Computer Graphics and Applications,Volume: 27,  Issue: 1, pp: 28-37, 2007.
  [ PDF ][ PS ]
• Tevfik Metin Sezgin and Randall Davis. HMM-Based Efficient Sketch Recognition. In Proceedings of the International Conference on Intelligent User Interfaces(IUI’05). New York, New York, January 9-12 2005.
  [ BibTeX ][ Extended PDF ][ Extended PS ][ PDF ][ PS ][ PPT ]
• Tevfik Metin Sezgin and Randall Davis. Modeling Sketching as a Dynamic Process. In CSW ’05 Gloucester, MA . 2005.
  [ BibTeX ][ PDF ][ PS ]
• Tevfik Metin Sezgin and Randall Davis. Efficient search space exploration for sketch recognition. In MIT Computer Science and Artificial Intelligence Laboratory Annual Research Abstract. 2004.
  [ BibTeX ][ PDF ][ PS ]
• Tevfik Metin Sezgin and Randall Davis. Early Sketch Processing with Application in HMM Based Sketch Recognition. In MIT Computer Science and Artificial Intelligence Laboratory Technical Report AIM-2004-016, July 2004.
  [ PDF ][ PS ]
• Tevfik Metin Sezgin. Generic and HMM based approaches to freehand sketch recognition. Proceedings of the MIT Student Oxygen Workshop. 2003.
  [ BibTeX ][ PDF ][ PS ]
• Tevfik Metin Sezgin. Recognition efficiency issues for freehand sketches.Proceedings of the MIT Student Oxygen Workshop. 2003.
  [ BibTeX ][ PDF ][ PS ]
• Tracy Hammond, Metin Sezgin, Olya Veselova, Aaron Adler, Michael Oltmans, Christine Alvarado, and Rebecca Hitchcock. Multi-Domain Sketch Recognition.Proceedings of the 2nd Annual MIT Student Oxygen Workshop . 2002.
• Tevfik Metin Sezgin. Generating Domain Specific Sketch Recognizers From Object Descriptions.Proceedings of the MIT Student Oxygen Workshop. 2002.
  [ BibTeX ][ PDF ][ PS ]
• Christine Alvarado, Metin Sezgin, Dana Scott, Tracy Hammond, Zardosht Kasheff, Michael Oltmans, and Randall Davis. A Framework for Multi-Domain Sketch Recognition. In MIT Artificial Intelligence Laboratory Annual Abstract . September 2001.
  [ BibTeX ][ PDF ][ PS ]

 

Readily Deployable Sketch-Based Applications

Part of our current research effort aims to construct and evaluate sketch-based applications for domains where recognition is robust enough to allow the deployment of these systems in real settings. Unlike my work in developing sketch recognition algorithms, in this line of research, the emphasis is on building systems that can readily be adopted by the intended audience and immediately integrated into their work flow. Therefore, the focus is on the construction and evaluation of pen-based interfaces for domains that are simple enough to yield reasonably high recognition rates with the current state of art in sketch recognition. Graphs and Course of Action Diagrams are two such domains.

Graph Manipulation

Along with collaborators, we developed and evaluated an application that allows computer science students to draw and interact with directed and undirected graphs using a pen-based interface. The recognition engine of this application used a variety of methods including Kohonen networks, iterative closest point and parallel sampling algorithms for recognizing user-drawn graphs and digits. Watch these clips to see this tool in action: [Clip1] [Clip 2]. You’ll need the Camtasiacodec to play the videos.

Course of Action Diagram Recognition

Course of action diagrams are drawings constructed by military commanders to depict military scenarios (e.g., locations and movements of friendly and enemy units). They are typically drawn by hand on layers of acetate overlaid on top of maps. We are currently working on systems that can recognize course of action diagrams as they are drawn. This is a three year long project and at the moment there is funding for two PhD students. We’re also looking for summer interns to work on related projects.

This project is in collaboration with Dr. Hammond from Texas A&M University and Dr. Alvarado from Harvey Mudd College, USA.

Selected Publications

• Blessing, T. M. Sezgin, R. Arandjelovic, P. Robinson, A multimodal interface for road design.Workshop on Sketch Recognition, International Conference on Intelligent User Interfaces, Sanibel, FL, February (2009).
• Hamdi Dibeklioglu, Tevfik Metin Sezgin and Ender Ozcan A Recognizer for Free-Hand Graph Drawings. In International Workshop on Pen-Based Learning Technologies, Catania, Italy, May 24-25, 2007.
  [ Extended PDF ][ Extended PS ][ PDF ][ PS ]
• Tevfik Metin Sezgin Overview of Recent Work in Pen-Centric Computing: Vision and Research Summary. In Invited Workshop on Pen-Centric Computing Research, Brown University, March 26-28 2007.
  [ PDF ][ PS ]

 

Affective Computing and Applications

In collaboration with colleagues from University of Cambridge, we are exploring ways of animating avatars to display emotions as people do. Our primary interest is in applications of machine learning for inferring people’s affective state and affective animation of avatars.

Driver Monitoring and Intelligent Interfaces for Automobiles

Automatic recognition of drivers’ affective state has received interest as a potential source of information for in-car driver monitoring systems. Although there have been studies describing the use of relatively invasive physiological measurements and expensive eye tracking information, facial appearance data has not been explored as much. We have investigated ways of inferring physical and mental states of drivers from video data. We compiled a video corpus by recording drivers subjected to a set of controlled driving conditions in a driving simulator. We are currently exploring ways of automatically processing the video data to facilitate higher fidelity annotation and mental state recognition. We’re looking for MS and PhD students to work on related projects.

Publications

• Afzal,T. M. Sezgin, Y. Gao, P. Robinson, Perception of Emotional Expressions through Facial Feature Points.International Conference on Affective Computing and Intelligent Interaction, Amsterdam, Netherlands, September 10-11, (2009).
• Gao,T. M. Sezgin, N. Dodgson., Automatic construction of 3D animatable facial models.International Conference on Computer Animation and Social Agents, Amsterdam, Netherlands, June 17-19, (2009).
• M. Sezgin, I. Davies, P. Robinson, Multimodal inference for driver-vehicle interaction.Workshop on Multimodal Interfaces for Automotive Applications, International Conference on Intelligent User Interfaces, Sanibel, FL, February (2009).

Tevfik Metin Sezgin, Peter Robinson, Affective Video Data Collection Using an Automobile Simulator. Second International Conference on Affective Computing and Intelligent Interaction, Lisbon, Portugal, September 12-14, (2007).

Xueni Pan, Marco Gillies, Tevfik Metin Sezgin, Celine Loscos, Expressing Complex Mental States Through Facial Expressions. Second International Conference on Affective Computing and Intelligent Interaction, Lisbon, Portugal, September 12-14, (2007).