3.Video Analysis Researcher

Dr. Mika Rautiainen

Dr. Mika Rautiainen,
MediaTeam Researcher.

Dr. Mika Rautiainen has worked in MediaTeam throughout its existence. During these ten years his career path has progressed quite traditionally from a part-time trainee to a post-doctoral researcher.

Rautiainen started in MediaTeam in the autumn of 1997 as a part-time trainee. In 1998, he visited the NEC Central Research Labs in Tokyo, where he worked on camera-based document analysis systems for six months. After this trip, he continued at MediaTeam as a research trainee, and, in 2000, became a Master’s Thesis researcher. By this time, he had already participated in two projects: Countess, funded by the National Technology Agency, and IDIR. Both projects focused on the research of image retrieval, albeit from a slightly different angle, so it is no wonder that his thesis, titled "Finding semantic knowledge from images in visual information retrieval and surveillance applications", concerned information retrieval.

In 2001, after receiving his M.Sc. diploma, Rautiainen left for a six month research exchange in the Laboratory for Language and Media Processing in the University of Maryland, located in the USA, near Washington D.C. He worked under the guidance of Dr. David Doermann, the co-director of the laboratory. During his stay there, the topic for his doctoral thesis began to focus on semantic video retrieval, as he was introduced to the TREC video retrieval evaluation at NIST (National Institute for Standards and Technology). His trip to Maryland was a part of the long-running cooperation between MediaTeam and the University of Maryland. The research exchange has resulted in numerous collaborative scientific publications and influenced several doctoral theses.

After returning to Oulu, Rautiainen continued his post-graduate studies and worked as a researcher in MediaTeam. He continued working towards his thesis, especially in projects funded by the Academy of Finland, namely Semantic Gap and CBIR (Content-based Information Retrieval). He also managed MediaTeam’s participation in the worldwide TREC video retrieval evaluation.

TREC

TREC, the Text Retrieval Conference, aims at annually mapping the development of the research of information retrieval with different retrieval system evaluations. The conference is open to all research groups, and the conference reports can be found from TREC’s web site. The 16th TREC conference was held at NIST (U.S. National Institute of Standards and Technology), Gaithersburg (Maryland), U.S. in 2007.

Originally a conference focused on text document retrieval, TREC broadened its activities towards multimedia in 2001 by including video databases in its evaluations. A considerable number of international research institutes take part in the evaluations. The purpose of their participation in the joint evaluation is to improve the efficiency of content-based retrieval systems in finding relevant information. Thus, the most central aim of the conference is to promote, through science, the development of multimedia retrieval systems on a global scale.

MediaTeam has participated in TREC since 2002.

During his post-graduate time, Rautiainen also participated in other development projects, such as Mobile Kärpät, Oulu Expo, Digital Oulu Cultural Database, Vikings and several subcontracted projects related to mobile devices. In December 2006, Rautiainen defended his doctoral thesis, which focused on content-based methods that assist users in locating relevant information from heterogeneous video databases. It introduced computational methods for estimating visual and conceptual similarities between video shots and a model for the content-based browsing of video databases.

Currently, Rautiainen is working at MediaTeam as a post-doctoral researcher, and his plan is to continue that for the next three years and focus on video analysis and its usage in a multimedia information system from the point of view of storage and retrieval. He points out that video analysis can also be utilised in several other application domains, making it a broad and interesting field of research.

The thing that attracted Rautiainen to an academic career was the opportunity to innovate and work with topics that are novel and uncharted. Concretely, research work has meant plenty of reading and investigating various phenomena, as well as writing scientific articles about the findings. In industry-related projects, it has meant tighter control on what should be investigated and how deep a given subject can be delved into. MediaTeam has also given him the opportunity to conduct international research, participate in international conferences and visit foreign research institutions.

Cooperation with University of Maryland

MediaTeam has an agreement on scientific cooperation with the Laboratory for Media and Language Processing (LAMP) at the University of Maryland, USA. LAMP has been a partner in several joint research projects, such as “Cooperative research on computer vision”, “Distributed media processing in hybrid networks”, “Content-based mobile multimedia retrieval”, “Content-based information retrieval” and the CAPNET program.

MediaTeam and LAMP have done joint research on content-based multimedia retrieval, including joint participation in TREC Video Track in 2001 and 2002. MediaTeam and LAMP also have a research visit program which has facilitated about 30 3–12-month research visits by MediaTeam personnel since 1999. The research visit program has been sponsored by the National Technology Agency.

When asked about MediaTeam’s future, Rautiainen says that he believes that MediaTeam’s focus on creating scientific information will probably produce innovations and information that can be perceived and adopted as new and relevant technological knowledge. He thinks it is quite possible that MediaTeam’s future will bring technological innovations that interest the scientific and international media. The successful establishment of an internationally renowned research brand is not easy, however, and requires decades of research on potentially disruptive technological phenomena. But once established, such reputation presents an opportunity to collaborate with researchers from different cultures while, at the same time, becoming a front-liner in global technological progress. Lastly, Rautiainen encourages beginner researchers to have open-minded dialogue with colleagues, anticipate upcoming tech trends, pursue ideas to their completion and strive towards higher knowledge in their research – that is, to “ask a lot of what’s and why’s.”

Mika Rautiainen wishes the 10-year-old MediaTeam a happy birthday!

Related MediaTeam Projects

CBIR

1/2003-12/2006

The objective of the CBIR research project was to reduce the semantic gap by multidisciplinary research based on close international and domestic collaboration between researchers with backgrounds in multimedia signal processing, mathematics, information studies and linguistics, and by integrating information from several media types into an efficient multimedia analysis.

Financiers and Business Partners

  • The Academy of Finland

Semantic Gap

8/2001-7/2004

Semantic Gap was a joint project of MediaTeam and the Department of Information Studies in the Faculty of Humanities focusing on the indexing of databases and content-based retrieval of audio and video recordings. Thematically, the project was closely connected with the Vikings project, and its results were extensively tested and applied in the Vikings project.

The central aim of the project was to narrow down the semantic gap between the concept-based and content-based approaches to database indexing. By narrowing the semantic gap, it would be possible to design more and more efficient databases and search engines. The research challenges concerned booming media types, such as digital speech, music, and image, where search criteria often included semantic concepts.

The research questions represented the interface between technology and semantic/cognitive information science, and only a genuinely cross-disciplinary team could hope to tackle the problems. Eventually, the undertaking turned out to be highly successful, as the results were efficiently utilized – the search engine was benchmarked in the international VideoTREC competition, an annual conference series sponsored by the National Institute of Standards and Technology and other U.S. government agencies.

Financiers and Business Partners

  • The Academy of Finland
Vikings

Vikings

6/2000-5/2003

The Vikings project was carried out in cooperation with VTT Electronics. In the project, new content-based retrieval systems for searches in movie and sound recording databases were developed.

The project’s goals were the development of methods required in content-based multimedia retrieval, the development of novel language technology and the testing of this technology in service applications. Key technologies included digital signal processing, digital image analysis, pattern recognition, visualization, and search engine technology.

The researchers developed new artificial intelligence technologies, by means of which it was possible to detect the emotional state of speakers (with a focus on the Finnish and English languages) from the speech signal almost as automatically and successfully as people do. New image processing techniques were also developed for interpreting video content: changes in color contents in the spatial and temporal domains were measured, and the images were classified accordingly. Finally, the algorithms were integrated into a search engine that combined the audio and video features to achieve higher-level semantic presentations.

Financiers and Business Partners

  • Jutel
  • Nokia
  • OPOY/Finnet Group
  • National Technology Agency
Countess

Countess

1/1999-12/2000

In the two-year Countess project, researchers developed solutions for content-based image retrieval. The search platform prototype developed by MediaTeam researchers can be used to search for pictures in digital databases on the basis of their content.

Financiers and Business Partners

  • Acta Systems
  • OPOY/Finnet Group
  • National Technology Agency
  • Yritys-Sampo

Selected Publications

Rautiainen M, Seppänen T & Ojala T (2006) On the significance of cluster-temporal browsing for generic video retrieval - a statistical analysis. ACM Multimedia 2006, Santa Barbara, CA, 125-128.

Juuso I & Seppänen T (2006) Novel tools for creating and visualising metadata for digital movie retrieval. Digital Humanities 2006, Paris, France, 107.

Rautiainen M & Seppänen T (2005) Comparison of visual features and fusion techniques in automatic detection of concepts from news video. Proc. 2005 IEEE International Conference on Multimedia & Expo, Amsterdam, The Netherlands. Details

Lilja J, Juuso I, Kortelainen T, Seppänen T & Suominen V (2004) Mitä katsoja kertoo elokuvasta – elokuvan sisäisten elementtien tunnistaminen ja sisällönkuvailu. Informaatiotutkimus 23(3):59–69 (in Finnish).

Rautiainen M, Ojala T & Seppänen T (2004) Cluster-temporal browsing of large news video databases. Proc. 2004 IEEE International Conference on Multimedia and Expo, Taipei, Taiwan, 2:751–754. Details

Rautiainen M, Ojala T & Seppänen T (2003) Cluster-temporal video browsing with semantic filtering. Proc. Advanced Concepts for Intelligent Vision Systems, Ghent, Belgium, 116-123. Details

Rautiainen M, Penttilä J, Pietarila P, Noponen K, Hosio M, Koskela T, Mäkelä SM, Peltola J, Liu J, Ojala T & Seppänen T (2003) TRECVID 2003 experiments at MediaTeam Oulu and VTT. Proc. TRECVID Workshop at Text Retrieval Conference TREC 2003, Gaithersburg, MD. Details

 

Ojala T, Pietikäinen M & Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with Local Binary Patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7):971 - 987. Details

Rautiainen M & Doermann D (2002) Temporal color correlograms for video retrieval. Proc. 16th International Conference on Pattern Recognition, Quebec, Canada, 1:267 - 270. Details

Rautiainen M, Penttilä J, Vorobiev D, Noponen K, Väyrynen P, Hosio M, Matinmikko E, Mäkelä SM, Peltola J, Ojala T & Seppänen T (2002) TREC 2002 Video Track experiments at MediaTeam Oulu and VTT. Proc. Text Retrieval Conference TREC 2002 Video Track, Gaithersburg, MD. Details

Ojala T, Kauniskangas H, Keränen H, Matinmikko E, Aittola M, Hagelberg K, Rautiainen M & Häkkinen M (2001) CMRS : Architecture for content-based multimedia retrieval. Proc. Infotech Oulu International Workshop on Information Retrieval, Oulu, Finland, 179-190. Details

Doermann D, Sauvola J, Kauniskangas H, Shin C, Pietikäinen M & Rosenfeld A (1997) The development of a general framework for intelligent document image retrieval. In: Document Analysis Systems II, Series in Machine Perception and Artificial Intelligence, World Scientific, 28 p.

Related Dissertations

Rautiainen M (2006) Content-based search and browsing in semantic multimedia retrieval. Dissertation, Acta Univ Oul C 262, Department of Electrical and Information Engineering, University of Oulu, Finland. Details

Kauniskangas H (1999) Document image retrieval with improvements in database quality. Dissertation, Acta Univ Oul C 140, Department of Electrical Engineering, University of Oulu, Finland Details

Ojala T (1997) Nonparametric texture analysis using spatial operators, with applications in visual inspection. Dissertation, Acta Univ Oul C 105, Department of Electrical Engineering, University of Oulu, Finland. Details

Sauvola J (1997) Document analysis techniques and system components with applications in image retrieval. Dissertation, Acta Univ Oul C 98, Department of Electrical Engineering, University of Oulu, Finland Details

Related Master's Theses

Matinmikko E (2002) Image database browsing system. M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish). Details

Keränen H (2001) A mobile retrieval user interface for heterogeneous multimedia document bases. M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish). Details

Rautiainen M (2001) Finding semantic knowledge from images in visual information retrieval and surveillance applications. M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish). Details

Hagelberg K (2000) Sisältöpohjaisten kuvanhakujärjestelmien hakutekniikat. Master´s Thesis, Department of Information Processing Science, University of Oulu, Finland (in Finnish). Details

Koivusaari M (1998) Implementation of content-based document image retrieval system. M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish). Details