Multi-perspective ontology for images in Jewish cultural heritage

Research funded by the ISF research grant 2007-2010.

The research was conducted in cooperation with Prof. Judit Bar-Ilan and Prof. Snunith Shoham and Dr. Yitzhak Miller from the Department of Information Science in Bar-Ilan University.
The research objective of this work was to develop a general framework that incorporates collaborative social tagging with a novel ontology scheme conveying multiple perspectives. We proposed a framework where multiple users tagged the same object (image in our case), and an ontology was created and extended based on these tags while being tolerant about different points of view. Both the tagging and the ontological models are intentionally designed to suit the multi-perspective environment. The proposed framework characterized the underlying processes for a collaborative development of a multi-perspective ontology and its application to improve image annotation, searching and browsing.
In order to construct the ontology each user initially tagged the images without seeing the tags provided by the other users. Then, the users saw the tags assigned by others and were also encouraged to interact. Results show that after the social interaction phase the tag sets converged and the popular tags became even more popular. Even though in all cases the total number of assigned tags increased after the social interaction phase, the number of distinct tags decreased in most cases. When viewing the image only, in some cases the users were not able to correctly identify what they saw in some of the pictures, but overcame the initial difficulties after interaction. From this experiment we concluded that social interaction may lead to convergence in tagging and that the “wisdom of the crowds” helps overcome the difficulties due to the lack of information.
Once the initial ontology was constructed based on the most popular user tags, we investigated the effectiveness of ontology-based interface for further image tagging and retrieval. To this end, we defined four evaluation criteria for tag quality and compared three types of user interfaces for image tagging: free-text based, ontology-based, and a mixed interface which incorporates both free-text based and ontology based tagging. We found that ontological tags always achieved a broader user agreement, the highest average popularity scores and were more stable during the tag modification stages of the experiment. On the other hand, the free-text interface when available before using or in parallel with the ontology was perceived as an easier in use option and therefore produced more tags. The conclusion was that ontology could be very effectively employed for image tagging, when no other interfaces were available at the time of or before seeing the ontology. The obtained results also revealed a complementary nature of the free-text and ontological tags, which created a basis for a dynamic process of collaborative ontology extension.
In another experiment each participant had to search images matching certain predefined scenarios, when using one of four retrieval interfaces: tag search in a search box; faceted tag search in a search box, selecting terms from the tag cloud of all the tags in the database and selecting concepts from an ontology created from the tags assigned to the images. The obtained results have shown that the highest recall on average was achieved by users of the ontology interface, for seven out of the ten tasks, however users were more satisfied with the textbox based search than the cloud or the ontology.
The significance of this research was that it focused on exploring effective ways for employment of the proposed multi-perspective ontological model both for retrieval and tagging of visual objects.
An additional contribution of this project was a creation of the annotated image collection of hundreds of pictures in the area of Jewish cultural heritage, which were also indexed by the ontology concepts and thus could be effectively retrieved.


  1. Zhitomirsky-Geffet  M., J. Bar-Ilan, Y. Miller and S. Shoham. 2012. Exploring the effectiveness of folksonomy based tagging vs. free text tagging. Book Chapter in Indexing and Retrieval of Non-Text Information. Ed. by Rasmussen Neal, Diane. Series: Knowledge and Information / Studies in Information Science.
  2. Bar-Ilan, J., Zhitomirsky-Geffet Maayan, Yitzhak Miller, and Snunith Shoham. 2012. Tag-based Retrieval of Images through Different Interfaces – A User Study. Online Information Review.
  3. Bar-Ilan, J., Zhitomirsky-Geffet Maayan, Yitzhak Miller, and Snunith Shoham. “The effects of background information and social interaction on image tagging”. The Journal of the American Society for Information Science and Technology (JASIST), 61(5), 940-951. 2010.
  4. Bar-Ilan, J., Zhitomirsky-Geffet, M.,  Miller, Y. and Shoham, S. “Tag cloud and ontology based retrieval of images”. In Proceedings of the Third Symposium of Information Interaction in Context (IIiX), 2010, pp. 85-94.
  5. Zhitomirsky-Geffet  M., J. Bar-Ilan, Y. Miller and S. Shoham. “A Generic Framework for Collaborative Multi-perspective Ontology Acquisition”. Online Information Review. Vol. 1. 2010.
  6. Zhitomirsky-Geffet  M., J. Bar-Ilan, Y. Miller and S. Shoham. “A Generic Framework for Collaborative Multi-perspective Ontology Acquisition”. The 17th International World Wide Web Conference (WWW2008). Beijing, China. 2008.