How See the Colorful Scenery?

2022. 05. 26

How See the Colorful Scenery?: The Color-centered Descriptive Text Generation for the Visually Impaired in Japan

It is difficult for the visually impaired to understand landscapes and visual arts. Since 2010s, technology development for converting visual information into tactile or auditory information has been investigated to improve the contents understanding of visually impaired.From this background, applications describing user’s surroundings from image recognition technology has been developed. However, these applications currently generate little description of color, which is insufficient for visually impaired people to enjoy landscapes and visual arts. Therefore, in this research, we have added additional explanation and experimented the viewing experience for the visually impaired on the focus to avoid elements that confuse the user from excessive amount of features.
First, we have discovered that colors are described intensively comparing to shapes and sizes from analyzing Japanese audio descriptions, and generated additional description of them in the implemented system. Next, we have evaluated the implemented scene describing system by conducting experiment on 17 visually impaired. As a result, there was an opinion that it became possible to imagine colors. Nevertheless, there were some opinions that explanation was difficult to understand. These were three types of opinions. 1.Unknown or abstract words 2.Unnatural combination of words 3.The big shift from the first imagination. As the future prospect, we aim to well assist the visually impaired to understand the landscape in detail and instantly by generate a corpus based on explanatory texts for people with disabilities, such as audio guides, and to generate sentences that take into account word-for-word affiliation and the order of the generated sentences.

Authors: Chieko Nishimura, Naruya Kondo, Takahito Murakami, Maya Torii, Ryogo Niwa, Yoichi Ochiai