Organizing photos

Introduction

Photos play a major in the personal preservation scenario. Particularly, photos are commonly used to remember and reminisce humans' life situations and, hence, earn special support to keep them at hand and preserve them. Thus, for ForgetIT's Personal Preservation application scenario, the Semantic Desktop also supports to manage photos. As a proof-of-concept for the benefits of the PIMO and preservation functionality, the PIMO infrastructure was extended with a photo organization app embedded in the PIMO5 user interface.

Organizing photos here includes uploading, sharing, and annotating photos, but also to decide which photos to rather keep (show, preserve) and which photos to rather delete (hide, forget). You can upload single photos and treat them individually. However, photos are often part of a bigger event, be it major life situations, parties, or alike. Photos are often taken in a group and, hence, they are often managed in photo collections rather than treated individually. In the following, managing a photo collection is shown step-by-step starting with importing to annotation.

4.1.1. Importing photos with the SemanticFileExplorer

The SemanticFileExplorer integration allows an easy creation of a photo collection as life situation in the PIMO.

4.1.2. Browsing & Annotating a life situation in PIMO5

In PIMO5, a life situation with its contained photos can be browsed & annotated. Embedding the Semantic Editor seed for adding notes to life situations (and images) allows for an easy annotation of the photo collection. Various possibilities will be shown with the net videos. This all supports an early contextualization of the photo collection which again also contributes for contextual remembering and synergetic preservation.

4.1.3. Annotating photos in PIMO5

As with a life situation, also photos can be described via notes using seed. Annotations of a photo are propagated to the life situation and listed as "aggregated topic".

4.1.4. Searching photos in PIMO5

In the following video, we show how the aggregated topic can be used in searching: when annotating a photo, this annotation is also added to the life situation, i.e., the life situation is described by the sum of all annotations contained in its photo collection. Thus, a life situation can be found by a topic only used as a description of a photo. Further, the annotations are also used in the PIMO, thus, we can find more information on things such as Cullen Skink. What this is? See the video.

4.1.5. Browsing the life situation with the SemanticFileExplorer

All annotations we did in PIMO5 can also be seen in the Windows File Explorer using the SFE sidebar. The following video shows a short browsing of the folder and annotated images.

Pilot II

In Pilot II, the PIMO Photo Organization app was extended with further functionality. In this Section, we list major enhancements to this app which are also used in the final evaluation of the personal preservation application scenario. A more detailed and step-by-step documentation is given in PIMO Photo Organization App Functions Explained.

Using CERTH media technology

In cooperation with CERTH ITI (The Centre for Research & Technology, Hellas, Information Technologies Institute), we use in the pilots their multimedia analysis services for image quality assessment (4.1.7), visual concept detection (4.1.8), and near duplicate detection (4.1.9).

4.1.6. Rating Images

One major change in the user interface was the removal of the keep/delete slider used in the PIMO Fringe evaluation (see D2.1, D9.5). The evaluation identified 4 categories as sufficient classification for a user's ratings in our setting, plus “trash” to delete an image. In the new UI, buttons now allow a rating of an individual image (see Figure below) with the following meaning (in parentheses is the respective icon used):

favourite (star): This photo is one of the user’s favourites. It will be shown in the home page of the photo collection as a thumbnail in the overview image gallery (see Figure 3) as well as used in the default set of the photo collection’s slideshow (by pressing the button “slideshow”; which is also a new feature in Pilot II).
show (check mark): This photo is shown if the user selects “Show whole collection” in Figure 3 together with the favourites.
not decided yet (question mark): No decision has been done yet. The photo is treated as if the “show” has been selected, i.e., the photo will be shown in the photo collection. The yellow question mark indicates that no decision has been done yet.
hide (X): This photo should be hidden in the photo collection but neither removed nor trashed. This might apply to those photos which serve a mere documentary purpose and should not hinder the enjoyment of watching the photo collection (e.g., a boring photo of the hotel bed as in Figure 2). Such photos can still be shown if the user sets the filter to show hidden images.
trash (trash bin): This photo shall be trashed. Similar to “hide”, the photo will be hid- den directly, can be shown when the filter is set to “trash”, but after a certain time, the photo will be deleted. Then a retrieval is not possible anymore (unless it was preserved beforehand).

The overview page of a photo collection in PIMO5 — The entry page of a photo collection shows the favourite images in a thumbnail gallery, all annotations in the photo collection, and a descriptive text. This overview serves as a condensed view of the whole photo collection.

Setting the rating for a photo also influences the Preservation Value (PV). The PV is now a part of the visible metadata of a photo as shown in Figure 4. The current result of the PV assessment is shown to the user using icons as badges in gold, silver, and bronze. This classification corresponds to the human readable categories as proposed in deliverable D3.3 with gold having the highest PV. If the photo is already preserved, the PIMOCloud (ForgetIT-) green icon is used (as introduced in Pilot I, see also D9.3; this icon is also used throughout the Semantic Desktop to indicate that a thing has been preserved; see also Figure 21).

4.1.7. Image Quality Assessment

In integrating the ForgetIT Image Services from WP4 (see D4.2, D4.3, and D4.4; and CERTH media technology in ), the Semantic Desktop infrastructure and especially the use case of photo organization benefits from the following three functionalities: The quality of an image is assessed by the image service and a number btw. 0 and 1 is returned (see Image Quality in Figure below ). This number is used, e.g., for choosing the best picture out of several equally qualified ones as representative of a diary entry (see also PIMO Diary and [Jilek et al., 2015]), for deciding which is the best picture in a set of near duplicates (see Figure below), and one of several evidences for Preservation Value calculation.

4.1.8. Visual Concept Detection

Photos can have visual concepts as indicated in the Figure below.

Exemplary visual concepts in an image which could be detected.

Using WP4’s visual concept detection, the Semantic Desktop gets a set of possible visual concepts in an image, each with a specific confidence value. The Semantic Desktop uses a subset of the proposed ones which are above a certain threshold 4 and automatically annotates an image with these concepts (To distinguish the automatically annotated concepts from manually annotated ones, those concepts are annotated using the PIMO property pimo:hasSuggestedTopic. In the PIMO, this is done every time where services propose concepts such as by using text services.)

Detected visual concepts are automatically annotated as suggested topics. The detail view shows also the confidence of the detected concept. By clicking on the magnifying glass of a detected concept, a semantic search for images and photo collections is started which are annotated with the same concept.

The automatic annotation allows to search for images with these concepts without previous manual annotation effort by the user as shown in the next Figure.

A facetted search for images using visual concepts. The screenshot shows the possibility to add even more concepts to the filter.

4.1.9. Near Duplicate Recognition

A photo collection can have very similar photos. To identify these, the near duplicate recognition service for photos in a collection is used. This service delivers none, one, or more set(s) of photos which are near duplicates of each other. As a benefit, such sets are shown in the photo collection view with a representative of the set and the near duplicates listed as thumbnails below the photo:

SWR3 concert stage at the Rheinland-Pfalz Tag — Near duplicates are shown in the photo collection as a set of images: the best of the set is shown as normal photo, its duplicates as thumbnails below.

Saga on stage at the SWR3 concert during the Rheinland-Pfalz Tag — Near duplicates are shown in the photo collection as a set of images: the best of the set is shown as normal photo, its duplicates as thumbnails below.

Furthermore, image quality and near duplicates are used to support users organizing their photo collection in the following way: During the initial upload of the photo collection, the image quality serves as an indicator to set a photo’s rating: If the image quality is high (e.g., above 0.7) the photo gets a “show” as default, if it is very low (e.g., less than 0.2), a “hide” is set. Otherwise, the “question mark” is the default to indicate that no decision has been done yet. Furthermore, for a set of near duplicates, the chosen representative gets a bonus on top of its image quality for each duplicate, thus, raising the probability that it is selected as “show”. The assumption behind this is that the scene might have been important to the user, therefore several photos were taken and/or several trials to get a good photo (as in the rightmost screenshot in the Figure above).

◀ 4.1. Organizing photos