Note (August 2018): According to our experience from last year, we will refine the interaction logging for VBS2019. The refined format will be announced in November 2018 at the latest.
Our motivation for VBS2019 is also to collect additional information for a planned journal publication summarizing the event and analyzing the results. Therefore, with each submission, we need the teams to submit also a high-level sequence of actions that are performed before the submission. For example, if you use keyword search, browse few pages or browse on an imagemap, then you try a color sketch, browse again, then your system could log a high-level sequence K; P; …; P; C; P; …, B; … that you submit with video and shot-id to the VBS server. The log should be cleared only in the beginning of each task (VBS server will remind it). Since more users from one team can control different tools, a unique tool ID should start the log text. For multiple submissions during one task, each subsequence should end with the actual time (see examples later).
As each tool has its own set of query initialization, filtering and browsing options, the vocabulary of logged actions will consist of generalized unified (mandatory) and tool specific (optional) parts. The unified mandatory part is represented by a capital letter.
K = keyword search in automatically detected annotations (most teams use DCNNs), use this letter also for automatically detected concepts
A = using extracted audio data (usually searched by keywords as well)
O = using optical character recognition (usually searched by keywords as well)
C = search by color-sketch
E = search by edge-sketch
M = search by motion-sketch
S = query by example similarity search (when users pick an example query object from results or external source)
F = filtering using various attributes or actual dataset ordering, part of the database is cut off
P = paging, visiting a next/previous page in the actual ordering
B = browsing using a tool specific browsing system, e.g., zoom in/out in a hierarchical imagemap
T = the tool used external results from other team members
X = whenever you stop the actual search strategy and start from the scratch (reset all)
– = if you turn off just one action, for example, if you turn off edge-sketch you print -E
In brackets, additional brief tool-specific details can be provided for each letter, for example K(value, position, annotation source, …); P(next); B(zoomout); … If you perform more actions in one step, you can just concatenate them …; CE; … represents using color-edge sketch at once or …; CF(0.2cr); … represents using only 20% of the ranking by color.
Interaction Logging: A Few Examples of Such Log Messages
“TID1;K(horse)F(0.3kr);P(next);P(next);time 23:45:02” telling tool id 1 submitted at 23:45:02 the sequence: keyword search by “horse” considering just 30% of the dataset, 2x next page, before submit
“TID1;K(horse)F(0.3kr);C(rerank);time 23:45:02” telling tool id 1 submitted at 23:45:02 the sequence: keyword search by “horse” considering just 30% of the dataset and then reranked by color before submit
“TID1;M;B(zoomin);B(pan);time 23:45:02” telling tool id 1 submitted at 23:45:02 the sequence: motion sketch query and then two specific browse actions before submit
“TID1;M(5s);B(10s,zoomin);B(15s,pan);time 23:45:02;X(25s);C(30s);P(35s,next);P(42s,next);time 23:47:02” example of the log for two submissions during one task, user decided to cancel all previous actions by X, the number of seconds (since the log was resetted) is logged for each action
Interaction Logging: Further Notes
- The tool specific information in parentheses will be used mainly to better understand ambiguities (e.g., if two annotation sources are used), BUT most of the planned statistics will consider only the high-level actions (% of browsing actions, how many teams used a color-sketch in visual tasks, etc.). Therefore, if it is difficult for you to implement the logging of the tool specific information, please provide at least the letter sequences.
- Instead of comma, semicolon is used to separate actions in the log. Tool specific items are still separated by comma.
- The tool performance AND the ability to submit the logs will play an important role for the participation on the planned journal.
- Please provide also the information about the actual second for the submission time attribute “TID1;K(horse)F(0.3kr);P(next);P(next);time 23:45:02” = submission at 23 hours, 45 minutes and 2 seconds
- If possible, could you please provide a time stamp info as the first attribute in brackets about each logged action? Time stamp of an action = the number of seconds after the log was resetted. “TID1;K(12s,horse)F(0.3kr);P(20s,next);P(30s,next);time 23:45:02”
- If you have any further questions, please don’t hesitate to contact Jakub Lokoc for more information.