We have finished the MSR Image Recognition Challenge (IRC) @ IEEE ICME 2016. More details here (opens in new tab).

(Please see FAQ for frequently asked questions)

We just finished the evaluation! More details below.

Important Dates:

• The dataset for this challenge is described here and can be downloaded here (opens in new tab).
• Feb 23, 2016: Registration web site is opened, details (opens in new tab).
• Feb 23, 2016: ICME site is open, please register a placeholder for your final report: select Track =”Grand Challenges” and select Subject Area = “MSR Grand Challenge: Image Recognition Challenge”
• Feb 24, 2016: Update about data sets, and faq, details (opens in new tab).
• Feb 26, 2016: Update about sample codes, and faq, details (opens in new tab).
• March 3, 2016: Update about test tool, team keys, and faq, details (opens in new tab).
• March 7~10, 2016: Dry run traffic sent to your system for testing/verification, and faq, details (opens in new tab).
• March 10, 2016: Update about final evaluation and faq, details (opens in new tab).
• March 14, 2016: Evaluation started, please keep your system running stably
• March 16, 2016: Evaluation ends (0:00am PDT)
• March 21, 2016: Evaluation results announced (see the rank table below)
• April 3, 2016: Grand Challenge Paper and Data Submission, details (opens in new tab).
• April 28: Paper acceptance notification
• May 13: Paper camera ready version due

We have finished the challenge in ACM MM 2015. More details here (opens in new tab).

Important Dates:

• Dataset available for download (Clickture-Lite) and hard-disk delivery (Clickture-Full).
• June 18, 2015: Trial set available for download and test.
• June 24, 2015: Final evaluation set for Task#1 available for download (encrypted)
• June 26, 2015: Evaluation starts (0:00am PDT)
• June 27, 2015: Evaluation ends (0:00am PDT)
• June 28, 2015: Evaluation results announce.
• July 7, 2015: Paper submission deadline
• July 27, 2015: Notification of Acceptance:
• August 15, 2015: Camera-ready Submission Deadline
• October 28,2015: Grand Challenge Workshop (opens in new tab)

Latest updates:

• May 22, 2015: Pre-registration form available.
• June 11, 2015: Training data set ready for downloading: details (opens in new tab)
• June 18, 2015: Trial set for Task#1 is available for download (the same as ACM MM 2014)
• June 18, 2015: Trial code samples for Task#2 is delivered by email. Contact us if you haven’t received it.
• June 19, 2015: Test tool for Task#2 is delivered by email. Contact us if you haven’t received it.
• June 24, 2015: Evaluation set for Task#1 available at here (opens in new tab) (encrypted), please download and unzip it
• June 24~June 25,2015: For task#2, dry run traffic will be sent to your recognition service,please keep your recognition service running!
• June 26, 2015: Password to decrypt Task#1 evaluation data is delivered to all participants by email on 0:00am PST, please let us know if you haven’t received it.
• June 28, 2015: Evaluation results are sent back to teams
• July 1, 2015: Evaluation result summary:

Important dates:

• April 21, 2015: Final evaluation set available for download.
• April 24, 2015: Evaluation starts (password for decrypt the evaluation set delivered at 2:30am on April 24, 2015, PDT)
• April 25, 2015: Evaluation end at 3:00AM PDT (very beginning of April 25). Result submission due.
• April 28, 2015: Evaluation results has been sent to corresponding teams.
• May 1, 2015: Paper submission (please follow the guideline of the main conference)
• May 10, 2015: Notification
• May 15, 2015: Paper camera ready due

Updates:

• May 1, 2015: More details about the evaluation results are shared with participants.
• April 28, 2015: Evaluation result sent back to corresponding teams.
• April 23, 2015: Password to decode the evaluation set was delivered to all participants by email. Please contact us if you haven’t recevied it.
• April 21, 2015: Evaluation set available.
• April 17, 2015: Add one more contact: Yuxiao Hu (yuxhu@microsoft.com)
• April 17, 2015: Trial set is available for download (the same as ACM MM 2014).
• April 17, 2015: CMT Website is online.

Details:

More details about the challenge, please visit:

1. The grand challenge page at ACM Multimedia 2014 (opens in new tab).
2. IRC @ MM 14 at this site.

Latest announcement will be posted here:

Updates:

• July 5, 2014: Evaluation results:
• June 26, 2014: Due to many requests, the MM14 grand challenge submission deadline was extended for a week. So we also extend MSR-Bing challenge result submission deadline for one week. Please check the updated dates below.
• June 25, 2014: Encrypted evaluation dataset is available for download now. Please follow the below steps to submit your prediction results:

1. Register a “paper” entry. Make sure to finish this step ASAP (at the latest 30 minutes before the challenge starts). Password to decrypt the evaluation set will be set through CMT.
2. Download the encrypted evaluation dataset. Please note the downloaded file was zipped twice (once with a password and once not).
3. Unzip the downloaded file (without password) to make sure the file is not corrupted.
4. Unzip the file you get from Step C with the password that will be sent to you through CMT. You will then get two files: one is a (key, image thumbnail) table, and the other is a (key, label) table. Please refer to this page know the details how to do generate prediction results.
5. Before the end of the challenge, submit your prediction results (up to 6 zipped files – see instructions below).
6. Submit your grand challenge paper according to the guideline in the ACM Multimedia 2014 website. Please note the CMT site is only for prediction results submission. Your paper should be submitted to EasyChair paper system. Make sure that you include your evaluation results in the paper (which will be sent to you before the paper submission deadline).

• June 25, 2014: Evaluation set will be available by EOD today. CMT will be also online at the same time.

Instructions: You are requested to register an entry at the CMT site to receive the password to decrypt the evaluation set as well as submit your prediction results. Please note prediction results based on Clickture-Lite (1M images) are mandatory, while the results on Clickture-Full (40M images) are optional. When submitting the prediction results, please name the files appropriately so we know which are based on 1M dataset (include “1M” in the file name) and which are based on 40M dataset (include 40M in the file name), as well as which are master runs (include “master” in the file name). If you submitted results based on both datasets, you are allowed to submit three runs for each dataset (including one master run for each dataset). Please note final evaluation will be based on the master runs though we will also return you the scores for other runs. (New!)

• June 25, 2014: Evaluation starts and ends dates changed (1 day delay).
• June 19, 2014: Trial set is available.

Schedule (updated on June 26, 2014):

• Feb 15, 2014: Dataset available for download (Clickture-Lite) and hard-disk delivery (Clickture-Full).
• June 18, 2014: Trail set available for download and test.
• June 25, 2014: Final evaluation set available for download (encrypted)
• July 3, 2014 (updated/firm): Evaluation starts (password for decrypt the evaluation set delivers at 11:30pm on July 2, PDT)
• July 4, 2014 (updated/firm): Evaluation end at 0:00AM PDT (very beginning of July 4)/Result submission due
• July 5, 2014: Evaluation results announce.
• July 6, 2014: Paper submission (please follow the guideline of the main conference)

• Clickture (opens in new tab)

Here are some frequently asked questions regarding IRC and clickture-dog data:

Q: Can I use ImageNet or other data for network pre-training?

A: Yes, you can use pre-trained CNN models (by ImageNet or other dataset), as long as you can fine tune that well to fit for the dog breed categories. Basically, we treat these pre-trained CNN models as feature extraction layer. Your system/algorithm output will be evaluated and compared with other team’s results.

But, if you use some data other than Clickture dataset as positive/negative examples during the training/tuning, please describe this clearly and notify us before the evaluation starts. Your algorithm/system will still be evaluated, we are considering to rank these systems in separate tracks for fairness.

Q: We found there are some noise/conflicting labels in Clickure-Dog dataset. Is this expected?

A: You may find that there are ~15K images appear in multiple dog breeds. The reason is as follows: we use the clicked queries as the “groundtruth” of dog breeds, but sometimes the same images are returned/clicked by multiple dog breed queries. We intentionally to NOT remove these 15K images from the data. Because we treat this as an important/interest topic to automatically remove the “wrong/conflicting” samples and keep the “correct” labels, during data collection and training.

Q: How is the evaluation data set constructed?

A: We extracted all the dog breeds that have matched names in pairs in Clickture Full, which result in 344 dog breeds in Clickture-Dog data set. The evaluation dataset will include 100 categories, including part of these 106 dog breeds which have more than 100 samples in Clickture-Dog dataset. But we will also include a few categories with small number (i.e. <100) of samples. This is to encourage participants to train a recognizer that can recognize as many as possible dog breeds. The Clickture-Dog is just one way to collect training data. You can also filter the Clickture-Full to find more data. This year, we also allow participants to collect training data outside of Clickture by themselves, but we will only control the evaluation set to define the problem.

Q: How to extract the images and associating class labels from clickture_dog_thumb.tsv file?

A: For the Clickture-Dog dataset, there are 95119 lines in the clickture_dog_thumb.tsv files, each is a data sample. There are three columns (delimited by “\t”) in each line. A sample line from clickture_dog_thumb.tsv file is shown below:

affenpinscher /LUkKqfrtLwqEw /9j/4AAQSkZJRgABAQEAAAAAAAD/2wBDAAoHBwgHBgoICAgLCgoLDhgQDg0NDh0VFhEYIx8lJCI…

the first column (“affenpinscher”) is the dog breed label, If you count all the unique dog breed strings in the first column, there are 344 dog breeds in this Clickture-Dog dataset.

the second column (“/LUkKqfrtLwqEw”) is the unique key for this record (which can be used to locate the corresponded record in the Clickture-Full dataset).

the third column (“/9j/4AAQSkZJR…”) is the jpeg image encoded as base64, which can be saved to file by “File.WriteAllBytes (“Sample.jpg”, Convert.FromBase64String(ImageData))” in C#. You can also use this website (opens in new tab) to decode this base64Encoded image data and save it to a jpg file manually.

Q: We are trying to run the sample server on our Linux machine and we are getting an Unhandled System.TypeLoadException?

A: Please install FSharp first and give it a try again: “sudo apt-get install fsharp”. If it still doesn’t work, please run the program with: “MONO_LOG_LEVEL=debug mono [***the exe program***]” and send us all the output.

Q: We updated the sample code with team name and GUIDs, it compiled and started successfuly, we run the CommandLineTool to test it, both Ping and Check steps work well, butwe see “Error: Connected to Prajna Hub Gateway, but get empty response, please check classifier xxxx and its response format” when sending the Recog request in step 2.(3),

A: Please check:

1. 1. Use some network monitoring tool to check whether the CommandLineTool.exe send out the request. Since both 2.(1) and 2.(2) worked well, it is very unlikely that 2.(3) is blocked;
   2. Use some network monitoring tool to check whether your machine received the recognition request from the gateway, when you use 2.(3) to send the recognition. It is possible that your firewall blocked this incoming requests from Prajna Gateway;
   3. Put a break point in the PredictionFunc(), to see whether your classifier wrapper (the sample code) really received the request, and which exe or function (which should be your classifier) it called to recognizer the image, here you may have put a wrong exe file path, or your classifier DLL may have some issue, or you forgot to build the IRC.SampleRecogCmdlineCSharp.exe, which the sample code will call by default to return dummy results.
   4. If the image is sent to your classifier.exe, check your classifier side to see whether it get executed correctly and return the result string like “tag1:0.95;tag2:0.32;tag3:0.05;tag4:0.04;tag5:0.01”

Q: When we try to run the sample code on linux, we encountered error message containing: “Mono: Could not load file or assembly ‘FSharp.Core, Version=4.3.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies.”

A: For Linux/Ubuntu users, please follow the instructions we provided in the readme file to install the up-to-date version of mono and fsharp.

Please note that the sample code has to be run with the newest version of fsharp (4.3.1) . The default mono and fsharp packages you get from Ubuntu may not be up-to-date. Please follow the readme file provided with the sample codes to install the newest mono and fsharp package.

If you have already install the old version, please do the following to fix the problem.

sudo apt-get autoremove mono-completesudo  apt-get autoremove fsharpsudo  apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo "deb http://download.mono-project.com/repo/debian (opens in new tab) wheezy main" | sudo tee /etc/apt/sources.list.d/mono-xamarin.list  sudo apt-get update esudo apt-get install mono-complete fsharp

Q: When we use CommandLineTest tool, we encountered error message when checking the the availability of our classifier:

Error: Connected to Prajna Hub Gateway vm-Hub.trafficManager.net, but get empty response, please check classifier 12345678-abcd-abcd-abcd-123456789abc and its response format.

A: Some teams confused serviceGuid with providerGuid, so they used the providerGuid to hit a classifier in step#2, which should be serviceGuid instead. ServiceGuid is used to identify your classifier, while providerGuid is used to identify your team. Please use the exact command lines we sent you in earlier email of update 3/3. No need to change anything.

Step#2: CommandLineTool.exe Check -pHub vm-Hub.trafficManager.net -serviceGuid "GUID_Of_Your_Classifier" Step#3: CommandLineTool.exe Recog -pHub vm-Hub.trafficManager.net -serviceGuid "GUID_Of_Your_Classifier" -providerGuid "GUID_Of_Your_Team" -infile c:\dog.jpg