...
AI王-クイズAI日本一決定戦-

人工知能クイズ王を作ろう!

View Detail Project Website
...
SHINRA2020-JP

ウィキペディア構造化プロジェクト

View Detail Project Website
...
SHINRA2020-ML

Categorizing Wikipedia entities to fine-grained ontology

View Detail Project Website

SHINRA2020-ML

SHINRA is a resource creation project started in the year 2017, aiming to structure the knowledge in Wikipedia. SHINRA2020-ML is the first shared-task of text categorization in SHINRA project, tackling the problem of classifying 30 language Wikipedia entities in fine-grained categories. The task is conducted as one of the NTCIR-15 tasks and the participants are required to register at NTCIR-15.

View more details:

Important: The evaluation on the leaderboard is based on a portion of the SHINRA2020-ML target data. Note: The code for the baseline model to generate an example submission can be found in following github repository: https://github.com/k141303/shinra2020ml_char_cnn.

SHINRA2020-ML

ID language (click to open) # submissions best micro F1
1 English (en) 6 0.765
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.46 0.46 0.46
11 Aug 2020 Rehoboam (IIIT Hyderabad, India)
0.722 0.722 0.722
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.752 0.792 0.765
12 Aug 2020 VLP (Data Science Lab, VNU)
0.568 0.568 0.568
12 Aug 2020 PriberamLabs (Priberam Labs)
0.739 0.761 0.746
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.722 0.722 0.722
2 Spanish (es) 4 0.777
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.479 0.479 0.479
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.764 0.803 0.777
12 Aug 2020 VLP (Data Science Lab, VNU)
0.56 0.56 0.56
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.733 0.731 0.732
3 French (fr) 4 0.766
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.382 0.382 0.382
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.723 0.859 0.766
12 Aug 2020 VLP (Data Science Lab, VNU)
0.621 0.621 0.621
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.683 0.683 0.683
4 German (de) 5 0.764
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.469 0.471 0.47
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.723 0.858 0.764
13 Aug 2020 Sosuke (Studio Ousia)
0.529 0.527 0.528
12 Aug 2020 VLP (Data Science Lab, VNU)
0.57 0.57 0.57
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.713 0.713 0.713
5 Chinese (zh) 4 0.763
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.074 0.074 0.074
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.738 0.814 0.763
12 Aug 2020 PriberamLabs (Priberam Labs)
0.743 0.757 0.748
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.725 0.725 0.725
6 Russian (ru) 4 0.768
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.469 0.469 0.469
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.726 0.863 0.768
12 Aug 2020 VLP (Data Science Lab, VNU)
0.434 0.434 0.434
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.698 0.698 0.698
7 Portuguese (pt) 5 0.747
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.399 0.399 0.399
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.736 0.768 0.747
12 Aug 2020 VLP (Data Science Lab, VNU)
0.555 0.554 0.554
12 Aug 2020 PriberamLabs (Priberam Labs)
0.711 0.745 0.722
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.695 0.695 0.695
8 Italian (it) 4 0.765
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.385 0.385 0.385
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.725 0.857 0.765
12 Aug 2020 VLP (Data Science Lab, VNU)
0.641 0.641 0.641
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.724 0.724 0.724
9 Arabic (ar) 3 0.736
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.303 0.303 0.303
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.71 0.793 0.736
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.707 0.707 0.707
10 Indonesian (id) 3 0.739
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.388 0.388 0.388
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.706 0.808 0.739
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.712 0.712 0.712
11 Turkish (tr) 4 0.732
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.438 0.438 0.438
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.725 0.747 0.732
12 Aug 2020 VLP (Data Science Lab, VNU)
0.522 0.522 0.522
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.695 0.695 0.695
12 Dutch (nl) 4 0.773
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.485 0.483 0.484
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.76 0.802 0.773
12 Aug 2020 VLP (Data Science Lab, VNU)
0.607 0.605 0.606
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.708 0.708 0.708
13 Polish (pl) 3 0.783
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.51 0.508 0.509
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.77 0.81 0.783
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.735 0.735 0.735
14 Persian (fa) 3 0.785
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.415 0.415 0.415
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.775 0.804 0.785
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.739 0.737 0.738
15 Swedish (sv) 4 0.772
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.502 0.5 0.501
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.733 0.857 0.772
12 Aug 2020 VLP (Data Science Lab, VNU)
0.606 0.604 0.605
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.697 0.697 0.697
16 Vietnamese (vi) 5 0.78
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.378 0.378 0.378
15 Jul 2020 nhanv.hust (HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY)
0.77 0.768 0.769
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.775 0.792 0.78
12 Aug 2020 VLP (Data Science Lab, VNU)
0.478 0.476 0.477
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.722 0.72 0.721
17 Korean (ko) 3 0.79
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.262 0.262 0.262
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.78 0.812 0.79
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.723 0.723 0.723
18 Hebrew (he) 3 0.748
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.429 0.429 0.429
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.734 0.776 0.748
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.686 0.686 0.686
19 Romanian (ro) 3 0.774
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.36 0.36 0.36
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.762 0.8 0.774
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.7 0.697 0.698
20 Norwegian (no) 3 0.788
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.528 0.528 0.528
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.747 0.88 0.788
12 Aug 2020 VLP (Data Science Lab, VNU)
0.597 0.597 0.597
21 Czech (cs) 3 0.751
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.326 0.326 0.326
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.706 0.852 0.751
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.682 0.68 0.681
22 Ukrainian (uk) 3 0.728
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.408 0.408 0.408
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.682 0.828 0.728
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.706 0.704 0.705
23 Hindi (hi) 3 0.753
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.359 0.359 0.359
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.737 0.787 0.753
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.66 0.658 0.659
24 Finnish (fi) 4 0.766
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.477 0.475 0.476
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.724 0.858 0.766
12 Aug 2020 VLP (Data Science Lab, VNU)
0.583 0.583 0.583
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.731 0.731 0.731
25 Hungarian (hu) 4 0.769
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.369 0.369 0.369
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.742 0.825 0.769
12 Aug 2020 VLP (Data Science Lab, VNU)
0.524 0.524 0.524
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.714 0.714 0.714
26 Danish (da) 4 0.78
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.439 0.439 0.439
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.765 0.81 0.78
12 Aug 2020 VLP (Data Science Lab, VNU)
0.639 0.639 0.639
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.732 0.732 0.732
27 Thai (th) 3 0.737
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.309 0.309 0.309
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.725 0.762 0.737
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.505 0.502 0.503
28 Catalan (ca) 3 0.754
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.439 0.437 0.437
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.742 0.779 0.754
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.717 0.716 0.716
29 Greek (el) 2 0.772
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.261 0.261 0.261
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.762 0.792 0.772
30 Bulgarian (bg) 3 0.783
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.387 0.387 0.387
13 Aug 2020 FPTAI (FPTAI)
VietBT
0.757 0.836 0.783
12 Aug 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
im4-vlbert
0.747 0.747 0.747

Important Dates

  • July 6th, 2020 : Leaderboard is now open.
  • August 31th, 2020: Submission deadline.

Rules

  • Please read this page carefully.
  • Leaderboard Specific Rule: You can submit 5 times per day. It is reset at midnight Japan time.

Specification of Submission

  • The submission file must be a single zip archive that contains the following json files.
      Submission.zip (The file name of the archive can be freely chosen.)
        + en.json
        + es.json
        + fr.json
        + de.json 
        + zh.json
        ....
  • Please refer to this page for the file format. However, you need to remove indentation and line breaks so that it is one line per page.
  • Example submission is here.

To submit, please apply for an account on the Signup page and login from the form at the top of the page.