...
AI王-クイズAI日本一決定戦-

人工知能クイズ王を作ろう!

View Detail Project Website
...
SHINRA2020-JP

ウィキペディア構造化プロジェクト

View Detail Project Website
...
SHINRA2020-ML

Categorizing Wikipedia entities to fine-grained ontology

View Detail Project Website

SHINRA2020-ML

SHINRA is a resource creation project started in the year 2017, aiming to structure the knowledge in Wikipedia. SHINRA2020-ML is the first shared-task of text categorization in SHINRA project, tackling the problem of classifying 30 language Wikipedia entities in fine-grained categories. The task is conducted as one of the NTCIR-15 tasks and the participants are required to register at NTCIR-15.

View more details:

Important: The evaluation on the leaderboard is based on a portion of the SHINRA2020-ML target data. Note: The code for the baseline model to generate an example submission can be found in following github repository: https://github.com/k141303/shinra2020ml_char_cnn.

SHINRA2020-ML

ID language (click to open) # submissions best micro F1
1 English (en) 7 0.766
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.46 0.46 0.46
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.755 0.772 0.76
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.747 0.807 0.766
15 Aug 2020 VLP (Data Science Lab, VNU)
0.568 0.568 0.568
01 Sep 2020 PriberamLabs (Priberam Labs)
0.733 0.752 0.739
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.742 0.752 0.745
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.506 0.506 0.506
2 Spanish (es) 7 0.777
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.479 0.479 0.479
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.748 0.767 0.754
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.764 0.803 0.777
15 Aug 2020 VLP (Data Science Lab, VNU)
0.56 0.56 0.56
01 Sep 2020 PriberamLabs (Priberam Labs)
0.752 0.751 0.751
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.733 0.731 0.732
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.567 0.567 0.567
3 French (fr) 7 0.766
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.382 0.382 0.382
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.735 0.761 0.743
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.723 0.859 0.766
15 Aug 2020 VLP (Data Science Lab, VNU)
0.621 0.621 0.621
01 Sep 2020 PriberamLabs (Priberam Labs)
0.735 0.735 0.735
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.676 0.762 0.704
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.482 0.482 0.482
4 German (de) 8 0.77
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.469 0.471 0.47
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.747 0.765 0.753
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.732 0.855 0.77
07 Oct 2020 Sosuke (Studio Ousia)
0.739 0.741 0.739
15 Aug 2020 VLP (Data Science Lab, VNU)
0.57 0.57 0.57
01 Sep 2020 PriberamLabs (Priberam Labs)
0.738 0.8 0.758
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.713 0.713 0.713
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.618 0.618 0.618
5 Chinese (zh) 6 0.767
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.074 0.074 0.074
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.748 0.777 0.758
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.749 0.807 0.767
01 Sep 2020 PriberamLabs (Priberam Labs)
0.755 0.753 0.754
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.725 0.725 0.725
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.597 0.597 0.597
6 Russian (ru) 7 0.774
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.469 0.469 0.469
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.73 0.744 0.734
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.756 0.81 0.774
15 Aug 2020 VLP (Data Science Lab, VNU)
0.434 0.434 0.434
01 Sep 2020 PriberamLabs (Priberam Labs)
0.745 0.745 0.745
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.698 0.698 0.698
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.557 0.557 0.557
7 Portuguese (pt) 7 0.754
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.399 0.399 0.399
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.714 0.744 0.724
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.738 0.787 0.754
15 Aug 2020 VLP (Data Science Lab, VNU)
0.555 0.554 0.554
01 Sep 2020 PriberamLabs (Priberam Labs)
0.711 0.709 0.71
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.695 0.695 0.695
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.526 0.526 0.526
8 Italian (it) 7 0.769
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.385 0.385 0.385
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.737 0.755 0.743
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.734 0.847 0.769
15 Aug 2020 VLP (Data Science Lab, VNU)
0.641 0.641 0.641
01 Sep 2020 PriberamLabs (Priberam Labs)
0.734 0.734 0.734
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.724 0.724 0.724
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.477 0.477 0.477
9 Arabic (ar) 7 0.757
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.303 0.303 0.303
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.712 0.745 0.723
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.74 0.793 0.757
01 Sep 2020 PriberamLabs (Priberam Labs)
0.702 0.702 0.702
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.707 0.707 0.707
27 Aug 2020 CMVS
BOW + BiLSTM + ATTENTION
0.606 0.606 0.606
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.5 0.5 0.5
10 Indonesian (id) 5 0.77
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.388 0.388 0.388
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.736 0.764 0.745
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.747 0.82 0.77
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.712 0.712 0.712
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.536 0.536 0.536
11 Turkish (tr) 7 0.785
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.438 0.438 0.438
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.729 0.759 0.739
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.766 0.825 0.785
15 Aug 2020 VLP (Data Science Lab, VNU)
0.522 0.522 0.522
01 Sep 2020 PriberamLabs (Priberam Labs)
0.725 0.747 0.732
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.695 0.695 0.695
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.582 0.582 0.582
12 Dutch (nl) 7 0.773
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.485 0.483 0.484
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.722 0.744 0.729
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.76 0.802 0.773
15 Aug 2020 VLP (Data Science Lab, VNU)
0.607 0.605 0.606
01 Sep 2020 PriberamLabs (Priberam Labs)
0.731 0.753 0.738
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.708 0.708 0.708
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.6 0.6 0.6
13 Polish (pl) 6 0.783
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.51 0.508 0.509
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.745 0.763 0.751
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.77 0.81 0.783
01 Sep 2020 PriberamLabs (Priberam Labs)
0.758 0.783 0.766
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.735 0.735 0.735
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.619 0.619 0.619
14 Persian (fa) 5 0.785
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.415 0.415 0.415
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.731 0.754 0.739
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.775 0.804 0.785
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.739 0.737 0.738
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.602 0.602 0.602
15 Swedish (sv) 6 0.774
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.502 0.5 0.501
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.731 0.741 0.734
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.755 0.813 0.774
15 Aug 2020 VLP (Data Science Lab, VNU)
0.606 0.604 0.605
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.697 0.697 0.697
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.589 0.589 0.589
16 Vietnamese (vi) 8 0.785
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.378 0.378 0.378
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.737 0.768 0.747
15 Jul 2020 nhanv.hust (HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY)
0.77 0.768 0.769
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.772 0.811 0.785
15 Aug 2020 VLP (Data Science Lab, VNU)
0.478 0.476 0.477
01 Sep 2020 PriberamLabs (Priberam Labs)
0.722 0.722 0.722
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.722 0.72 0.721
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.593 0.593 0.593
17 Korean (ko) 6 0.79
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.262 0.262 0.262
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.704 0.715 0.708
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.78 0.812 0.79
01 Sep 2020 PriberamLabs (Priberam Labs)
0.746 0.746 0.746
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.723 0.723 0.723
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.535 0.535 0.535
18 Hebrew (he) 5 0.75
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.429 0.429 0.429
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.714 0.735 0.721
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.733 0.788 0.75
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.686 0.686 0.686
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.527 0.527 0.527
19 Romanian (ro) 5 0.774
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.36 0.36 0.36
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.744 0.761 0.75
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.762 0.8 0.774
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.7 0.697 0.698
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.527 0.527 0.527
20 Norwegian (no) 7 0.789
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.528 0.528 0.528
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.717 0.734 0.722
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.77 0.828 0.789
15 Aug 2020 VLP (Data Science Lab, VNU)
0.597 0.597 0.597
01 Sep 2020 PriberamLabs (Priberam Labs)
0.717 0.717 0.717
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.708 0.708 0.708
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.498 0.498 0.498
21 Czech (cs) 6 0.755
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.326 0.326 0.326
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.715 0.738 0.722
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.733 0.803 0.755
01 Sep 2020 PriberamLabs (Priberam Labs)
0.693 0.692 0.692
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.682 0.68 0.681
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.536 0.536 0.536
22 Ukrainian (uk) 6 0.758
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.408 0.408 0.408
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.707 0.731 0.715
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.738 0.801 0.758
01 Sep 2020 PriberamLabs (Priberam Labs)
0.696 0.696 0.696
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.706 0.704 0.705
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.589 0.589 0.589
23 Hindi (hi) 6 0.756
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.359 0.359 0.359
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.713 0.736 0.721
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.739 0.791 0.756
01 Sep 2020 PriberamLabs (Priberam Labs)
0.605 0.605 0.605
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.66 0.658 0.659
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.441 0.441 0.441
24 Finnish (fi) 6 0.771
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.477 0.475 0.476
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.722 0.744 0.729
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.755 0.805 0.771
15 Aug 2020 VLP (Data Science Lab, VNU)
0.583 0.583 0.583
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.731 0.731 0.731
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.515 0.515 0.515
25 Hungarian (hu) 6 0.78
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.369 0.369 0.369
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.74 0.757 0.746
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.761 0.82 0.78
15 Aug 2020 VLP (Data Science Lab, VNU)
0.524 0.524 0.524
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.714 0.714 0.714
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.534 0.534 0.534
26 Danish (da) 7 0.78
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.439 0.439 0.439
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.732 0.751 0.738
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.765 0.81 0.78
15 Aug 2020 VLP (Data Science Lab, VNU)
0.639 0.639 0.639
01 Sep 2020 PriberamLabs (Priberam Labs)
0.722 0.722 0.722
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.732 0.732 0.732
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.488 0.488 0.488
27 Thai (th) 5 0.752
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.309 0.309 0.309
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.748 0.762 0.752
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.723 0.782 0.743
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.505 0.502 0.503
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.603 0.603 0.603
28 Catalan (ca) 5 0.755
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.439 0.437 0.437
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.715 0.727 0.719
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.738 0.79 0.755
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.717 0.716 0.716
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.424 0.424 0.424
29 Greek (el) 3 0.779
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.261 0.261 0.261
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.76 0.821 0.779
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.527 0.527 0.527
30 Bulgarian (bg) 5 0.805
#UserPrecisionRecallmicro F1
06 Jul 2020 SHINRA Comittee (RIKEN)
Example Submission
0.387 0.387 0.387
20 Sep 2020 Rehoboam (IIIT Hyderabad, India)
0.76 0.781 0.767
14 Sep 2020 FPTAI (FPTAI)
VietBT
0.783 0.85 0.805
24 Sep 2020 UOM-FJ (University of Melbourne / Fujitsu Labs.)
0.747 0.747 0.747
07 Oct 2020 HUKB (Hokkaido University, Japan)
0.562 0.562 0.562

Important Dates

  • July 6th, 2020 : Leaderboard is now open.
  • August 31th, 2020 (Timezone: Baker Island(USA), UTC-12) : Official Submission deadline.
  • Notice: We will keep the site running for a while after the official results submission deadline.

Rules

  • Please read this page carefully.
  • Leaderboard Specific Rule: You can submit 5 times per day. It is reset at midnight Japan time.

Specification of Submission

  • The submission file must be a single zip archive that contains the following json files.
      Submission.zip (The file name of the archive can be freely chosen.)
        + en.json
        + es.json
        + fr.json
        + de.json 
        + zh.json
        ....
  • Please refer to this page for the file format. However, you need to remove indentation and line breaks so that it is one line per page.
  • Example submission is here.

To submit, please apply for an account on the Signup page and login from the form at the top of the page.