Hugging Face's logo
Papers
arxiv:2412.09587

OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages

Published on Dec 12, 2024
Upvote
3
Authors:
Chester Palen-Michel ,
,
,
,
Constantine Lignos

Abstract

OpenNER 1.0 is a standardized collection of multilingual NER datasets with consistent annotations and baseline models for performance comparison.

AI-generated summary

We present OpenNER 1.0, a standardized collection of openly available named entity recognition (NER) datasets. OpenNER contains 34 datasets spanning 51 languages, annotated in varying named entity ontologies. We correct annotation format issues, standardize the original datasets into a uniform representation, map entity type names to be more consistent across corpora, and provide the collection in a structure that enables research in multilingual and multi-ontology NER. We provide baseline models using three pretrained multilingual language models to compare the performance of recent models and facilitate future research in NER.

View arXiv page View PDF Add to collection

Community

stefan-it

Hi @lignos , I really like this effort of having standardized NER datasets across different languages :) Do you plan to release the final dataset in near future 🤔

lignos
 Paper author

Thanks for asking, and sorry about there not being a dataset right now. We had to preprint this a little ahead of time so it can be cited by another paper, so the paper is slightly ahead of the data. We're going to put up a new preprint next week along with the dataset release and a GitHub repo.

·
janeenmadena

Hello, will you still send this dataset? This dataset will be of great help to me.

lignos
 Paper author

Hi @stefan-it and @janeenmadena , I wanted to let you know we've updated the preprint with a new set of LLM results and a number of other improvements. We had originally posted the preprint earlier than would be ideal because we needed another submission under review to cite it. We are still making improvements to OpenNER based on reviewer feedback, so we are waiting until it's accepted for publication to share the data. We found that reviewers are pointing out a lot of important things we want to address before we do a full public release.

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2412.09587 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2412.09587 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2412.09587 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.