Distant Supervision and Noisy Label Learning for Low Resource Named Entity Recognition: A Study on Hausa and Yorùbá

• David I. Adelani, Michael A. Hedderich, Dawei Zhu, Esther van den Berg, Dietrich Klakow

• Workshops AfricaNLP (oral presentation) and Practical ML for Developing Countries (oral presentation) at ICLR 2020

Paper: Arxiv

The lack of labeled training data has limited the development of natural language processing tools, such as named entity recognition, for many languages spoken in developing countries. Techniques such as distant and weak supervision can be used to create labeled data in a (semi-) automatic way. Additionally, to alleviate some of the negative effects of the errors in automatic annotation, noise-handling methods can be integrated. Pretrained word embeddings are another key component of most neural named entity classifiers. With the advent of more complex contextual word embeddings, an interesting trade-off between model size and performance arises. While these techniques have been shown to work well in high-resource settings, we want to study how they perform in low-resource scenarios. In this work, we perform named entity recognition for Hausa and Yorùbá, two languages that are widely spoken in several developing countries. We evaluate different embedding approaches and show that distant supervision can be successfully leveraged in a realistic low-resource scenario where it can more than double a classifier's performance.

Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels

• Lukas Lange, Michael A. Hedderich, Dietrich Klakow

• EMNLP 2019

Paper: Arxiv
Code: GitHub

In low-resource settings, the performance of supervised labeling models can be improved with automatically annotated or distantly supervised data, which is cheap to create but often noisy. Previous works have shown that significant improvements can be reached by injecting information about the confusion between clean and noisy labels in this additional training data into the classifier training. However, for noise estimation, these approaches either do not take the input features (in our case word embeddings) into account, or they need to learn the noise modeling from scratch which can be difficult in a low-resource setting. We propose to cluster the training data using the input features and then compute different confusion matrices for each cluster. To the best of our knowledge, our approach is the first to leverage feature-dependent noise modeling with pre-initialized confusion matrices. We evaluate on low-resource named entity recognition settings in several languages, showing that our methods improve upon other confusion-matrix based methods by up to 9%.

Using Multi-Sense Vector Embeddings for Reverse Dictionaries

• Michael A. Hedderich, Andrew Yates, Dietrich Klakow and Gerard de Melo

• International Conference on Computational Semantics (IWCS 2019)

• May 2019, oral presentation

Popular word embedding methods such as word2vec and GloVe assign a single vector representation to each word, even if a word has multiple distinct meanings. Multi-sense embeddings instead provide different vectors for each sense of a word. However, they typically cannot serve as a drop-in replacement for conventional single-sense embeddings, because the correct sense vector needs to be selected for each word. In this work, we study the effect of multi-sense embeddings on the task of reverse dictionaries. We propose a technique to easily integrate them into an existing neural network architecture using an attention mechanism. Our experiments demonstrate that large improvements can be obtained when employing multi-sense embeddings both in the input sequence as well as for the target representation. An analysis of the sense distributions and of the learned attention is provided as well.

Paper: ACL Anthology
Presentation: Slides
Data: GitHub

@inproceedings{Hedderich19MultiSense,
title = "Using Multi-Sense Vector Embeddings for Reverse Dictionaries",
author = "Hedderich, Michael A. and Yates, Andrew and Klakow, Dietrich and de Melo, Gerard",
booktitle = "Proceedings of the 13th International Conference on Computational Semantics - Long Papers",
year = "2019",
address = "Gothenburg, Sweden",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/W19-0421",
pages = "247--258"
}

Handling Noisy Labels for Robustly Learning from Self-Training Data for Low-Resource Sequence Labeling

• Debjit Paul, Mittul Singh, Michael A. Hedderich and Dietrich Klakow

• NAACL Student Research Workshop (NAACL SRW) 2019

• June 2019, oral presentation

In this paper, we address the problem of effectively self-training neural networks in a low-resource setting. Self-training is frequently used to automatically increase the amount of training data. However, in a low-resource scenario, it is less effective due to unreliable annotations created using self-labeling of unlabeled data. We propose to combine self-training with noise handling on the self-labeled data. Directly estimating noise on the combined clean training set and self-labeled data can lead to corruption of the clean data and hence, performs worse. Thus, we propose the Clean and Noisy Label Neural Network which trains on clean and noisy self-labeled data simultaneously by explicitly modelling clean and noisy labels separately. In our experiments on Chunking and NER, this approach performs more robustly than the baselines. Complementary to this explicit approach, noise can also be handled implicitly with the help of an auxiliary learning task. To such a complementary approach, our method is more beneficial than other baseline methods and together provides the best performance overall.

Paper: https://arxiv.org/abs/1903.12008

@InProceedings{Hedderich19MultiSense,
author = "Paul, Debjit and Singh, Mittul and Hedderich, Michael A and Klakow, Dietrich",
title = "Handling Noisy Labels for Robustly Learning from Self-Training Data for Low-Resource Sequence Labeling",
booktitle = "To appear in the Proceedings of the NAACL Student Research Workshop (NAACL SRW 2019)",
year = "2019",
publisher = "Association for Computational Linguistics",
location = "Minneapolis"
}


Training a Neural Network in a Low-Resource Setting on Automatically Annotated Noisy Data

• Michael A. Hedderich and Dietrich Klakow

• ACL-Workshop Deep Learning Approaches for Low Resource Natural Language Processing (DeepLo)

• July 2018, oral presentation

Manually labeled corpora are expensive to create and often not available for low-resource languages or domains. Automatic labeling approaches are an alternative way to obtain labeled data in a quicker and cheaper way. However, these labels often contain more errors which can deteriorate a classifier's performance when trained on this data. We propose a noise layer that is added to a neural network architecture. This allows modeling the noise and train on a combination of clean and noisy data. We show that in a low-resource NER task we can improve performance by up to 35% by using additional, noisy data and handling the noise.

Paper: https://arxiv.org/abs/1807.00745 (latest version) or http://aclweb.org/anthology/W18-3402
Presentation: Slides and poster
Code: GitHub

@InProceedings{W18-3402,
author = "Hedderich, Michael A. and Klakow, Dietrich",
title = "Training a Neural Network in a Low-Resource Setting on Automatically Annotated Noisy Data",
booktitle = "Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP",
year = "2018",
publisher = "Association for Computational Linguistics",
pages = "12--18",
location = "Melbourne",
url = "http://aclweb.org/anthology/W18-3402"
}


TF-NNLM-TK

Tensorflow Neural Language Model Toolkit

While not being the original author, I'm the current mantainer of TF-NNLM-TK, a Tensorflow neural language model toolkit, at the Spoken Language Systems group. Being used both by business and research groups it functions as a stand-alone language model and as a language model component for more complex architectures.

For more information, please visit the Github page. If you have any questions, you can submit an issue there or contact me directly.

From Noise to Nice

Noisy Labeled Data and How to Learn with It

This guest lecture was given as part of the course “Machine Learning for Mobile and Pervasive Systems” at Stephan Sigg's Ambient Intelligence group at Aalto University. It discusses ways to obtain large amounts of data through crowd sourcing and automatic annotation techniques and how to deal with noise in this data using different noise modeling techniques like MACE or the noise layer approach.

Slides

KamiKatze

Local Multiplayer Game

• Little Factory
• Mai 2018

"Grab some friends, jump on the couch and get ready for this fast-paced and colorful arena game. In the world of these cats, there can only be one winner!"

This is the third game we develped at Little Factory. More infos at www.little-factory-games.com.

Semantic Embeddings for Paraphrase Detection and Reverse Dictionaries Using Memory Networks and External Knowledge from WordNet

• Master's Thesis
• August 2017

This thesis examines two tasks in the context of creating sentence embeddings with neural networks. We first look at reverse dictionaries which retrieve a target word from a large vocabulary based on the word’s description or definition. In the second task, we detect paraphrases, i.e. sentences that have the same meaning. In both cases, our specific network architectures create vector representations of the sentences using established techniques like LSTMs. We enhance these representations by adding a memory network component that allows integrating information from an external knowledge source. This information can be directly selected by the neural network. In our case, we incorporate additional linguistic knowledge from WordNet (like synonyms or hypernyms) about the words the network processes. Extensive experiments are performed and evaluated for both tasks. As part of the thesis, we give an introduction to basic and more advanced techniques from the areas of word representations and neural networks as well as a review of the related literature. We also reproduce existing work on reverse dictionaries and we create large paraphrase corpora based on existing video description data. Our analysis of word-sense ambiguity in the datasets can be used as a basis for future work.

If you are interested in the PDF of the thesis, please contact me.

Tiny Taxis

Android Game

• Little Factory
• February 2017

"Welcome to your new taxi business. Your job is to control the taxis in your city. To be successful you have to keep your passengers requests in mind. Businessmen are always in a hurry, be quick enough to catch them. Tourists like to see the city’s highlights and everyone likes a ride in a fancy car. But be aware to not get stuck in the rush hour.

Invest your money wisely: Hire new drivers, buy new cars and upgrade your fleet. If you are good enough, you might even be able to buy a limousine to transport your clients in style. Gain some extra money by completing challenges and try to beat the highscores."

You can download it for free at www.little-factory-games.com/TinyTaxis.

Mages

2D Multiplayer Arena Game

• Little Factory
• July 2016

Mages is a 2D multiplayer arena game in a fantasy setting that offers a combination of fast gameplay and tactical resource managment. It was developed and published by Little Factory. You can download the prototype at www.little-factory-games.com/Mages.

Seminar Deep Learning

• Seminar Talk
• January 2016

Seminar talk about

Venugopalan, Xu, Donahue, Rohrbach, Mooney, Saenko:
Translating Videos to Natural Language Using Deep Recurrent Neural Networks,
NAACL HLT (2015).

You can find the slides here.

All images and figures I created for the presentation are licensed under Creative Commons.