A lightweight repo for filtering dialog data with entropy-based methods.
The program reads the dataset, runs clustering if needed, computes the entropy of individual utterances, and then removes high entropy utterances based on the threshold, and saves the filtered dataset to the output directory. See the paper or the poster for more details.
💾 Cluster and filter any dialog data that you provide, or use pre-downloaded datasets
🚀 Various parameters can be used to adjust the algorithm
👌 Choose between different entropy computation methods
🔀 Choose between different clustering and filtering types
🎥 Visualize clustering and filtering results
Run setup.py which installs required packages and steps you through downloading additional data:
python setup.py
You can download all trained models used in this paper from here. Each training contains two checkpoints, one for the validation loss minimum and another after 150 epochs. The data and the trainings folder structure match each other exactly.
The main file can be called from anywhere, but when specifying paths to directories you should give them from the root of the repository.
python code/main.py -h
For the complete documentation visit the wiki.
- identity: In this method there is basically no clustering, the entropy of utterances is calculated based on the conditional probability of utterance pairs.
- avg-embedding: This clustering type uses average word embedding sentence representations as in this paper.
- sent2vec: This clustering type should use sent2vec sentence embeddings, but currently uses any embeddings you provide to it.
- source: Filters utterance pairs in which the source utterance's entropy is above the threshold.
- target: Filters utterance pairs in which the target utterance's entropy is above the threshold.
- both: Filters utterance pairs in which either the source or target utterance's entropy is above the threshold.
In this jupyter notebook you can easily try out the identity filtering method implemented in less than 40 lines, and it filters DailyDialog in a couple of seconds (you only need to provide a sources and targets file). In the second part of the notebook there are some cool visualizations for entropy, frequency and sentence length.
Visualize clustering and filtering results by running the visualization jupyter notebook. The notebook is pretty self-explanatory, you just have to provide the directory containing the clustering files.
High Entropy Utterances and Clusters from DailyDialog
A high entropy cluster found by sent2vec.
Transformer Trained on DailyDialog
For an explanation of the metrics please check this repo or the paper.
More examples can be found in the appendix of the paper.
Transformer Trained on Cornell and Twitter
For an explanation of the metrics please check this repo or the paper.
Check the issues for some additions where help is appreciated. Any contributions are welcome ❤️
Please try to follow the code syntax style used in the repo (flake8, 2 spaces indent, 80 char lines, commenting a lot, etc.)
New clustering methods can be added, by subclassing the FilterProblem class, check Identity for a minimal example. Normally you only have to redefine the clustering function, which does the clustering of sentences.
Loading and saving data is taken care of, and you should use the Cluster and DataPoint objects. Use the data_point list to get the sentences for your clustering algorithm, and use the clusters list to save the results of your clustering. These can also be subclassed if you want to add extra data to your DataPoint and Cluster objects (like a vector).
Finally add your class to the dictionary in main, and to the command-line argument choices.
- Richard Csaky (If you need any help with running the code: ricsinaruto@hotmail.com)
- Patrik Purgai (clustering part)
This project is licensed under the MIT License - see the LICENSE file for details.
Please include a link to this repo if you use it in your work and consider citing the following paper:
@inproceedings{Csaky:2019,
title = "Improving Neural Conversational Models with Entropy-Based Data Filtering",
author = "Cs{\'a}ky, Rich{\'a}rd and Purgai, Patrik and Recski, G{\'a}bor",
booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
month = jul,
year = "2019",
address = "Florence, Italy",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/P19-1567",
pages = "5650--5669",
}