language: en
datasets:
- ValueNet
tags:
- regression
- classification
- stance detection
- DeBERTa
license: mit
Study Overview
In this study, we employ the Microsoft DeBERTa v3 model, which introduces an additional embedding for positional indexing, enhancing the model used in Qiu et al. (2022). To date, no model has empirically validated the impact of the positional index on regression or classification tasks. For training and fine-tuning, we developed a custom trainer to evaluate Mean-Squared Error (MSE) as outlined in Qiu et al. (2022). However, the implementation details of the sigmoid activation function with a threshold of ([-1, 1])—where (-1) indicates a personal stance not aligned with the value in question, (1) indicates alignment, and (0) denotes neutrality (irrelevance)—were not clearly specified.
To model this threshold effectively, we opted for the tanh activation function, which provides a more appropriate representation. Consequently, we implemented an MSE loss function with tanh activation, followed by rounding to the nearest integer for evaluation purposes.
Utilizing this approach, we demonstrated improvements in regression tasks for evaluating stances on each test scenario. While the overall MSE did not show significant improvement, we observed higher accuracy, recall, and precision for the regression tasks. It is important to note that the classification task specified in Qiu et al. (2022) solely determines the presence or absence of the value in question, without considering the specific stance presented in the text. Therefore, our regression task, which assesses the particular stance, should not be directly compared with the classification task from Qiu et al. (2022).
Acknowledgements
We would like to acknowledge the authors of the ValueNet dataset for their valuable contribution to this work.
@article{Qiu_Zhao_Li_Lu_Peng_Gao_Zhu_2022,
title={ValueNet: A New Dataset for Human Value Driven Dialogue System},
volume={36},
url={https://ojs.aaai.org/index.php/AAAI/article/view/21368},
DOI={10.1609/aaai.v36i10.21368},
abstractNote={Building a socially intelligent agent involves many challenges, one of which is to teach the agent to speak guided by its value like a human. However, value-driven chatbots are still understudied in the area of dialogue systems. Most existing datasets focus on commonsense reasoning or social norm modeling. In this work, we present a new large-scale human value dataset called ValueNet, which contains human attitudes on 21,374 text scenarios. The dataset is organized in ten dimensions that conform to the basic human value theory in intercultural research. We further develop a Transformer-based value regression model on ValueNet to learn the utility distribution. Comprehensive empirical results show that the learned value model could benefit a wide range of dialogue tasks. For example, by teaching a generative agent with reinforcement learning and the rewards from the value model, our method attains state-of-the-art performance on the personalized dialog generation dataset: Persona-Chat. With values as additional features, existing emotion recognition models enable capturing rich human emotions in the context, which further improves the empathetic response generation performance in the EmpatheticDialogues dataset. To the best of our knowledge, ValueNet is the first large-scale text dataset for human value modeling, and we are the first one trying to incorporate a value model into emotionally intelligent dialogue systems. The dataset is available at https://liang-qiu.github.io/ValueNet/.},
number={10},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
author={Qiu, Liang and Zhao, Yizhou and Li, Jinchao and Lu, Pan and Peng, Baolin and Gao, Jianfeng and Zhu, Song-Chun},
year={2022},
month={Jun.},
pages={11183-11191}
}