Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering

Anonymous

Abstract:

Scope of Reproducibility
In this process, in order to evaluate the accuracy of the claims mentioned in the paper and also the reusability of the paper codes, the codes introduced in GitHub were used. For this purpose, it was carried out in accordance with the instructions mentioned in it. Due to severe hardware limitations, it was not possible to learn the model and re-implement the code using Google Colab.

Methodology
Contrary to what was mentioned in the article about executable hardware, Google Colab was used with the following specifications: GPU 13GB RAM and 80GB Disk were used.
The duration of the model evaluation process on the Google Colab with the mentioned features is approximately 18 minutes and 30 seconds, and the disk and GPU consumed in this process are 49GB and 4GB, respectively.

Results
In the evaluation performed using the proposed RoBERTa model of the paper, the criterion F1 67.73891723166825 was obtained, which is quite similar to the accuracy reported by the paper itself.

What was easy
The hardware requirements and the initial setup of the experiment were fully described in the paper in Section B (Hyperparameters), which was very helpful in re-executing the code. A description of all usable datasets was also provided in Section 4.1 (Datasets). The documentation published at the Git by the authors was almost comprehensive and practical, include installation requirements, hardware, data sets, how the model is taught and how the model is evaluated.

What was difficult
The authors used a 24GB GPU (RTX TITAN). Execution with such conditions is not possible due to the free features provided by Google Colab. Due to the mentioned limitation, we tried to change the batch size, which was set to 12 by default in the article, to 2; But we still had a lack of RAM from Colab. It should be noted that by reducing the batch value, we also changed the number of epochs, but there were still problems.

Communication with original authors
Due to the comprehensive documentation provided in the gate as well as in the text of the article, there was no need to interact with the authors. Of course, the gateway account and the authors' research gate account were available in ways to communicate with the authors, including email.

Paper Url: https://arxiv.org/abs/2106.11575

Paper Venue: Not in list

Venue Name: arXiv preprint arXiv:2106.11575, 2021 - arxiv.org