Read + Verify: Machine Reading Comprehension with Unanswerable Questions

squad2

Abstract

本文在于解决 squad 数据集中的unanswerable 问题。在本文中，我们提出了 a novel read-then-verify system， 该系统combines a base neural reader with a sentence-level answer verifier trained to further validate if the predicted answer is entailed by input snippets. 并且， we augment the
base reader with two auxiliary losses to better handle answer extraction and no-answer detection respectively, and investigate three different architectures for the answer verifier.

Introduction

如图1所示，我们的系统包含两个部分：

1. a no-answer reader for extracting candidate answers and detecting unanswerable questions
2. an answer verifier for validating if the candidate answer is actually supported by its surrounding sentence and the question.

Approach

本节中心详细介绍我们的 read-then-verify 系统。 我们的系统首先利用一个reader提取候选答案并且检测该问题是否为unanswerable。 然后利用一个answer verifier 来进一步的识别 答案片段与问题。

2.1 No-Answer Reader with Auxiliary Losses

我们的no-answer reader 参考下面两篇论文：
1. Zero-shot relation extraction via reading comprehension.
2. Simple and effective multi-paragraph reading comprehension.

We start from the standard no-answer reader that jointly learns answer extraction and no-answer detection， 然后引入两个auxiliary losses来分别优化两个任务， 两个任务之间互不干扰。

• No-Answer Option: 我们采用《Simple and effective multi-paragraph reading comprehension》中的方法， 在该方法中， 损失函数如下：

• $s_i$ 与 $g_i$ 表示the predicted scores of the answer start/end positions for token i
• a,b are the ground-truth start and end positions
• n 表示passage的长度
• z： the score for an additional no-answer possibility.
• $\delta$ ： 1 if the question is answerable and 0 otherwise.

• Independent Span Loss：该loss的目的是专注于 answer extration task, 在该task中，模型的作用是提取问题的候选答案。这也意味着我们将unanswerable cases 视作 answerable examples， 除了直接使用original span scores s 和 g， 我们提出使用 multi-head answer pointer 来产生score 对 ： $\hat{s}$ 与 $\hat{g}$。Span loss 如下：

• Independent No-Answer Loss:

2.2 Answer Verfiers

Answer Verfiers 是用来进一步 recognize the local, fine-grained entailment between the answer sentence and the question. 我们针对该任务实验了三种架构，如图所示：

1. asequentialmodelthatconsidersthe inputs as a long sequence
2. an interactive model that learns the interaction between the answer sentence and the question
3. a hybrid model that takes both of the two approaches into account.

2.2.1 Model 1： Sequential Architecture

2.2.2 Model-II: Interactive Architecture

2.2.3 Model-III: Hybrid Architecture

3. Experimental Setup

3.1 数据集

数据集采用 SQuAD 2.0, 它包含 SQuAD 1.1 的53775 个 answerable文件， 同时添加了很多 unanswerable 文件。

3.2 Training and Inference

3.3 Implenention

我们使用 Reinforced Mnemonic Reader ：《 Reinforced mnemonic reader for machine reading comprehension》来作为我们的base reader。