23. Dialogue Natural Language Inference (2019)
목차
- Abstract
- Introduction
- Dialogue Consistency and Natural Language Inference
- Dialogue generation
- Persona-Based Dialogue
- Consistency
- Dialogue NLI Dataset
- Triple Generation
- Triple Annotation
Abstract
-
frame “Consistency of dialogue agents” as NLI
-
create new NLI dataset, Dialogue NLI
( model trained on this dataset improved consistency )
1. Introduction
INConsistency example
-
Q1) what is your job \(\rightarrow\) A1) doctor
Q2) what do you do for a living \(\rightarrow\) A1) soccer player
Natural Language Inference (NLI)
- learning a mapping between sentence pair & entailment category
- learning “general sentence representations” & “evaluating NLP models”
Proposal : “Reduce the problem of consistency in dialogue to NLI”
- 1) create a dataset, “Dialogue NLI”
- 2) demonstrate that NLI can be used to improve the consistency of dialogue models
2. Dialogue Consistency and Natural Language Inference
1) Dialogue generation
( = Next Utterance Prediction )
-
\(u_{t+1}\) is predicted, given conversation prefix \(u_{\leq t}\)
-
sequence of utterances = dialogue between agents
( \(u_{1}^{A}, u_{2}^{B}, u_{3}^{A}, u_{4}^{B}, \ldots, u_{T}^{B}\) )
2) Persona-Based Dialogue
-
each agent is associated with a persona, \(P_A\) and \(P_B\)
- conditionally dependent on its persona
- typical approach for representing the persona : use a set of sentences!
- \(P = \{ p_1, ... , p_m \}\).
3) Consistency
Consistency error
- when agent produces an utterance that contradicts one of previous utterances
Persona consistency error
- when agent produces an utterance that contradicts a subset of its persona
4) Natural Language Inference
input pair \((s_1, s_2)\) \(\rightarrow\) into one of \(y \in\{\text { entailment, neutral, contradiction }\}\)
- \(s_1\) : premise
- \(s_2\) : hypothesis
5) Reducing Dialogue Consistency to NLI
Identifying utterance which contradict (1) previous utterances or an (2) agent’s persona can be reduced,
by assuming that contradictions are contained in a sentence pair.
That is …
- 1) dialogue contradiction for agent A : contained in utterance pair \((u_i^A, u_j^A)\)
- 2) persona contradiction : contained in a pair \((u_i^A, p_k^A)\)
3. Dialogue NLI Dataset
consists of E(entailment), N(neutral), C(contradiction)
sentences originate from two-agent persona-based dialogue dataset
-
dialogue between agents \(A\) and \(B\) consists of a sequence of utterances \(u_{1}^{A}, u_{2}^{B}, u_{3}^{A}, u_{4}^{B}, \ldots, u_{T}^{B}\)
-
each agent has a persona represented by a set of persona sentences
( \(\left\{p_{1}^{A}, \ldots, p_{m_{A}}^{A}\right\}\) and \(\left\{p_{1}^{B}, \ldots, p_{m_{B}}^{B}\right\}\) )
- Dialogue NLI dataset consists of \(\left(u_{i}, p_{j}\right)\) and \(\left(p_{i}, p_{j}\right)\) pairs
3-1) Triple Generation
First, Human labeled triple
-
to determine labels for dataset, require human annotation of utterances & persona sentences
-
\((e_1, r, e_2)\) with each persona & utterance
( ex. (i, have_pet, dog ) )
-
explained in 3-2)
Then,
- Persona sentences & utterances are grouped by their triple!
- explained in 3-1) ( right below )
[ Triple Generation ]
Persona sentences & utterances are grouped by their triple!
( = Pairs ( \((u,p)\) and \((p,p)\) ) are defined as E, N, C, based on their triple )
-
(a) Entailment : share same triple
-
(b) Neutral
-
miscellaneous utterance : \((u,p)\) which \(u\) is not associated with any triple
-
personal pairing : neither redundant nor contradictory
-
pair \((p,p')\) is selected from persona, if \(p\) and \(p'\) do not share triple
-
then, each sentence associated with the same triple as \(p\) is paired with each sentence assoicated with the same triple as \(p'\)
-
-
relation swaps
- \((r,r')\) , whose triples are assumed to represent independent facts
- ex) “have_vehicle”, “have_pet”
- \((\cdot, r, \cdot)\) and \((\cdot, r', \cdot)\)
-
-
(c) Contradiction
- contradicting relation pairs \((r, r')\)
- \((e_1, r, e_2)\) and \((e_1, r', e_2)\)
3-2) Triple Annotation
-
using Amazon Mechanical Turk task
-
<category><relation><category> rules