Sequence models and tokenization
Structured labels are aligned to words, subwords, or characters.
Structured NLP and information extraction
Tokens, spans, relations, and graphs have different legal structures and matching rules. This route makes the output contract part of the method.
The original CRF paper (2001) is the structured-prediction prerequisite: score a complete label sequence given the complete observation sequence, normalize globally, compute marginals with forward–backward, and decode with Viterbi. It explains why a later BiLSTM–CRF has two separable parts—contextual emission features and a globally normalized label transition model—and why valid-looking BIO output still does not solve nested, discontinuous, or ontology-ambiguous entities.
8 maintained papers · literature reviewed through 2 October 2026
Tagging, parsing, span extraction, relation extraction, and event extraction require explicit output structures and matching rules. Tokenization, label ontology, nested spans, document context, and cross-sentence evidence define the task as much as the encoder.
Prerequisite contract
Task-specific evaluation
Token, span, relation, and graph F1 have different denominators.
How the field developed
2001
Globally normalized conditional sequence models
The original CRF paper conditions a globally normalized undirected model on the full observation sequence, coordinates label transitions, and avoids the local-normalization label bias of maximum-entropy Markov models.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data A conditional random field models an entire structured label sequence given an observed sequence and normalizes over complete candidate labelings. In the paper's linear-chain case, transition and observation feature functions share one global log-linear score. This combines discriminative use of arbitrary, overlapping input features with sequence-level consistency, avoiding the locally normalized state-transition bottleneck that causes label bias in MEMMs. — Work: Derive the linear-chain partition function and Viterbi decoding, then compare with locally normalized token transitions.
Exit evidence: Explain features, global normalization, forward–backward marginals, decoding, and label bias on one sequence.
2016
Shared syntactic annotation
Universal Dependencies establishes cross-lingual word, morphology, and dependency conventions.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 Universal Dependencies v1: A Multilingual Treebank Collection Universal Dependencies (UD) v1 is an open, dependency-based annotation framework designed to make morphology and syntax comparable across languages. The paper fixes a shared lexicalist analysis, universal inventories, and a common serialized representation while permitting language-specific extensions. Its accompanying v1.2 snapshot comprised 37 treebanks in 33 languages; this is a historical resource paper, not the specification of today’s larger UD release. Universal Dependencies Work: Inspect annotation decisions and quantify disagreement across languages and constructions.
Exit evidence: Explain tokenization, heads, relations, and enhanced dependencies for one sentence.
2016
Neural sequence labelling
BiLSTM-CRF combines contextual features with globally normalized label transitions for NER.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 Neural Architectures for Named Entity Recognition The LSTM-CRF NER model combines three evidence paths. A character-level bidirectional LSTM encodes spelling, a pretrained word lookup supplies distributional knowledge, and a sentence-level bidirectional LSTM represents each token with left and right context. A linear-chain CRF then scores whole tag sequences using both token emissions and adjacent-tag transitions, so decoding respects learned label dependencies rather than making independent choices. Code Work: Compare independent token decoding with CRF decoding under the same encoder.
Exit evidence: Separate boundary, type, and invalid-transition errors.
2017
Document-level coreference as span ranking
The end-to-end coreference model scores every candidate span and antecedent, replacing a hand-built mention detector while making pruning part of the computation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 End-to-end Neural Coreference Resolution The model turns coreference resolution into end-to-end span ranking. It enumerates candidate text spans, builds a context-sensitive representation for each span, estimates whether it is a mention, and scores every retained span against earlier antecedent candidates plus a dummy no-antecedent option. Clusters emerge from repeated antecedent links; the pipeline no longer requires a separate mention detector whose mistakes are irreversible. Official implementation Work: Trace span construction, antecedent pruning, marginal likelihood, and cluster decoding on one document.
Exit evidence: Separate mention detection, antecedent choice, and cluster-level errors.
2020
Span-aware pretraining
SpanBERT masks contiguous spans and predicts span content from boundary representations, benefiting span selection tasks.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 SpanBERT: Improving Pre-training by Representing and Predicting Spans SpanBERT changes BERT pre-training so the unit of corruption and recovery matches span-centered downstream tasks. It masks contiguous whole-word spans and adds a span-boundary objective (SBO): representations immediately outside a masked span, together with a relative-position embedding, must predict every token inside it. The architecture stays BERT-large; the contribution is a span-aware self-supervised objective and a carefully separated baseline analysis. Official code and models Work: Evaluate flat, nested, and cross-sentence spans with exact and partial matching.
Exit evidence: Document how subword predictions map back to canonical spans.
2021
Entity and relation extraction with typed spans
PURE separates entity and relation encoders while injecting predicted typed entity markers, showing that a carefully controlled pipeline can rival more entangled joint systems.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 A Frustratingly Easy Approach for Entity and Relation Extraction PURE shows that joint entity-and-relation extraction does not require a complicated joint architecture. One pretrained encoder enumerates and classifies entity spans. A second, independently parameterized encoder receives the original sentence plus typed boundary markers around a proposed subject and object, then classifies their relation. The stages communicate discrete entity predictions, while strong contextual encoders and carefully exposed entity types carry most of the useful interaction. Official implementation Work: Compare gold-entity, predicted-entity, and end-to-end relation scores without changing the split.
Exit evidence: Report entity-boundary, entity-type, relation-type, and pipeline-propagation errors separately.
2021
Constrained sequence-to-structure event extraction
Text2Event serializes event records for generation, then constrains decoding with a schema so structural validity becomes part of inference.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction Text2Event represents an event record as a labeled tree, linearizes that tree into a target string, and asks a pretrained encoder-decoder to generate it directly from text. Event types, triggers, argument roles, and argument spans share one sequence interface. A schema-derived trie constrains decoding so generated structural labels and copied spans remain legal, while curriculum learning progresses from simpler event substructures to complete records. Official implementation Work: Write the serialization grammar and compare unconstrained with schema-constrained decoding.
Exit evidence: Measure trigger, argument, role, structural-validity, and ontology-transfer errors separately.
2026
Joint segmentation and glossing
PolyGloss makes morphological boundaries and interlinear gloss labels a coupled structured prediction problem rather than two disconnected pipelines.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 08.01 Massively Multilingual Joint Segmentation and GlossingPeer-reviewed ACL 2026 long paper and Outstanding Paper award recipient. PolyGloss treats morphological segmentation and interlinear glossing as one structured generation problem instead of two isolated pipelines. A multilingual encoder-decoder emits aligned morpheme and gloss sequences in an interleaved representation, so every predicted boundary is trained together with its linguistic label. This joint contract lets segmentation evidence improve glossing and gives low-resource languages a shared model rather than one model per language. Author manuscript · Official code · Official models and data Work: Jointly evaluate morpheme segmentation and interlinear gloss prediction with language-held-out splits, gold boundary controls, separate boundary and gloss metrics, and adaptation curves.
Exit evidence: Report corpus and family coverage, annotation provenance, tokenization, unseen morphemes and gloss labels, error propagation, and low-resource variance by language.
Research branches
Syntax and dependency structure
Which tokenization and relation ontology defines a valid tree or graph?
Universal Dependencies v1: A Multilingual Treebank Collection
Sequence labels
Which label dependencies and boundary conventions are enforced?
Coreference clusters
How are mentions, antecedents, pruning, and document-level clusters evaluated?
End-to-end Neural Coreference Resolution · SpanBERT: Improving Pre-training by Representing and Predicting Spans
Entities and relations
Does relation evidence use gold entities, predicted entities, or jointly learned spans?
SpanBERT: Improving Pre-training by Representing and Predicting Spans · A Frustratingly Easy Approach for Entity and Relation Extraction
Events and arguments
Which schema defines valid triggers, roles, nesting, and generated records?
Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction
Joint morphological segmentation and gloss extraction
Does coupling boundaries and gloss labels help genuinely unseen languages, or mostly exploit annotation and family overlap?
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Dependency graph | Predicts heads and typed relations under structural constraints. | Annotation schemes differ across languages and genres. | 2016 |
| Token sequence | Assigns coordinated labels to an ordered sequence. | Subword and boundary conventions change reported F1. | 2016 |
| Span representation | Represents entities or arguments through boundaries and content. | Flat matching hides nested and discontinuous phenomena. | 2020 |
| Coreference cluster | Links spans to antecedents and evaluates document-level clusters. | Pruning and incompatible cluster metrics can hide mention failures. | 2017 |
| Relation / event structure | Predicts typed links or serializes a schema-constrained structured record. | Gold-entity access or constrained decoding can make systems incomparable. | 2021 · 2021 |
| Joint segmentation and glossing | PolyGloss treats morpheme boundaries and interlinear gloss labels as one coupled structured prediction and adaptation problem. | Aggregate scores can hide language-family overlap, inconsistent annotation, rare-label failure, and boundary errors inherited by gloss prediction. | 2026 |
Concept lineage
Universal Dependencies v1: A Multilingual Treebank CollectionNeural Architectures for Named Entity Recognition
Shared linguistic annotation clarifies the units that neural structured decoders must predict.
Neural Architectures for Named Entity RecognitionSpanBERT: Improving Pre-training by Representing and Predicting Spans
Output-structure bias is complemented by a pretraining objective designed around contiguous spans.
End-to-end Neural Coreference ResolutionSpanBERT: Improving Pre-training by Representing and Predicting Spans
Candidate-span ranking gains a pretraining objective explicitly designed to represent contiguous spans and their boundaries.
SpanBERT: Improving Pre-training by Representing and Predicting SpansA Frustratingly Easy Approach for Entity and Relation Extraction
Span-aware representations are placed inside an explicit entity-first relation pipeline with typed markers.
A Frustratingly Easy Approach for Entity and Relation ExtractionText2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction
Typed pair prediction broadens into generation of multi-argument event records under a structural schema.
Failure modes to audit
- Scoring subwords as if they were gold entities.
- Ignoring nested or discontinuous annotations.
- Leaking document templates across splits.
- Collapsing boundary and relation errors into one F1.
Questions for every paper
- What is the prediction unit?
- Which structures are legal?
- How is partial overlap scored?
- What document context is available?
Complete maintained paper route
Section titled “Complete maintained paper route”8 canonical readings · 2001–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 8 readings in maintained order
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira
Metadata checked
Universal Dependencies v1: A Multilingual Treebank Collection
Joakim Nivre et al.
Official artifacts: Universal Dependencies
Metadata checked
Neural Architectures for Named Entity Recognition
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer
Official artifacts: Code
Metadata checked
End-to-end Neural Coreference Resolution
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer
Official artifacts: Official implementation
Metadata checked
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Mandar Joshi et al.
Official artifacts: Official code and models
Metadata checked
A Frustratingly Easy Approach for Entity and Relation Extraction
Zexuan Zhong and Danqi Chen
Official artifacts: Official implementation
Metadata checked
Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen
Official artifacts: Official implementation
Metadata checked
Massively Multilingual Joint Segmentation and Glossing
Michael Ginn, Lindia Tjuatja, Enora Rice, Ali Marashian, Maria Valentini, Jasmine Xu, Graham Neubig, and Alexis Palmer
Peer-reviewed ACL 2026 long paper and Outstanding Paper award recipient.
Official artifacts: Author manuscript · Official code · Official models and data
Metadata checked