Learning Galois-Structured Representations of Natural Language

Michael Thomas

thommi2@furman.edu

Furman University

December 8, 2025

Motivating Examples

• Premise: “Two women are embracing while holding to go packages”
• Hypothesis: “Two women are holding packages”

Relationship? → Entailment

• Premise: Two women are embracing while holding to go packages
• Hypothesis: Two women are holding packages
• Think of these as boxes
• The premise is more specific and the hypothesis is more general, so the premise is smaller and contained within the hypothesis.
• t.f. Entailment is expressed as box containment.

• Premise: “Two children in blue jerseys… in a bathroom washing hands”
• Hypothesis: “Two kids at a ballgame wash their hands”

Relationship? → Contradiction

• Premise: Two children in blue jerseys… in a bathroom washing hands
• Hypothesis: Two kids at a ballgame wash their hands
• Location conflict: bathroom vs. ballgame
• These boxes should be disjoint or violate containment ordering

The Challenge

Central Problem: Representing sentence meaning for reasoning, composition, and interpretability

• Neural language models (e.g., BERT) produce powerful distributed representations
• But these embeddings lack explicit structure
• Struggle to capture semantic relationships like entailment

• Core problem: represent sentence meaning for reasoning and interpretation
• Neural models (BERT) produce powerful but opaque embeddings
• No explicit structure → hard to capture semantic relationships like entailment

Our Approach

Learn a Galois connection between natural language and structured geometric abstractions

• Sentences → Boxes
• Semantic entailment ↔︎ Geometric containment
• Interpretable representations

• Learn Galois connection: natural language ↔︎ geometric abstractions
• Sentences → box embeddings (axis-aligned hyperrectangles)
• semantic entailment = geometric containment
• Interpretable representations we can visualize and reason about

What is a Galois Connection?

A Galois connection is a correspondence between two partially ordered domains

Abstraction (\(\alpha\)) maps sentences to boxes; Concretization (\(\gamma\)) reconstructs language

• Concrete domain (\(C\)): Natural language sentences

  • Ordered by semantic specificity

• Abstract domain (\(A\)): Geometric boxes (hyperrectangles)

  • Ordered by containment (\(\subseteq\))

• Abstraction function (\(\alpha: C \to A\)):

  • Maps sentences to boxes

• Concretization function (\(\gamma: A \to C\)):

  • Reconstructs language from boxes

• \(\alpha\) and \(\gamma\) preserve the ordering relationship between domains

• More specific phrases → smaller boxes (contained within)

• More general phrases → larger boxes

• Example: “Tom goes to East High School” ⊆ “Tom goes to school”

Natural Language Inference (NLI)

Task: Classify relationship between premise and hypothesis

1. Entailment: Hypothesis logically follows from premise
   • Premise box ⊇ Hypothesis box
2. Contradiction: Hypothesis conflicts with premise
   • Boxes are disjoint or violate containment ordering
3. Neutral: No logical relationship
   • Partial or no overlap

• NLI: classify relationship between premise and hypothesis
• Three classes:
  • Entailment: hypothesis follows from premise → premise box ⊇ hypothesis box
  • Contradiction: conflict → disjoint boxes or violation
  • Neutral: no relationship → partial/no overlap

Dataset: SNLI Corpus

Stanford Natural Language Inference

• ~570,000 human-written English sentence pairs
• Three balanced classes: Entailment, Contradiction, Neutral
• Derived from image captions (Flickr 30k)

Premise	Relation	Hypothesis
two women are embracing while holding to go packages.	entailment	two woman are holding packages.
two women are embracing while holding to go packages.	contradiction	the sisters are hugging goodbye while holding to go packages after just eating lunch.
two women are embracing while holding to go packages.	neutral	the men are fighting outside a deli.

• SNLI corpus: ~570K human-written sentence pairs
• Balanced across three classes
• Derived from Flickr 30k image captions → naturalistic language
• Table examples: entailment (simpler restatement), contradiction (conflicting details), neutral (unrelated)

Model Architecture

Architecture Overview

1. Abstraction Network (\(\alpha\)): BERT encoder → Box embeddings
   • Maps sentences to axis-aligned hyperrectangles \([l, u]\)

2. Concretization Network (\(\gamma\)): Transformer decoder
   • Reconstructs natural language from boxes

3. Box Lattice Operations: Geometric reasoning
   • Volume, intersection, containment violations

4. Geometric Feature Classifier: 3-layer feedforward
   • 8 geometric features → Entailment/Contradiction/Neutral

Reference Figure 2 in paper (Model architecture diagram).

• Abstraction network: BERT encoder → box embeddings
• Concretization network: transformer decoder → natural language
• Box lattice operations: volume, intersection, containment violations
• Geometric feature classifier: 3 layers of feedforward

Box Intersection Techniques

Hard vs. Probabilistic (Gumbel) intersection techniques

Soft [1] and Gumbel-smoothed [2] embeddings provide smoother gradients for optimization

• Key design: how to compute box intersections
• Hard: exact set-theoretic operations
• Gumbel: probabilistic distributions, smoother gradients
• Our finding: no performance difference → use simpler hard intersections

Counterexample-Guided Abstraction Refinement

• Maintains buffer of misclassified examples
• Iteratively refines abstraction function
• Example: If model wrongly infers “Tom goes” ⊆ “Tom goes to jail”
  • CEGAR detects counterexample
  • Penalizes incorrect box containment
  • Refines encoder to correct the relationship

• CEGAR: technique from program verification adapted for neural learning
• Maintains buffer of misclassified sentence pairs
• Example: wrong prediction “Tom goes” ⊆ “Tom goes to jail”
  • Detect counterexample → add to buffer → penalize during training
• Goal: learn more precise semantic boundaries

Training Objective

Training aims to minimize the multi-component loss function:

\[ L = \lambda_{\text{recon}} L_{\text{recon}} + \lambda_{\text{Galois}} L_{\text{Galois}} + \lambda_{\text{task}} L_{\text{task}} + \lambda_{\text{CEGAR}} L_{\text{CEGAR}} + \lambda_{\text{reg}} L_{\text{reg}} \]

• \(L_{recon}\): Reconstruction quality
• \(L_{Galois}\): Geometric consistency
• \(L_{task}\): NLI classification
• \(L_{CEGAR}\): Refinement from counterexamples
• \(L_{reg}\): Prevent excessive box growth

• Loss function: minimized during training
• Five loss components:
  • \(L_{recon}\): reconstruction quality
  • \(L_{Galois}\): geometric consistency (minimize violations for entailment, maximize for contradiction)
  • \(L_{task}\): cross-entropy for 3-way NLI classification
  • \(L_{CEGAR}\): refinement from counterexample buffer
  • \(L_{reg}\): prevent unbounded box growth
• Weighted combination balances all objectives

Results

Overall Performance

• 80% validation accuracy on SNLI corpus
• F1 score: 80% across all classes
• Demonstrates viability of Galois connection framework
• Semantic entailment successfully learned as geometric containment

• 80% validation accuracy, F1 = 80%
• Not state-of-the-art but promising for novel architecture
• Key: validates hypothesis that entailment = geometric containment

Per-Class Performance

• Entailment learned first (straightforward geometric principle)
• Neutral and Contradiction improve steadily
• Challenge: Distinguishing subtle boundaries in shared geometric space

• Entailment learned first (clear geometric signature)
• Neutral/contradiction improve steadily
• Entailment shows slight degradation from peak
• Challenge: optimize all three classes in shared space
  • Hard to distinguish between contradiction and neutral

Reconstruction Quality

Strong correlation between phrase length and reconstruction loss (\(p < 0.001\))

• Shorter phrases → Effective compression and reconstruction
• Longer phrases → Information bottleneck in fixed-dimensional boxes

• Strong correlation: length vs. reconstruction loss (p < 0.001)
• Shorter phrases: effective compression
• Longer phrases: higher loss → information bottleneck
• HYP: Fixed-dimensional boxes lack capacity for complex sentences

CEGAR Evaluation

Unexpected Result: Buffer size had no significant impact on performance

• All configurations converged to ~80% accuracy
• Larger buffers introduced computational overhead
• Future work needed on alternative refinement strategies

• Tested three buffer sizes: 0, 10K, 100K
• No significant performance difference
• Larger buffers → computational overhead
• Suggests implementation issues or CEGAR doesn’t help in continuous learning
• Important area for future work

Example: Entailment

	True Value	Reconstructed/Predicted
Premise	two women are embracing while holding to go packages.	two women are embracing while holding to go packages.
Hypothesis	two woman are holding packages.	two woman are holding packages.
Label	entailment	entailment

Box representations successfully preserved semantic information

• Correct entailment classification
• Perfect reconstruction of both premise and hypothesis
• Shows box representations preserve semantic information

Example: Contradiction

	True Value	Reconstructed/Predicted
Premise	two women are embracing while holding to go packages.	two women are embracing while holding to go packages.
Hypothesis	the sisters are hugging goodbye while holding to go packages after just eating lunch.	the sisters are hugging after hugging while holding up to go lunch break.
Label	contradiction	contradiction

Specificity conflict identified, but longer hypothesis poorly reconstructed

• Correct contradiction detection (location conflict: bathroom vs. ballgame)
• Reconstruction quality differs:
  • Hypothesis: good
  • Premise: poor (jerseys → t-shirts, numbers lost, unrelated details added)
• Illustrates information bottleneck for longer phrases

Limitations

1. Phrase Length Dependency
   • Fixed-dimensional boxes → information bottleneck
   • Longer phrases lose compositional complexity
2. CEGAR Performance
   • No measurable improvement from counterexample buffer
   • Implementation vs. conceptual issues unclear
3. Representational Capacity
   • Questions about box lattices for complex semantics

1. Phrase length dependency → information bottleneck for complex sentences
2. CEGAR didn’t improve performance (implementation or conceptual issue?)
3. Open question: can box lattices scale to full NL complexity?

Key Contributions

1. First end-to-end differentiable framework learning Galois connection for text

2. Structured latent space as complete lattice of boxes

   • Interpretable, compositional abstractions

3. CEGAR mechanism for iterative semantic refinement

4. Integration of formal methods with deep learning

Four key contributions: 1. First end-to-end differentiable Galois connection for NL 2. Structured latent space as box lattice → interpretable abstractions 3. CEGAR mechanism adapted for neural learning 4. Demonstrated formal methods + deep learning integration

Future Directions

1. Multiple Datasets: MultiNLI, SICK for generalizability

2. Hyperparameter Optimization: Box dimensionality, loss weights, architecture choices

3. Baseline Comparisons: SBERT, fine-tuned LLMs

4. Interpretability Focus:

   • Replace neural classifier with decision trees
   • Visualization tools for high-dimensional box lattices
   • Alignment with human semantic intuitions

Future directions: 1. Multiple datasets (MultiNLI, SICK) for generalizability 2. Hyperparameter optimization (box dim, loss weights, architecture) 3. Baseline comparisons (SBERT, fine-tuned LLMs) 4. Interpretability focus: - Decision trees for human-readable rules - Visualization tools for box lattices - Study alignment with human semantic intuitions

Conclusions

• Abstract interpretation and representation learning can be combined for natural language understanding
• Semantic relationships can be captured as geometric containment in structured space
• Opens pathways for interpretable AI with formal guarantees
• Further opportunities for research in representational capacity and refinement mechanisms

Key takeaways: - Formal methods + deep learning is viable for NLU - Semantic relationships as geometric containment in structured space - Opens pathways for interpretable AI with formal guarantees - Limitations point to challenging research questions - Demonstrates value of mathematical foundations for meaning representation

Thank You

Questions?

Michael Thomas
Department of Computer Science
Furman University

References

[1]

X. Li, L. Vilnis, D. Zhang, M. Boratko, and A. McCallum, “Smoothing the geometry of probabilistic box embeddings,” in International conference on learning representations, 2019. Available: https://openreview.net/forum?id=H1xSNiRcF7

[2]

S. S. Dasgupta, M. Boratko, D. Zhang, L. Vilnis, X. L. Li, and A. McCallum, “Improving local identifiability in probabilistic box embeddings,” arXiv preprint arXiv:2010.04831, 2020.