DH2026 - Reclaiming Epistemic Agency in AI-Assisted Research: A Recommender System That Reveals, Rather Than Hides, IR Logic

Alexia Schneider
alexia.schneider@umontreal.ca
Marcello Vitali-Rosati
marcello.vitali.rosati@umontreal.ca
Pierre Levy
pierre.levy@mac.com

2026-07-30

Presentation outline

  • Introduction
  • Brief SOTA
  • Demonstration
  • Evaluation
  • Conclusion

Introduction

Larger context of the research: Scientific documentary research (esp. for SSH researchers)

Recommendation systems have changed: from algorithms that extracted “you might also like” suggestions alongside search engine results to tools that support natural language queries and provide full reports.

Screenshot of Isidore and standard suggested article functions

Screenshot of Consensus.app. Support for natural language queries, combining all research steps (retrieval, ranking, synthesis)

Problem Statement

With the current adoption of these “AI research assistants” we are seeing a change in research practices. With it comes certain aspects that we have still to grapple with:

Research Question

How can we design explainable systems situated at the center of an environment that fosters critical exploration and enables intellectual engagement with the algorithm?

Examples of Alternative Recommendation Systems

  • Bridger by Portenoy et al. (2022) exposes researchers to authors outside their usual intellectual networks to promote disciplinary cross-pollination.
  • VITALITY by Narechania et al. (2022) proposes a literature review approach based on visualization of word embedding space from a seed article.
  • STAK by Martin, Greenspan, and Quan-Haase (2017) was inspired by the material affordances of physical libraries, attempting to recreate spatialized navigation environments.

IEML

Information Economy MetaLanguage (IEML) (Lévy 2023) != YAML

Sentence or higher level of conceptualisation

IEML decomposition matrix based on 9 semantic facets for the keyword editorialisation
Semantic Facet Value
Theme/root theory (s.y.-g.-o.p.-’)
What edition or publishing (t.a.-b.a.-p.-’)
Who -
To whom -
By which means *has property ~multiple process
When -
Where -
Why -
How -

Visualization of the grid in the IEML-RS plugin

Rules :

  • the grid doesn’t have to be filled
  • flexions are represented with *, ~
  • le concepts have to be in the IEML dictionnary: https://ieml.intlekt.io/

Lexical level of deeper level of conceptualisation

publishing: t.a.-b.a.-p.-’

Rules:

  • Any concept can be described from the combination of 6 ‘primitives’: E (emptiness), U (virtual), A (actual), S (sign), B (being), and T (thing).
  • A combination between two ‘primitives’ is a lowercase letter: there are 26 symbols.
  • There are up to 6 layers of semantic desambiguisation represented with punctuation (ex: the dot is the first layer).
  • Each layer is represented by a triad following the syntax: substance (role 1), attribute (role 2), and mode (role 3). An empty space in a layer is simply not represented.

Modulation of the primitive M

Modulation of the primitive M on the first segment shifts meaning from b associated with ‘language’ in literature to a t meaming ‘memory’ and creating publishing

Demonstration

Summary of Main Features

  • Exploration and extansion of a semantic field thanks to IEML starting from a keyword describing the seed article.
  • Automatic generation and correction (human-in-the-loop) of missing ‘translations’.
  • Construction of a query sent to the search engine Isidore.
  • Side by side comparison of article lists retrieved from the user-selected keywords and an LLM-augmented query from those same keywords.

Prompts

“Translation” Prompt into IEML

English version:

You are a semantics expert. You must semantically decompose the keyword “${keyword}” based on the following 9 values: ‘theme, who, what, to whom, by what means, when, where, why, how’ ## Examples ${examples}

## Dictionary Words You must use the words below to define the keyword “${keyword}”: ${context}

Your response must take the form of a CSV with 9 columns; the column headers are: ‘theme, who, what, to whom, by what means, when, where, why, how’ It is not necessary to fill all fields. A field can remain empty between two commas, as in the examples. Respond only with a final CSV line, without explanation.

Original in French:

Tu es un expert en sémantique. Tu dois décomposer sémantiquement le mot-clé "${keyword}" à partir des 9 valeurs suivantes :
'thème, qui, quoi, à qui, par quoi, quand, où, pourquoi, comment'
## Exemples
${examples}

## Mots du dictionnaire 
Tu dois utiliser les mots ci-dessous pour définir le mot-clé "${keyword}":
${context}

Ta réponse prendra la forme d'un CSV à 9 colonnes, les entêtes de colonnes sont: 
'thème, qui, quoi, à qui, par quoi, quand, où, pourquoi, comment'
Il n'est pas nécessaire de remplir tous les champs. Un champs peut rester vide entre deux virgules, comme dans les exemples. 
Répond uniquement avec une ligne CSV finale, sans explication.

Query Augmentation Prompt

English version:

Generate 10 variants of the following Boolean query “${keywords}”. Combine the proposed queries using the OR operator as in the example:

Keywords: “impact of climate change on biodiversity”.

Response: “(climate change biodiversity impact) OR (effects of climate change on ecosystems) OR (biodiversity loss due to climate change) OR (climate change species extinction) OR (impact of global warming on wildlife) OR (effects of climate change on ecosystems and species diversity) OR (how climate change impacts wildlife and biodiversity) OR (climate change consequences for biological diversity) OR (relationship between climate change and loss of biodiversity) OR (climate change threats to flora and fauna diversity) OR (impact of climate change on biodiversity) Now it is your turn with”${keywords}“.

Respond only with the query without giving any explanation.

Original in French::

Produit 10 variants de la requête booléenne suivante "${keywords}". Combine les requêtes proposées à l'aide de l'opérateur OU comme dans l'exemple : Mots-clés:  "impact of climate change on biodiversity". Réponse: "
(climate change biodiversity impact) OU (effects of climate change on ecosystems) OU (biodiversity loss due to climate change) OU (climate change species extinction) OU (impact of global warming on wildlife) OU (effects of climate change on ecosystems and species diversity) OU (how climate change impacts wildlife and biodiversity) OR (climate change consequences for biological diversity) OU (relationship between climate change and loss of biodiversity) OU (climate change threats to flora and fauna diversity) OU (impact of climate change on biodiversity)
C'est à ton tour avec "${keywords}". Répond uniquement avec la requête sans donner d'explication.

Evaluations

  • Quantitative evaluation of the translation produced by the LLM in IEML (RAG to anchor translations in IEML dictionary words).
  • Qualitative evaluation (user study) of the application as a whole.

Evaluation of Automatic Translation into IEML

word to translate theme or root who what to whom by what means when where why how
digital space digital technique - space - - - - - -
singer play or sing a melody person - - *by means of voice - - -

Expected translation

word translated theme or root who what to whom by what means when where why how
digital space space cyberspace everyone via internet and social networks - virtually monetize and share information
singer music singer performing a song to an audience by their voice - - to express an emotion using song metadata

Examples of translations produced by rag-llama-fewshot.

LLMs still do not perform well for structured meaning representation e.g. with Abstract Meaning Representation (AMR) (Ettinger et al. 2023).

Evaluation Summary

Models (via Together AI API):

  • Meta-Llama-3-70B-Instruct-Turbo
  • Gemma-3n-E4B-it
  • GPT-oss-20B
Table of model evaluation for automatic translation
model/strategy BLEU Cosine avg
baseline 0.056 0.515 0.286
llama_fewshot 0.0222 0.555 0.289
gemma_fewshot 0.020 0.491 0.255
openai_fewshot 0.0167 0.498 0.257
llama_zeroshot 0.0129 0.514 0.263
gemma_zeroshot 0.0160 0.479 0.248
openai_zeroshot 0.016 0.493 0.254

The context (i.e., dictionary entries) marginally improves performance.

Limits:

  • BLEU and cosine are meant for the evaluation of natural language translation therefore are ill-suited for structured semantic language parsing.
  • the retrieval part of the RAG makes choices that can potentially go against the principles of IEML representation. For instance the IR for ‘editorialisation’ will very likely retrieve from the IEML dictionnary ‘publishing’ with a vector-base search or ‘edition’ with BM25 but probably not ‘theory’ as these are farther apart. But the main advantage of IEML lies in its ability to bridge faraway concepts from common denominators.

Qualitative Evaluation of IEML-RS

6 users (PhD in Digital Humanities). Demonstration followed by 10 minutes of usage observation and a 15-minute interview.

Criteria:

  • User-friendliness,
  • Usefulness.

Features:

  • Navigation (keywords and concepts),
  • Translation into IEML,
  • Comparison of article list panels.

User Study

  • Ease of use:
    • Confusion regarding the integration of concepts into the query,
    • Automatic translation and validation: initial hurdle (limited database),
  • Usefulness:
    • Reveals strong disparities in user research practices (network and contextual IR vs. keyword search),
    • Strength: comparative panels,
    • Mixed-quality translation encourages user agency and dialogic and collaborative work with LLMs (Schneider 2026).

Conclusion and Perspectives

Example of an environment design fostering serendipity.

Tools being developed in Digital Humanities:

  • Barista from the Impresso project: query building assistant (Finn et al. 2026)
  • Ongoing developments at Isidore for RAG limited to specific use cases (Sacy et al. 2025)

Acknowledgements

Research funded by the SSHRC through the Revue3.0 partnership project, as well as a fellowship from the Circé Network and the Publica Coalition for mutualization and research on scientific journals.

Thank you for your attention!

Link to download the Firefox plugin:

Bibliographie

Archambault, Susan, and José Rincón. 2024. “An Evaluation of Cutting-Edge AI Research Tools Using the REACT Framework.” Computers in Libraries 44 (8).
DeRewal, Tiffany. 2025. “Evaluating LLM ‘Research Assistants’ and Their Risks for Novice Researchers.” Critical AI 3 (2). https://doi.org/10.1215/2834703X-12095982.
Dijkstra, Jaap J., Wim B. G. Liebrand, and Ellen Timminga. 1998. “Persuasiveness of Expert Systems.” Behaviour & Information Technology 17 (3): 155–63. https://doi.org/10.1080/014492998119526.
Ettinger, Allyson, Jena D. Hwang, Valentina Pyatkin, Chandra Bhagavatula, and Yejin Choi. 2023. “"You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation.” In EMNLP 2023 Findings. arXiv. https://doi.org/10.48550/ARXIV.2310.17793.
Finn, Finola, Donal Khosrowi, Caio Mello, Marten Düring, and Danielle Guido. 2026. “Shaping History, Responsibly: Seven Principles to Guide the Design of Archival AI Assistants for Cultural Heritage Collections.” In ACM FAccT. Montréal.
Lévy, Pierre. 2023. “Semantic Computing with IEML.” Collective Intelligence 2 (4): 26339137231207634. https://doi.org/10.1177/26339137231207634.
Logg, Jennifer M., Julia A. Minson, and Don A. Moore. 2019. “Algorithm Appreciation: People Prefer Algorithmic to Human Judgment.” Organizational Behavior and Human Decision Processes 151 (March): 90–103. https://doi.org/10.1016/j.obhdp.2018.12.005.
Martin, Kim, Brian Greenspan, and Anabel Quan-Haase. 2017. “STAK – Serendipitous Tool for Augmenting Knowledge: A Conceptual Tool for Bridging Digital and Physical Resources.” Digital Studies / Le Champ Numérique 6 (1). https://doi.org/10.16995/dscn.265.
Merton, R. K. 1968. “The Matthew Effect in Science. The Reward and Communication Systems of Science Are Considered.” Science (New York, N.Y.) 159 (3810): 56–63.
Narechania, Arpit, Alireza Karduni, Ryan Wesslen, and Emily Wall. 2022. “VITALITY: Promoting Serendipitous Discovery of Academic Literature with Transformers & Visual Analytics.” IEEE Transactions on Visualization and Computer Graphics 28 (1): 486–96. https://doi.org/10.1109/TVCG.2021.3114820.
Nielsen, Mathias Wullum, and Jens Peter Andersen. 2021. “Global Citation Inequality Is on the Rise.” Proceedings of the National Academy of Sciences 118 (7): e2012208118. https://doi.org/10.1073/pnas.2012208118.
Patterson, Brandon, Carlos Flores, Nick Hauschild, and Jared Branch. 2025. “Which AI Tools Work Best for Research? Using Librarian and Student Perspectives to Inform a Rating Rubric.” Journal of Electronic Resources in Medical Libraries 22 (3): 133–38. https://doi.org/10.1080/15424065.2025.2546052.
Portenoy, Jason, Marissa Radensky, Jevin D West, Eric Horvitz, Daniel S Weld, and Tom Hope. 2022. “Bursting Scientific Filter Bubbles: Boosting Innovation via Novel Author Discovery.” CHI Conference on Human Factors in Computing Systems, April, 1–13. https://doi.org/10.1145/3491102.3501905.
Sacy, Antoine Silvestre De, Adam Faci, Léa Maronet, and Stéphane Pouyllau. 2025. “Lire à Différentes Échelles. Une Analyse Multiscalaire de La Retrieval-Augmented Generation En Sciences Humaines Et Sociales.” Humanités Numériques 12. https://doi.org/10.4000/15icj.
Schneider, Alexia. 2026. “An Imperfect AI to Uphold Human Agency : Vindicating the ‘Yes but’ of This World.” Short {{Talk}}. Montréal.
Tay, Aaron. 2025. “The Reproducibility and Interpretability of Academic AI Search Engines Like Primo Research Assistant, Web of Science Research Assistant, Scopus AI and More.” Blog. Aaron Tay’s Musings about Librarianship.
Underwood, Ted. 2014. “Theorizing Research Practices We Forgot to Theorize Twenty Years Ago.” Representations 127 (1): 64–72. https://doi.org/10.1525/rep.2014.127.1.64.
Varga, Attila. 2022. “The Narrowing of Literature Use and the Restricted Mobility of Papers in the Sciences.” Proceedings of the National Academy of Sciences 119 (17): e2117488119. https://doi.org/10.1073/pnas.2117488119.
Vicente, Lucía, and Helena Matute. 2023. “Humans Inherit Artificial Intelligence Biases.” Scientific Reports 13 (1): 15737. https://doi.org/10.1038/s41598-023-42384-8.

Screenshots

First step: from seed article, the panel opens and retrieves the article keywords

User can select a keyword already translated in IEML (in blue), displaying the grid containing concepts defining the keyword

If the keyword hasn’t been translated a LLM translation is suggested (from the IEML dictionnary) and user can modify it

After validation of the translation, the translation is added to the IEML database

User can build a query from selected concepts and keywords

The query returns 2 lists of articles, one from the keywords selected and one from a query augmentation of those keywords