Title: Can Long-Context Language Models Subsume Retrieval, RAG, and More?

Speaker: Dr. Jinhyuk Lee @ Google DeepMind

Time : 10:30 ~ 11:30, May 14th, 2025

Location: Online

• In-person: X
• Online: https://hli.skku.edu/InvitedTalk250514 
  • Meeting ID: 811 6267 7029
  • Passcode: 663754

Language: English speech & English slides

Abstract: 

Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. Leveraging LCLMs' ability to natively ingest and process entire corpora of information offers numerous advantages. It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system. To assess this paradigm shift, we introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks. However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks. Notably, prompting strategies significantly influence performance, emphasizing the need for continued research as context lengths grow. Overall, LOFT provides a rigorous testing ground for LCLMs, showcasing their potential to supplant existing paradigms and tackle novel tasks as model capabilities scale.

Bio:

Jinhyuk Lee is a Staff Research Scientist at Google DeepMind, where his work contributes to state-of-the-art language and embedding models. Before his current role at DeepMind, he gained diverse experience as a Research Scientist at Google Research's Brain Team, a Visiting Postdoctoral Research Associate at Princeton University, and a Research Professor at Korea University, where he also earned his Ph.D.. Dr. Lee has driven significant research in generalizable text embeddings through key projects like Gemini Embedding and Gecko, and played a role in the development of Gemini, a family of highly capable multimodal models. His impactful earlier work includes the development of BioBERT, a pre-trained biomedical language representation model for biomedical text mining. Demonstrating his strong contributions to the field, Dr. Lee has a consistent track record of publishing his research in top-tier NLP conferences such as NAACL, NeurIPS, EMNLP, and ACL.