Project DetailsThis project evaluates the ability of Large Language Models (LLMs) to extract and structure halakhic disputes from complex rabbinic legal texts, focusing on excerpts from the Tur and Beit Yosef. These texts are characterized by dense language, implicit references, multi-level attribution chains, and a mixture of rabbinic Hebrew and Aramaic. As a result, the task requires more than general summarization: the model must identify the core dispute, distinguish between halakhic categories, and correctly assign each opinion or source chain to the appropriate category. As part of the project, we constructed a dedicated benchmark consisting of source excerpts and manually created Ground Truth annotations. Each Ground Truth entry follows a hierarchical structure that includes a dispute title, halakhic opinion categories, and attribution chains supported by textual evidence. Several LLMs were then prompted to generate structured outputs for the same excerpts, and their outputs were evaluated using an LLM-as-a-Judge framework based on a quantitative rubric measuring precision and recall for both categories and opinions. In addition to comparing model performance, the project examined the effect of text preparation on extraction quality. Three input configurations were evaluated: original full-text excerpts, smaller thematic portions, and pre-processed texts. The results indicate that moderate thematic segmentation improves model performance compared to the original full-text setting, while aggressive preprocessing may reduce performance by damaging contextual cues. The best result was achieved by Gemini 3 Pro under the smaller-portions configuration, with an average global score of 90.8. The project demonstrates a practical methodology for evaluating logical understanding in LLMs on complex halakhic texts and provides a foundation for future work in natural language processing, argument mining, and digital humanities.
