THEMATIC SESSION #12
Making AI Measurable: LLM Quality Metrics, Explainability, and Big Data Benchmarking
ORGANIZED BY
Matthias Volk
Matthias Volk
Otto-von-Guericke-University
Daniel Staegemann
Daniel Staegemann
Otto-von-Guericke-University
Matthias Pohl
Matthias Pohl
Deutsches Zentrum fĂĽr Luft- und Raumfahrt e.V.
THEMATIC SESSION DESCRIPTION
As artificial intelligence (AI) and large language models (LLM) become central to modern applications, the need to rigorously measure their performance, quality, reliability, and efficiency grows rapidly. This track explores emerging methods for AI and LLM benchmarking, metrics for model performance, and approaches to explainability in content generation. Furthermore, also implications of adjacent technological areas are targeted, including inter alia the impact on Big Data analytics and the engineering of related systems. By bridging research and industry perspectives, this session aims to define the standards and methodologies that will make AI systems measurable, comparable, and accountable. Hence, in this thematic session, we welcome a variety of research approaches related to the investigation of related topics.
TOPICS
The list of topics includes (but is not limited to) the following:
- LLM quality metrics and benchmarking;
- Explainability of AI/LLM content generation;
- Performance measurement in Big Data analytics;
- Utilization of big data analytics and AI for benchmarking;
- Benchmarking methodologies for large-scale distributed environments;
- Technological specifications for the engineering of related systems;
- Real-World use case benchmarks and industry-specific workloads;
- Cost and efficiency of AI/LLM utilization;
- Data quality and governance in the context of AI.
ABOUT THE ORGANIZERS
Matthias Volk was born in 1991. He received the master’s degree in business informatics from Otto von Guericke University Magdeburg (OVGU). After that, in 2016, he continued his work as a researcher at OVGU, where he became, in 2020, a Research Coordinator and the Operational Head of the Very Large Business Application Laboratory (VLBA Lab). In 2022, he obtained his Ph.D. in engineering and entered industry while continuing his scientific work as an associate researcher. During his time at the OVGU, he published over 60 papers in prestigious outlets, such as the Americas Conference on Information Systems (AMCIS), Pacific Asia Conference on Information Systems (PACIS), International Conference on Business Information Systems (BIS), Hawaii International Conference on System Sciences (HICSS), and IEEE Access. Besides being an author and a speaker at conferences and a mini-track and workshop chair, he regularly acts as a reviewer.
Daniel Staegemann was born in Berlin, in 1989. He received the master’s degree in computer science from the Technical University Berlin (TUB), in 2017. Since 2018, he has been a scientific researcher at Otto von Guericke University Magdeburg (OVGU), where he obtained the Ph.D. degree in engineering in 2024. He has published over 100 papers in prestigious outlets, such as the Americas Conference on Information Systems (AMCIS), the Pacific Asia Conference on Information Systems (PACIS), the International Conference on Business Information Systems (BIS), the Hawaii International Conference on System Sciences (HICSS), the Journal of Big Data, and IEEE Access. Besides being an author and a speaker at conferences, as well as a mini-track and workshop chair, he also regularly acts as a reviewer. His research interests include big data, applications of AI, and all other related topics.
Matthias Pohl is a Research Group Lead at the Institute of Data Science at the German Aerospace Center (DLR) in Jena, Germany. At DLR, he focuses on data management, large language models, and efficient data applications. Prior to joining DLR, he was a research associate at the Magdeburg Research and Competence Cluster (MRCC) at Otto von Guericke University Magdeburg (OvGU), where he worked in the Very Large Business Application Lab from 2016. During his time at the university, his research interests centered on data science, statistical modeling, information systems, and the efficient design of innovative IT solutions. His work has particularly focused on data science project management, business goal alignment in analytics projects, and the practical application of methodologies like CRISP-DM. He received a diploma degree in Mathematics from OvGU in 2014.
