About Me

I am a postdoctoral researcher in the Knowledge Technology Group, University of Hamburg. My research interest is in building robust multimodal language models that generalize to new tasks while retaining previously learned knowledge. My approach is to employ explainable AI and neuro-symbolic AI to elucidate and facilitate the construction of robust models. Previously, I worked on symbolic approaches to AI such as spatio-temporal reasoning and multiagent systems.

News

• [May 2024] I am organizing the LLM+XAI Reading Group at Knowledge Technology, University of Hamburg.
• [Apr 2024] Our paper From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks has been accepted in Neurosymbolic Artificial Intelligence Journal.
• [Apr 2024] Our paper Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic has been accepted to LREC-COLING 2024.
• [Mar 2024] I am co-organizing the 3nd International Workshop on Spatio-Temporal Reasoning and Learning.
• [Feb 2024] Program committee member, ECAI 2024.
• [Jan 2024] Our paper Causal State Distillation for Explainable Reinforcement Learning has been accepted to Causal Learning and Reasoning (CLeaR) 2024.

Research Experience

• Neurocognitive Models of Crossmodal Language Learning (2020–present)
  • University of Hamburg (PIs: Stefan Wermter and Cornelius Weber)
  • Topics: Multimodal Language Learning, XAI, Neuro-Symbolic AI
• Formal Lexically Informed Logics for Searching the Web (2018–2020)
  • Cardiff University (PI: Steven Schockaert)
  • Topic: Natural Language Processing
• Feodor Lynen Research Fellowship by the Humboldt Foundation (2016–2017)
  • University of Technology Sydney (Host: Sanjiang Li)
  • Topics: Spatio-Temporal Reasoning and Multiagent Systems
• Artificial Intelligence Meets Wireless Sensor Networks (2015)
  • The Australian National University (PI: Jochen Renz)
  • Topic: Spatio-Temporal Reasoning and Learning
• Reasoning about Paths, Shapes, and Configuration (2009–2014)
  • University of Bremen (PIs: Christian Freksa^† and Diedrich Wolter)
  • Topic: Spatio-Temporal Reasoning
• The International Research Training Group “Semantic Integration of Geospatial Information” (2010-2013)
  • University of Bremen
  • Topic: Spatio-Temporal Reasoning

Education

• Dr. rer. nat. in Computer Science, University of Bremen (2009–2013)
• Visiting PhD student, North Carolina State University (2012)
• Visiting PhD student, University at Buffalo (2011)
• Diplom in Mathematics, University of Bremen (2003–2009)
• DAAD exchange student, Seoul National University (2008)

Selected Publications

: code : arXiv : project page.
Full publication list available on DBLP, Google Scholar, and Semantic Scholar.

(Multimodal) Language Learning

• K. Ahrens, L. Bengtson, J. H. Lee , S. Wermter, Visually Grounded Continual Language Learning with Selective Specialization, EMNLP Findings 2023.
• J. H. Lee, M. Kerzel, K. Ahrens, C. Weber, S. Wermter, What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning, IJCAI 2022.
• B. Plüster, J. Ambsdorf, L. Braach, J. H. Lee, S. Wermter, Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations, arXiv preprint, 2022.
• J. H. Lee, Y. Yao, O. Özdemir, M. Li, C. Weber, Z. Liu, S. Wermter, Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics, 2022.
• O. Özdemir, M. Kerzel, C. Weber, J. H. Lee, and S. Wermter, Language Model-Based Paired Variational Autoencoders for Robotic Language Learning, IEEE Transactions on Cognitive and Developmental Systems, 2022.
• C. Volquardsen, J. H. Lee, C. Weber, and S. Wermter, More Diverse Training, Better Compositionality! Evidence from Multimodal Language Learning, ICANN 2022.
• M. Li, C. Weber, M. Kerzel, J. H. Lee, Z. Zeng, Z. Liu, S. Wermter, Robotic Occlusion Reasoning for Efficient Object Existence Prediction IROS 2021.
• A. Eisermann, J. H. Lee, C. Weber, S. Wermter, Generalization in Multimodal Language Learning from Simulation, IJCNN 2021.
• J. H. Lee, J. Camacho-Collados, L. Espinosa Anke, S. Schockaert, Capturing Word Order in Averaging Based Sentence Embeddings, ECAI 2020.

Explainable Artificial Intelligence

• J. H. Lee, S. Lanza, S. Wermter, From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks, to appear in Neurosymbolic Artificial Intelligence Journal, 2024.
• W. Lu, X. Zhao, T. Fryen, J. H. Lee, M. Li, S. Magg, S. Wermter, Causal State Distillation for Explainable Reinforcement Learning, accepted to CLeaR 2024.
• B. Plüster, J. Ambsdorf, L. Braach, J. H. Lee, S. Wermter, Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations, arXiv preprint, 2022.

Neuro-Symbolic Artificial Intelligence:

• J. H. Lee, S. Lanza, S. Wermter, From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks, to appear in Neurosymbolic Artificial Intelligence Journal, 2024.
• X. Zhao, M. Li, W. Lu, C. Weber, J. H. Lee, K. Chu, S. Wermter, Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic, accepted to LREC-COLING 2024.
• J. H. Lee, M. Sioutis, K. Ahrens, M. Alirezaie, M. Kerzel, S. Wermter, Neuro-Symbolic Spatio-Temporal Reasoning, in Compendium of Neurosymbolic Artificial Intelligence, IOS Press, 2023, pp. 410–429.
• J. H. Lee, Y. Yao, O. Özdemir, M. Li, C. Weber, Z. Liu, S. Wermter, Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics, 2022.

Compositional Generalization

• C. Volquardsen, J. H. Lee, C. Weber, and S. Wermter, More Diverse Training, Better Compositionality! Evidence from Multimodal Language Learning, ICANN 2022.
• A. Eisermann, J. H. Lee, C. Weber, S. Wermter, Generalization in Multimodal Language Learning from Simulation, IJCNN 2021.

Continual Learning

• K. Ahrens, H. H. Lehmann, J. H. Lee, S. Wermter, Read Between the Layers: Leveraging Intra-Layer Representations for Rehearsal-Free Continual Learning with Pre-Trained Models, arXiv preprint, 2023.
• K. Ahrens, L. Bengtson, J. H. Lee , and S. Wermter, Visually Grounded Continual Language Learning with Selective Specialization, EMNLP Findings 2023.

Other Machine Learning Topics

• M. Li, X. Zhao, J. H. Lee, C. Weber, S. Wermter, Internally Rewarded Reinforcement Learning, ICML 2023.
• S. Kong, J. Bai, J. H. Lee, D. Chen, A. Allyn, M. Stuart, M. Pinsky, K. Mills, C. Gomes, Deep Hurdle Networks for Zero-Inflated Multi-Target Regression: Application to Multiple Species Abundance Estimation, IJCAI 2020.

Spatio-Temporal Reasoning and Learning

• J. H. Lee, M. Sioutis, K. Ahrens, M. Alirezaie, M. Kerzel, S. Wermter, Neuro-Symbolic Spatio-Temporal Reasoning, in Compendium of Neurosymbolic Artificial Intelligence, IOS Press, 2023, pp. 410–429.
• J. H. Lee, M. Kerzel, K. Ahrens, C. Weber, S. Wermter, What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning, IJCAI 2022.
• J. H. Lee et al. Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics, 2022.
• S. Kong, J. H. Lee, S. Li, Multiagent Simple Temporal Problem: The Arc-Consistency Approach AAAI 2018.
• F. Dylla, J. H. Lee, T. Mossakowski, T. Schneider, A.V. Delden, J.V.D. Ven, D. Wolter, A Survey of Qualitative Spatial and Temporal Calculi: Algebraic and Computational Properties, ACM Computing Surveys, 2017.
• J. H. Lee, S. Li, Z. Long, M. Sioutis, On Redundancy in Simple Temporal Networks, ECAI 2016.
• D. Wolter, J. H. Lee, Connecting Qualitative Spatial and Temporal Representations by Propositional Closure, IJCAI 2016.
• X. Ge, J. H. Lee, J. Renz, P. Zhang, Trend-Based Prediction of Spatial Change, IJCAI 2016.
• S. Schockaert, J. H. Lee, Qualitative Reasoning About Directions in Semantic Spaces, IJCAI 2015.
• P. Zhang, J. H. Lee, J. Renz, From Raw Sensor Data to Detailed Spatial Knowledge, IJCAI 2015.
• J. H. Lee, The Complexity of Reasoning with Relative Directions, ECAI 2014.
• J. H. Lee, J. Renz, D. Wolter, StarVars: Effective Reasoning About Relative Directions, IJCAI 2013.
• D. Wolter, J. H. Lee, Qualitative reasoning with directional relations, Artificial Intelligence, 2010.

Multiagent Systems

• S. Kong, J. H. Lee, S. Li, Multiagent Simple Temporal Problem: The Arc-Consistency Approach AAAI 2018.
• S. Kong, J. H. Lee, S. Li, A new distributed algorithm for efficient generalized arc-consistency propagation, Autonomous Agent Multi-Agent Systems 32, 2018.
• S. Kong, J. H. Lee, S. Li, A Deterministic Distributed Algorithm for Reasoning with Connected Row-Convex Constraints, AAMAS 2017.

Recent Teaching Activities

• Organizer, LLM+XAI Reading Group, Knowledge Technology, University of Hamburg (2024)
• Lecture on “Transformers and Crossmodal Learning”, University of Hamburg (2023) [Slides]
• Supervisor, Neural Networks Seminar, University of Hamburg (2023)
• Organizer, WISDUM meeting, Knowledge Technology, University of Hamburg (since 2021)
• Supervisor, Bio-inspired Artificial Intelligence Seminar, University of Hamburg (2022)

Thesis Supervision

• Continual Learning for Language-Conditioned Robotic Manipulation, Lennart Bengtson, MSc (2023)
• Multivariate Normal Methods in Pre-trained Models for Continual Learning, Hans Hergen Lehmann, BSc (2023) improved version submitted to a journal
• Generalization of Transformer-Based Models on Visual Question Answering Tasks, Frederic Voigt, MSc (2023)
• Improving Compositional Generalization By Learning Concepts Individually, Ramtin Nouri, MSc (2023)
• Learning Concepts a Developmental Lifelong Learning Approach to Visual Question Answering, Ramin Farkhondeh, BSc (2022)
• Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Tasks, Jakob Ambsdorf, MSc (2022), now PhD student at University of Copenhagen
• Tackling The Binding Problem And Compositional Generalization In Multimodal Language Learning, Caspar Volquardsen, BSc (2021), ICANN 2022
• Learning Bidirectional Translation Between Robot Actions and Linguistic Descriptions, Markus Heidrich, BSc (2021)
• Using the Reformer for Efficient Summarization, Yannick Wehr, BSc (2020)
• Generalization in Multi-Modal Language Learning from Simulation, Aaron Eisermann, BSc (2020), IJCNN 2021
• Commonsense Validation and Explanation, Christian Rahe, BSc (2020)

Blog Posts

• Deep Learning Courses and Books for Beginners
• Common LaTeX Mistakes by Newbies

Contact

Address

Knowledge Technology Research Group
Vogt-Koelln-Str. 30
22527 Hamburg, Germany
Google Maps

Email

ed.eeleeheaj@eaj