TextToHBM: A Generalised Approach to Learning Models of Human Behaviour for Activity Recognition from Textual Instructions is a project funded by the DFG. It started in September 2016 and continued for three years. Kristina Yordanova is the principal investigator in the project and within its context she investigateed methods and developed tools for the automatic learning of behaviour models from textual instructions.

Computational models for activity recognition aim at recognising the user actions and goals based on precondition-effect rules. One problem such approaches face, is how to obtain the model structure. To reduce the need of domain experts or sensor data during the model building, methods for learning models of human behaviour from textual data have been investigated. Existing approaches, however, make various simplifying assumptions during the learning process. This renders the model inapplicable for activity recognition problems. To address this problem, this project aims at developing a generalised methodology for learning the model structure from textual instructions. The methodology shall combine existing and novel methods for model learning.

  • A methodology for extracting the action semantics from text was developed. The methodology addresses the challenge of identifying causal relations between elements in texts with short and simple sentence structure.
  • To ensure the model generalisation and the incorporation of context information, methods for ontology learning were investigated. A core challenge here is the ontology extension based on causal, spatial, and functional properties of the entities in the problem domain.
  • To learn the model semantics, methods for language grounding were investigated. The semantics ere represented in terms of precondition-effect rules. The resulting methodology addresses the issues associated with learning these rules and translating them into an appropriate for activity recognition format. It also addresses the problem of learning an optimal model through reinforcement learning methods.
  • To evaluate the methodology, the learned models were applied to various activity recognition tasks and their performance compared to that of handcrafted models.

The methodology reduces the time and resources needed for developing computational models of human behaviour for activity recognition. Furthermore, it contributes to the better understanding of the general problem of structured behaviour representation and the learning of complex behaviour models.

Short Facts

  • Project title: TextToHBM: A Generalised Approach to Learning Models of Human Behaviour for Activity Recognition from Textual Instructions
  • Project homepage:https://text2hbm.org/
  • Runtime: September 2016–August 2019
  • Sponsor:German Research Foundation (DFG)
  • Budget: >300.000 Euro

Project-related publications

2020

  • Kristina Yordanova. Towards Automated Generation of Semantic Annotation for Activity Recognition Problems. In Proceedings of IEEE International Conference on Pervasive Computing and Communications (PerCom Workshops). Austin, Texas. 2020 [full text]
  • Debajyoti Paul Chowdhury, Arghya Biswas, Tomasz Sosnowski, Kristina Yordanova. Towards Evaluating Plan Generation Approaches with Instructional Texts. arXiv preprint. arXiv:2001.04186. January 2020. [full text]
  • Kristina Yordanova, Albert Hein, Thomas Kirste. Kitchen Task Assessment Dataset for Measuring Errors due to Cognitive Impairments. In Proceedings of IEEE International Conference on Pervasive Computing and Communications (PerCom Workshops). Austin, Texas. 2020 [full text]
  • Kristina Yordanova, Albert Hein, Thomas Kirste. Artifact Abstract: Kitchen Task Assessment Dataset for Measuring Errors due to Cognitive Impairments. In Proceedings of IEEE International Conference on Pervasive Computing and Communications (PerCom). Austin, Texas. March 2020. [full text]

2019

  • Kristina Y. Yordanova, Burcu Demiray, Matthias R. Mehl, Mike Martin. Automatic Detection of Everyday Social Behaviours and Environments from Verbatim Transcripts of Daily Conversations. In Proceedings of IEEE International Conference on Pervasive Computing and Communications. Kyoto, Japan. 2019. [full text]
  • Kristina Yordanova. Challenges Providing Ground Truth for Pervasive Healthcare Systems. In IEEE Pervasive Computing 18(2):100-104. 2019 [full text]
  • Kristina Yordanova, Stefan Lüdtke, Samuel Whitehouse, Frank Krüger, Adeline Paiement, Majid Mirmehdi, Ian Craddock, Thomas Kirste. Analysing Cooking Behaviour in Home Settings: Towards Health Monitoring. In Sensors, 2019 [full text]
  • David Schindler, Kristina Yordanova, Frank Krüger. An annotation scheme for references to research artefacts in scientific publications. Workshops Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom Workshops). Kyoto, Japan. 2019. [full text]

2018

  • Kristina Yordanova. Extracting Planning Operators from Instructional Texts for Behaviour Interpretation. In German Conference on Artificial Intelligence, Berlin, Germany, 2018. [full text]
  • Kristina Yordanova, Sebastian Bader, Sarah Weschke, Frank Krüger, Judith Henf, Stefan Teipel, Thomas Kirste. Discovery of Causal Relations in the Challenging Behaviour of People with Dementia. In IEEE International Conference on Pervasive Computing and Communications (PerCom WiP Session), Athens, Greece, 2018. [full text]
  • Kristina Yordanova and Frank Krüger. Creating and Exploring Semantic Annotation for Behaviour Analysis. In Sensors, 2018 [full text]
  • Kristina Yordanova, Frank Krüger, Thomas Kirste. Providing Semantic Annotation for the CMU Grand Challenge Dataset. Workshops Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom Workshops), Athens, Greece, 2018 [full text]

2017

  • Kristina Yordanova, Carlos Monserrat, David Nieves, José Hernández-Orallo. Knowledge Extraction from Task Narratives. In Proceedings of 4th International Workshop on Sensor-based Activity Recognition and Interaction, Rostock, Germany, 2017 [full text]
  • Kristina Yordanova. Automatic Generation of Situation Models for Plan Recognition Problems. In Proceedings of Recent Advances in Natural Language Processing, Varna, Bulgaria, 2017 [full text]
  • Kristina Yordanova. A Simple Model for Improving the Performance of the Stanford Parser for Action Detection in Textual Instructions. In Proceedings of Recent Advances in Natural Language Processing, Varna, Bulgaria, 2017 [full text]
  • Kristina Yordanova. TextToHBM: A Generalised Approach to Learning Models of Human Behaviour for Activity Recognition from Textual Instructions.  In AAAI Workshop Proceedings (PAIR 2017), 2017 [full text]

2016

  • Kristina Yordanova. From Textual Instructions to Sensor-based Recognition of User Behaviour.  In Companion Proceedings of the ACM International Conference on Intelligent User Interfaces (IUI 2016), Sonoma, CA, 2016 [full text]
  • Kristina Yordanova, Thomas Kirste. Learning Models of Human Behaviour from Textual Instructions.  In Proceedings of the 8th International Conference on Agents and Artificial Intelligence (ICAART 2016), Rome, Italy, 2016 [full text]

2015

  • Kristina Yordanova. Time Series from Textual Instructions for Causal Relations Discovery (Causal Relations Dataset). , University Library, University of Rostock, 2015 [full text]
  • Kristina Yordanova. Discovering Causal Relations in Textual Instructions.  In Proceedings of Recent Advances in Natural Language Processing, Hissar, Bulgaria, 2015 [full text]