Background: Adaptive educational systems have gained increasing attention due to their ability to personalise educational content based on individual learner progress. Prior research highlights that intelligent tutoring systems (ITSs) and adaptive courseware models improve learning outcomes by dynamically adjusting instructional materials. However, despite advancements in adaptive learning environments, usability remains a critical factor influencing their effectiveness and adoption. Therefore, a need exists to evaluate the usability of adaptive tutoring systems to ensure they provide optimal user experience whilst maintaining high instructional effectiveness. Objectives: This study examines the usability and learning experience of Tutomat, an adaptive courseware system designed for automated, real-time content adaptation. Specifically, it aims to examine usability based on user interactions and feedback, assess learning effectiveness and engagement through pre-test/post-test comparisons and user feedback, identify key usability challenges and propose improvements for enhancing user experience. Methods: A mixed-methods usability evaluation was conducted with 14 participants, applying both quantitative and qualitative research designs. The quantitative study used pre-test/post-test comparisons and System Usability Scale (SUS) scores to measure knowledge acquisition and usability perceptions. The qualitative study involved scenario-based usability testing, where participants completed eight structured tasks. Data analysis included Wilcoxon Signed-Ranks tests for learning gains, descriptive statistics for usability scores and analysis for qualitative feedback. Results: Findings indicate that Tutomat significantly improved student knowledge, as demonstrated by pre-test/post-test comparisons (W = 0, p < 0.05). The SUS score of 68.21 suggests acceptable usability, though strict answer validation and unclear navigation caused user frustration. Scenario-based testing identified usability challenges, particularly in feature discoverability and response flexibility, which impacted the user experience. Participants valued adaptive feedback and structured content but expressed concerns regarding interface design limitations. Conclusion: This study highlights the importance of usability in adaptive educational systems, demonstrating that real-time adaptive courseware can enhance learning engagement when designed with user-centred principles. The findings provide practical design recommendations for improving usability in ITSs, emphasising enhanced response flexibility, improved feedback mechanisms and clearer navigation features. Future research should explore longitudinal usability assessments to evaluate long-term user engagement and learning retention in adaptive environments.

Rich in information and annotated instances, a reference annotated dataset is essential for the training and evaluation of Natural Language Processing (NLP) tools. However, the creation of such linguistic resources is a tedious and time-consuming task involving lexical, syntactic, and semantic annotations, typically at the sentence level. Assuming we could speed up the human annotation process, we employed pre-trained models (spaCy, AllenNLP, EWISER) to automatically annotate a dataset of 664 sentences (6853 tokens, including 1598 predicates) taken from grammar books. A multi-layered annotation task encompassed Lemmatization (LEM), Part-of-Speech Tagging (UPOS, XPOS), Named Entity Recognition (NER), Dependency Parsing (DEP, HEAD), Coreference Resolution (COREF), Semantic Role Labelling (SRL), Predicate Sense Disambiguation (PSD) and Word Sense Disambiguation (WSD). Three annotators post-edited the noisy automatic annotations, and their average Inter-Annotator Agreement (IAA) for all annotation tasks at the token level was 0.91 and at the sentence level 0.74. Evaluation metrics including Accuracy, Precision, Recall, and F1 revealed disparities between machine and human annotations, along with correlations between machine annotations at both token and sentence levels. Manual error analysis identified instances where NLP tools failed to generate accurate annotations. A comparison of time spent per layer revealed that refining a pre-annotated subset of sentences required significantly less time than annotating them manually from scratch. This process resulted in an English reference dataset, tailored for the development of a hypergraph-based knowledge extraction model, known as the Natural Language 2 Semantic Hyper-graph Dataset (NL2SH) 1.0), which is accessible through CLARIN.

Part-of-speech (POS) tagging is one of the fundamental tasks in the field of natural language processing. This paper gives an overview of methods used for POS tagging and explores the research done so far, with a focus on POS tagging of the Croatian language. The latest techniques of POS tagging come from the field of deep learning. Commonly used deep learning models are various types of artificial neural networks, such as RNN, LSTM and BiLSTM models. In order to determine the efficiency of using different neural networks for POS tagging, RNN and LSTM models were implemented. The models were trained and evaluated on the hr500k dataset (reference corpus of Croatian texts). The results showed the overfitting problems of RNN models which is why the highest accuracy models achieved on the test dataset was 78.57

This paper describes and evaluates the performance of a semi-automatic authoring tool (SAAT) for knowledge extraction in the AC & NL Tutor, highlighting its strengths and weaknesses. We assessed the accuracy of automatic annotation tasks (Part-of-Speech tagging, Name Entity Recognition, Dependency parsing, and Coreference Resolution) performed on a dataset of 160 sentences from unstructured Wikipedia text on a computer. We compared the automatic annotations to the gold standard, created after human post-editing and validation. Human-error analysis included 3769 words, 582 subsentences, 1129 questions, 917 propositions, 1020 concepts, and 667 relations. It resulted in the error type classification and the set of custom rules further used for automatic error identification and correction. The results showed that an average of 68.7% of the error corrections referred to CoreNLP performance and 31.3% to the SAAT extraction algorithms. Our main contributions include an integrated approach to the comprehensive pre-processing of the text, knowledge extraction and visualization; the consolidated evaluation of natural language processing tasks and knowledge extraction output (sentences, subsentences, questions, concept maps) and the newly developed reference dataset.

There is a growing interest in automatic text processing and knowledge extraction from text repositories which often requires building new language resources and technologies. We present the KEMMRL model designed for the under-resourced but morphologically rich Croatian language. The proposed model uses natural language processing techniques, state-of-the-art deep learning algorithms and a rule-based approach to generate knowledge representations. The output of the newly developed HRtagger and HRparser methods in combination with the KEMMRL model is knowledge represented in the form of an ordered recursive hypergraph. Since the performance of KEMMRL is highly dependent on the applied deep learning methods, we evaluated them using hr500k reference corpus in the training and testing phase and manually designed out-of-domain Semantic Hypergraph Corpus (SemCro). The results of standard evaluation metrics showed that the HRtagger and HRparser achieved significantly better results than other state-of-the-art methods. These methods also showed the best results in measuring the structural similarity of hypergraphs, the highest average similarity to the manually annotated semantic hypergraphs and the number of semantic hyperedges correctly annotated by the model. The semantic hypergraph proved to be an ideal structure to capture and represent knowledge from more complex sentences without information loss. Researchers and developers of similar morphologically rich languages can customize and extend KEMMRL to their requirements. This article highlights the potential benefits of implementing the KEMMRL model into an Intelligent Tutoring System (ITS), and future research may focus on developing and testing such implementations.

Various approaches to text simplification have been proposed in an attempt to increase text readability. The rephrasing of syntactically and semantically complex structures is still challenging. A pedagogically motivated simplified version of the same text can have both positive and negative side effects. On the one hand, it can facilitate reading comprehension because of much shorter sentences and a limited vocabulary, but on the other hand, the simplified text often lacks coherence, unity and style. Therefore, reasonable trade-offs among linguistic simplicity, naturalness and informativeness are highly needed. This is a survey paper that discusses state-of-the-art approaches to sentence/text simplification and evaluation methods, along with an empirical evaluation of our approach. The quality of sentence splitting, using the knowledge extraction tool SAAT was compared to state-of-the-art syntactic simplification systems. The research was carried out on the WikiSplit, the HSplit and MinWikiSplit simplification corpora. Automatic metrics for the HSplit showed that the SAAT outperformed other TS systems in all categories. For the WikiSplit dataset, automatic metrics scores were slightly lower than that of the baseline system DisSim. However, the human evaluation showed that DisSim outperformed the SAAT in terms of simplicity and grammar. The quality of AG18copy output corresponded to that of the SAAT. The inter-annotator agreement was calculated. Research limitations as well as suggestions for future research were also provided.

In this paper, a novel approach to automatic question generation (AQG) using semantic role labeling (SRL) for morphologically rich languages is presented. A model for AQG is developed for our native speaking language, Croatian. Croatian language is a highly inflected language that belongs to Balto-Slavic family of languages. Globally this article can be divided into two stages. In the first stage we present a novel approach to SRL of texts written in Croatian language that uses Conditional Random Fields (CRF). SRL traditionally consists of predicate disambiguation, argument identification and argument classification. After these steps most approaches use beam search to find optimal sequence of arguments based on given predicate. We propose the architecture for predicate identification and argument classification in which finding the best sequence of arguments is handled by Viterbi decoding. We enrich SRL features with custom attributes that are custom made for this language. Our SRL system achieves F1 score of 78% in argument classification step on Croatian hr 500k corpus. In the second stage the proposed SRL model is used to develop AQG system for question generation from texts written in Croatian language. We proposed custom templates for AQG that were used to generate a total of 628 questions which were evaluated by experts scoring every question on a Likert scale. Expert evaluation of the system showed that our AQG achieved good results. The evaluation showed that 68% of the generated questions could be used for educational purposes. With these results the proposed AQG system could be used for possible implementation inside educational systems such as Intelligent Tutoring Systems.

In this article we present an knowledge extraction approach that can be used in systems that implement teaching in a fully automated manner. These systems are called Intelligent Tutoring Systems (ITS) and are conceived around the idea of one-to-one teaching. Many such systems use natural language processing to improve the communication interface between student and the system. These techniques can be also used on the content creator side to semi-automate or fully automate the task of teaching content creation. In such systems the knowledge representation plays a crucial role to successfully implement teaching and encourage learning. The output of the knowledge extraction phase is a knowledge in the form of a hyper graph that can be used for adaption to the students current knowledge level. We present a deep neural network architecture for precise POS tagging of words written in languages that are morphologically rich. Using sparse representations for words in this task increases the vector space and makes learning more complex. This problem can be solved to some extent by using traditional vector representations but there is also the problem with representing words that are ambiguous. Proposed architecture uses a Bidirectional Encoder Representations from Transformers (BERT) model that is pre-trained on Croatian language to achieve state-of-the-art accuracy for POS tagging.

The research investigates how note-taking practice affects the learning process in Tutomat, an intelligent tutoring system. The complete analysis includes (i) the identification of learning analytics variables to describe student-Tutomat interaction; (ii) the description of experimental student groups using learning analytics variables; (iii) data-driven clustering and (iv) the comparison of the experimental groups and revealed clusters. The results show that there is a difference in how a student interacts with Tutomat based on note-taking practice. It is revealed that the note-taking practice can be detected using the proposed learning analytics variables with the prediction accuracy of the clustering approach of 85 %.

E-Learning environment implies self-motivation and perseverance in study and completion of learning tasks. However, the more autonomy students have in managing their e-Learning, the harder they cope with distractions and remaining focused and engaged. This research study aims to assess the level of student engagement in four e-Learning platforms (CoLaB Tutor, AC-ware Tutor, CM Tutor and Moodle) in higher education. A model for Tracking Student Learning and Knowledge (TSLAK) is developed and based on two sets of variables: variables tracking student's learning activities (VTL) and variables tracking student's knowledge (VTK). This study aims to provide answers on how a model for tracking student online learning and knowledge can be formalized for the four e-Learning platforms and how can student learning and knowledge acquisition processes be described and measured by VTL and VTK. The results obtained by VTL and VTK indicate a significant decline in students' engagement. Out of 218 the most engaged students, 77 (35%) of them used the CoLaB Tutor, 41 (19%) used the AC-ware Tutor, 52 (24%) used the CM Tutor, and 48 (22%) used the Moodle. The research showed that out of the total number of students only 88 (13%) of them were the most engaged and the most successful or more precisely, 63 (71%) graduates and 25 (29%) undergraduates. Such student engagement and success measured by VTL and VTK indicate the necessity of increasing students' motivation in blended learning environments, strengthening their preparation and introduction to e-Learning platforms, and observing their feedback during a research study.

In intelligent e-learning systems that adapt a learning and teaching process to student knowledge, it is important to adapt the system as quickly as possible. However, adaptation is not possible until the student model is initialized. In this paper, a new approach to student model initialization using domain knowledge representative subset is described. The approach defines which concepts from domain knowledge should be included in the initial test so the system can make conclusions about what students truly know about domain knowledge. This representative subset of domain knowledge is defined using non-semantic mathematical approach based on graph theory. The initial test, created over a domain knowledge representative subset, guarantees encompassing all concepts that are relevant to domain knowledge. A two-level case study is conducted on what would be the representative subset of one selected domain knowledge. It compares semantically selected domain knowledge representative subsets (semantical analysis was done by domain area experts) to a non-semantical, mathematically selected domain knowledge representative subset. The results of the case study show that problems of inequality of semantically selected domain knowledge representative subsets are easily overcome using the presented approach.

The reasoning process about the level of student’s knowledge can be challenging even for experienced human tutors. The Bayesian networks are a formalism for reasoning under uncertainty, which has been successfully used for various artificial intelligence applications, including student modeling. While Bayesian networks are a highly flexible graphical and probabilistic modeling framework, its main challenges are related to the structural design and the definition of “a priori” and conditional probabilities. Since the AC&NL Tutor’s authoring tool automatically generates tutoring elements of different linguistic complexity, the generated sentences and questions fall into three difficulty levels. Based on these levels, the probability-based Bayesian student model is proposed for mastery-based learning in intelligent tutoring system. The Bayesian network structure is defined by generated questions related to the node representing knowledge in a sentence. Also, there are relations between inverse questions at the same difficulty level. After the structure is defined, the process of assigning “a priori” and conditional probabilities is automated using several heuristic expert-based rules.

Automatic knowledge acquisition is a rather complex and challenging task. This paper focuses on the description and evaluation of a semi-automatic authoring tool (SAAT) that has been developed as a part of the Adaptive Courseware based on Natural Language AC&NL Tutor project. The SAAT analyzes a natural language text and, as a result of the declarative knowledge extraction process, it generates domain knowledge that is presented in a form of natural language sentences, questions and domain knowledge graphs. Generated domain knowledge presents expert knowledge in the intelligent tutoring system Tutomat. The natural language processing techniques are applied and the tool’s functionalities are thoroughly explained. This tool is, to our knowledge, the only one that enables natural language question and sentence generation of different levels of complexity. Using an unstructured and unprocessed Wikipedia text in computer science, evaluation of domain knowledge extraction algorithm, i.e. the correctness of extraction outcomes and the effectiveness of extraction methods, was performed. The SAAT outputs were compared with the gold standard, manually developed by two experts. The results showed that 68.7% of detected errors referred to the performance of the integrated linguistic resources, such as CoreNLP, Senna, WordNet, whereas 31.3% of errors referred to the proposed extraction algorithms.

Over the last few decades, researchers put efforts to improve intelligent tutoring systems' abilities with the aim to get them as close as possible to the ultimate goal of one-to-one tutoring. CoLaB Tutor and AC-ware Tutor are intelligent tutoring systems based on conceptual knowledge learning and are notable due to the fact they are relatively easy to generalize to multiple knowledge domains. CoLaB Tutor's forte lies in teacher-learner communication in controlled natural language, while AC-ware Tutor focuses on the automatic and dynamic generation of adaptive courseware. In order to compare various intelligent tutoring system supported education environments, in this chapter, the authors summarize several empirical evaluations of CoLaB Tutor and AC-ware Tutor. The results of intelligent tutoring systems' effectiveness in these environments offer the possibility to observe the specific intelligent tutoring system across various education levels, as well as to compare the intelligent tutoring systems' supported education environments.

In this research we propose a comprehensive set of knowledge indicators aimed to enhance learners' self-reflection and awareness in the learning and testing process. Since examined intelligent tutoring systems do not include additional messaging features, the introduction of common set of knowledge indicators differentiates our approach from the previous studies. In order to investigate the relation between proposed knowledge indicators and learner performance, the correlation and regression analysis were performed for 3 different courses and each examined intelligent tutoring system. The results of correlation and regression analysis, as well as learners' feedback, guided us in discussion about the introduction of knowledge indicators in dashboard-like visualizations of integrated intelligent tutoring system.

Purpose– In order to initialize a student model in intelligent tutoring systems, some form of initial knowledge test should be given to a student. Since the authors cannot include all domain knowledge in that initial test, a domain knowledge subset should be selected. The paper aims to discuss this issue.Design/methodology/approach– In order to generate a knowledge sample that represents truly a certain domain knowledge, the authors can use sampling algorithms. In this paper, the authors present five sampling algorithms (Random Walk, Metropolis-Hastings Random Walk, Forest Fire, Snowball and Represent algorithm) and investigate which structural properties of the domain knowledge sample are preserved after sampling process is conducted.Findings– The samples that the authors got using these algorithms are compared and the authors have compared their cumulative node degree distributions, clustering coefficients and the length of the shortest paths in a sampled graph in order to find the best one.Originality/value– This approach is original as the authors could not find any similar work that uses graph sampling methods for student modeling.

We present results of empirical evaluation of intelligent tutoring systems (ITS) with ontological domain knowledge representation. This research was done as a first step in the process of developing a new model of intelligent tutoring system that will include all the characteristics of evaluated systems: adaptive content, communication based on controlled natural language, graphical presentation of ontological domain knowledge representation. The case study results revealed extraordinary effectiveness of evaluated adaptive intelligent tutoring systems when compared with traditional learning and teaching process.

This paper gives a detailed description of how adaptivity is realized in e-learning systems, with a focus on adaptivity of courseware. The review aims to answer three basic questions on adaptive courseware: what has been adapted, how and why. Researchers have tried to adapt according to these different elements: student knowledge, behaviour, interests, preferences, learning style, cognitive style, goals, courseware presentation, navigation, annotation, sequencing and knowledge testing. An historical background of adaptivity is described: from word definition, through adaptive instructions and adaptive systems, all the way to adaptive courseware. The biggest challenge was to find the most prominent representatives of adaptive systems in courseware generation that were selected after an exhaustive search through relevant scientific paper databases. Each prevailing system is briefly described and the student features, to which the system adapts, are highlighted, together with the level of adaptation. This paper aims to provide important information to researchers, educators and software developers of computer-based educational software ranging from e- learning systems in general to intelligent tutoring systems in particular. Such an analysis has not been done so far, especially in a way that adaptive systems are described uniformly. Finally, a comparative analysis of those systems and conclusions and demands about the ultimate adaptive system is given. This paper can be used as a guide for making decisions about what to adapt, how and why, while designing an adaptive e-learning system.

This paper describes an Adaptive Courseware Tutor – an intelligent tutoring system based on stereotypes, Bayesian networks and Bloom's knowledge taxonomy. The main feature of our approach is the automatization of learning object generation and courseware adaptivity in every stage of learning and teaching process. The student module is enhanced by double stereotypes based on student's knowledge level and on Bloom's knowledge taxonomy, as well as, by Bayesian networks. The tutor module is responsible for the automatic generation of courseware elements, their dynamic selection and sorting, as well as their adaptive presentation using templates for statements and questions. In order to evaluate the model’s effectiveness, a controlled experiment with a large sample was conducted. Keywords-component; Intelligent tutoring systems, adaptive e-learning systems, adaptive courseware, stereotypes, Bloom's knowledge taxonomy