Large Language Models (LLMs) have shown remarkable capabilities, demonstrating their potential to transform the Natural Language Processing (NLP) field by achieving strong performance across the entire spectrum of tasks. In this context, there is growing interest in leveraging LLMs as automated crowdworkers to streamline the traditionally labor-intensive process of manual annotation and linguistic content generation. This paper specifically examines the feasibility of using LLMs to generate high-quality contextual information — an increasingly recognized element for enhancing generative systems — by producing linguistic contexts from given premise sentences. We evaluate four prominent LLMs — LLaMA 2, PaLM 2, Vicuna, and GPT-3.5 — by making them generate 180 contextual outputs, which are then compared to 60 contexts manually crafted by linguistic experts. To systematically assess the appropriateness of these generated contexts, we introduce CATS (Contextual Appropriateness in Texts for Spanish), a novel and adaptable evaluation metric for measuring the appropriateness of contextual information generated texts. CATS is rooted in established discourse theories and provides a robust framework for analyzing linguistic context quality. The implementation of CATS is made publicly available at https://github.com/gplsi/cats. The reliability of CATS is validated through a manual evaluation conducted by two independent human referees. The results indicate that the quality of contexts generated by LLMs is comparable to those produced by human specialists, as evidenced by both CATS scores (0.214 vs. 0.150) and human judgment. Our findings underscore the efficiency and cost-effectiveness of employing LLMs as alternatives to human crowdworkers for generating contextual information, offering promising implications for the scalability and advancement of generative systems.
Large language models have become the dominant paradigm in Natural Language Processing, yet their explainability remains fundamentally limited, making automatic error detection challenging. This study conducts a systematic comparison of explainability signatures between discrete graph-based and continuous token-based architectures for error pattern recognition in text classification. Using text-to-graph conversion, LLM-as-teacher distillation, and unified Shapley-based attribution, we identify four complementary architectural patterns that enable reliable discrimination between correct and incorrect predictions: AUC-based confidence trajectories, feature concentration profiles, margin-preservation patterns, and fidelity-asymmetry signatures.We demonstrate that hierarchical graph-based architectures produce substantially sharper discriminative patterns (99.7--100.0\% error detection accuracy) compared to token-based architectures (88.1--89.6\% accuracy) under fair comparison. This performance gap persists across datasets, graph topologies, and prediction correctness conditions. Integrated logistic regression analysis reveals that all architectures rely on necessity-based features, but hierarchical discrete structures produce significantly stronger necessity signals through binary node participation, while continuous token embeddings produce attenuated signals through distributed compensatory pathways.For high-stakes applications requiring transparent automated error detection, hierarchical graph-based explainability patterns offer clear advantages over token-based approaches. Our four-dimensional evaluation framework provides a reproducible methodology for architectural signature analysis in explainability research.
Researchers have relegated natural language processing tasks to Transformer-type models, particularly generative models, because these models exhibit high versatility when performing generation and classification tasks. As the size of these models increases, they achieve outstanding results. Given their widespread use, many explainability techniques are developed based on these models. However, this process becomes computationally expensive due to the large size of the models. Additionally, transformers interpret input information through tokens that fragment input words into sequences lacking inherent semantic meaning, complicating the explanation of the model from the very beginning. This study proposes a novel methodology to achieve explainability in natural language processing tasks by automatically converting sentences into graphs and maintaining semantics through nodes and relations that express fundamental linguistic concepts. It also allows the subsequent exploitation of this knowledge in subsequent tasks, making it possible to obtain trends and understand how the model associates the different elements inside the text with the explained task. The experiments delivered promising results in determining the most critical components within the text structure for a given classification.
Generative language models achieve the state of the art in many tasks within natural language processing (NLP).Although these models correctly capture syntactic information, they fail to interpret knowledge (semantics).Moreover, the lack of interpretability of these models promotes the use of other technologies as a replacement or complement to generative language models.This is the case with research focused on incorporating knowledge by resorting to knowledge bases mainly in the form of graphs.The generation of large knowledge graphs is carried out with unsupervised or semi-supervised techniques, which promotes the validation of this knowledge with the same type of techniques due to the size of the generated databases.In this review, we will explain the different techniques used to test and infer knowledge from graph structures with machine learning algorithms.The motivation of validating and inferring knowledge is to use correct knowledge in subsequent tasks with improved embeddings.
Ministerio de Ciencia, Innovacion y Universidades del Gobierno de Espana, programa PROMETEO de la Generalitat Valenciana, COST Action Distant Reading for European Literary History (CA16204 - Distant-Reading)
The analysis of discourse and the study of what characterizes it in terms of communicative objectives is essential to most tasks of Natural Language Processing. Consequently, research on textual genres as expressions of such objectives presents an opportunity to enhance both automatic techniques and resources. To conduct an investigation of this kind, it is necessary to have a good understanding of what defines and distinguishes each textual genre. This research presents a data-driven approach to discover and analyze patterns in several textual genres with the aim of identifying and quantifying the differences between them, considering how language is employed and meaning expressed in each particular case. To identify and analyze patterns within genres, a set of linguistic features is first defined, extracted and computed by using several Natural Language Processing tools. Specifically, the analysis is performed over a corpora of documents-containing news, tales and reviews-gathered from different sources to ensure an heterogeneous representation. Once the feature dataset has been generated, machine learning techniques are used to ascertain how and to what extent each of the features should be present in a document depending on its genre. The results show that the set of features defined is relevant for characterizing the different genres. Furthermore, the findings allow us to perform a qualitative analysis of such features, so that their usefulness and suitability is corroborated. The results of the research can benefit natural language discourse processing tasks, which are useful both for understanding and generating language.
The research presented comes within the framework of the national project: Platform for the management and dissemination of open content through the use of MOOCs (Massive Open Online Course) in 2018. This is part of the policy of the Agrarian University of Havana to promote the use of MOOCs in the races that are offered. The work constitutes an analysis of the different MOOC definitions with their distinctive characteristics, passing through the different types of MOOCs that are known up to the adoption of that technology from the pedagogical point of view. There is also an analysis of a set of platforms that allow the development of MOOCs, deciding to use Edx for its potential from the pedagogical point of view or TalentLMS for being in the Cloud. Finally, the relationship between MOOCs and Learning Objects is established based on the advantages they offer for students and teachers, ending with some examples of MOOCs used or developed in Cuba as part of the strategy of computerization of Cuban society.
Research partially supported by the Spanish Government (grants TIN2015-65100-R; TIN2015-65136-C02-2-R).
As empirically demonstrated by the Word Sense Disambiguation (WSD) tasks of the last SensEval/SemEval exercises, assigning the appropriate meaning to words in context has resisted all attempts to be successfully addressed. Many authors argue that one possible reason could be the use of inappropriate sets of word meanings. In particular, WordNet has been used as a de-facto standard repository of word meanings in most of these tasks. Thus, instead of using the word senses defined in WordNet, some approaches have derived semantic classes representing groups of word senses. However, the meanings represented by WordNet have been only used for WSD at a very fine-grained sense level or at a very coarse-grained semantic class level (also called SuperSenses). We suspect that an appropriate level of abstraction could be on between both levels. The contributions of this paper are manifold. First, we propose a simple method to automatically derive semantic classes at intermediate levels of abstraction covering all nominal and verbal WordNet meanings. Second, we empirically demonstrate that our automatically derived semantic classes outperform classical approaches based on word senses and more coarse-grained sense groupings. Third, we also demonstrate that our supervised WSD system benefits from using these new semantic classes as additional semantic features while reducing the amount of training examples. Finally, we also demonstrate the robustness of our supervised semantic class-based WSD system when tested on out of domain corpus.
El examen consta de 3 partes: Test de teoria, problemas de teoria y SQL. Hay dos versiones, una para informatica y otra para multimedia.