Hierarchical Reinforcement Learning promises, among other benefits, to efficiently capture and utilize the temporal structure of a decision-making problem and to enhance transfer learning capabilities, but theoretical guarantees lag behind practice. In this paper, we propose a Feudal Q-learning scheme and investigate under which conditions its coupled updates converge and are stable. By leveraging the theory of Stochastic Approximation and the ODE method, we present a theorem stating the convergence and stability properties of Feudal Q-learning. This provides a principled convergence and stability analysis tailored to Feudal RL. Moreover, we show that the updates converge to a point that can be interpreted as an equilibrium of a suitably defined game, opening the door to game-theoretic approaches to Hierarchical RL. Lastly, experiments based on the Feudal Q-learning algorithm support the outcomes anticipated by theory. Code and data available at: https://github.com/col-tasas/2025-convergence-and-stability-HRL.
This paper proposes a generalized active classification framework designed to classify the environment of a robot through controlled tactile interaction, with direct application to robotic disassembly of Waste Electrical and Electronic Equipment (WEEE). In this domain, purely vision-based systems frequently fail due to occlusions, dust, and reflective surfaces, making haptic identification an operational necessity. By formulating environment identification as an active sensing problem, we utilize the entropy of learned interaction models and a receding-horizon optimization to select control inputs that maximize information gain from force-torque feedback. Our approach transitions from a passive classifier to an active framework that dynamically probes the environment to resolve belief uncertainty across six distinct tactile classes, demonstrated through the high-precision detection and unfastening of screws using an industrial 6-DOF robotic arm. Experimental results on a consumer microwave demonstrate that the active framework achieves a 51% reduction in detection latency for screws compared to passive baselines and consistently outperforms expert-designed heuristics. Furthermore, the system exhibits robust zero-shot generalization to novel industrial objects, indicating that learned tactile signatures are invariant to the global object context. These results provide a scalable foundation for autonomous disassembly in high-variance environments like electronic waste recycling, while offering a broader methodology for any robotic classification task where control can be leveraged to accelerate information gain.
A sparse-Lagrangian particle implementation of the multiple mapping conditioning (MMC) model coupled to large eddy simulation (LES) for two-phase flows with reacting solid fuel particles is proposed. The MMC-LES model is developed and validated by means of carrier-phase direct numerical simulations (CP-DNS) of the devolatilization and volatile combustion from pulverized coal particles in statistically homogeneous isotropic turbulence (HIT). Two distinct Lagrangian particle clouds are introduced: The first cloud represents the inertial coal particles under going heat-up and pyrolysis, while the second cloud consists of stochastic MMC particles representing the reacting gas mixture of coal volatiles burning in air. To account for the heat and mass transfer between the two clouds, the one-to-one model for two-phase coupling in MMC is employed. To enhance model accuracy and practicality, a dynamic parameterization approach is proposed by deriving the governing MMC conditioning parameters, which are usually assumed to be constant, adaptively from the transient field of the volatile mixture fraction. Results show that conventional MMC predictions following best practice are in good agreement with the DNS for low fuel particle loadings, if suitable a-priori information (ideally from DNS) is available to calibrate the model. For high fuel particle loadings stronger deviations are observed. Moreover, we find that in conventional MMC the nominal conditioning parameters in physical and mixture fraction space r(m) and f(m) cannot be realized as physical and mixture fraction distances d(x) and d(f) for the most part of the transient ignition and volatile combustion process as expected. Differently, the new dynamic approach does not need a-priori calibration information from DNS, allows for a faithful match of the nominal and realized MMC conditioning parameters over time, and provides favorable predictions of the ignition and volatile combustion process. The sensitivity of the MMC results to the dynamic scaling factor alpha(f) and constant C-xi in the mixing time scale model is explored. A scaling factor alpha(f) = 0.03 is recommended, and while the standard value C-xi = 0.1 already yields reasonable predictions, improved results can be obtained for slightly smaller C-xi.
The storage and transport of renewable energy represent major challenges in the transition to a sustainable energy economy. Commonly proposed solutions involve chemical storage of hydrogen in molecules like ammonia, methanol, or liquid organic hydrogen carriers (LOHCs). In such cases, the efficient release of hydrogen from the carrier becomes a critical step. However, reforming ammonia into hydrogen and nitrogen under elevated pressures remains an underexplored topic. Catalytic performance and kinetic data at high pressures and temperatures are absent from the literature, primarily due to the technical challenges in obtaining reliable results and ensuring safe reactor operation. To overcome this gap, a double-walled reactor concept has been developed and constructed for high-pressure ammonia reforming, capable of operating at up to 50 bar and 750 degrees C. The reactor features axially resolved temperature monitoring and preventive measures have been implemented to avoid unintended catalytic activity on reactor walls, thermocouple surfaces, and dilution materials, ensuring reliable performance data. Reactor safety is maintained by preventing ammonia nitriding of sensitive reactor components. Prior to catalytic measurements, the technical features of the reactor have been systematically validated. A Ni/Al2O3 catalyst was tested at maximum operating conditions, achieving near-full NH3 conversion for GHSV values between 10 000 h-1 and 30 000 h-1. The highest recorded H2 productivity was 0.52 mmolH2/gcat/s. The mass transfer limitations were thoroughly investigated and excluded for the reported catalytic data. Finally, kinetic parameters for a modified power-law type kinetic model were determined for the first time at 50 bar and temperatures up to 750 degrees C, determining an apparent activation energy of 170.1 kJ mol-1, while the reaction orders for NH3 and H2 were found to be 0.76 and-1.22, respectively.
Abstract Welded hollow sections trusses are widely used in structural engineering, such as pedestrian bridges and crane constructions. High-strength steels (HSS) enable lightweight, high-capacity designs. Current design rules for hollow section joints, based on tests with mild-strength steels (MSS), assume that plastic load redistribution mitigates local stress peaks. However, HSS only allows limited plastic redistribution due to manufacturing-induced hardening. Due to the limited test data available for HSS hollow-section joints, existing design rules developed for mild-strength steel joints apply conservative reduction factors ( C f ) based on the material strength, which decrease the design resistance. To improve the competitiveness and resource efficiency of HSS hollow section joints, further experimental and numerical investigations are required. This paper presents comprehensive numerical investigations of HSS K-joints and the revision of existing material-dependent reduction factors, C f , for steel grades exceeding S355. Numerical models based on damage mechanics were validated against large-scale K-joint tests with circular and square hollow sections of varying geometries and chord preloading. The underlying material models were derived from extensive laboratory tests on the investigated steels to simulate the joint behavior up to failure accurately. The results include a detailed evaluation of the influence of the weld size on the load-bearing behavior and a presentation of the different observed failure modes of the HSS K-joints. Within the approximately 700 full-scale damage mechanical simulations, each individual parameter of the design rules could be evaluated. While the reduction factors currently applied provide a safe design, the design rules should be revised and adapted more specifically to the failure modes. This facilitates more appropriate structural designs and enables the full exploitation of the potential of higher steel grades.
