Pulse neural networks, suitable for hardware implementation based on memristors, are promising for robotics due to their energy efficiency. However, reinforcement learning algorithms using such networks remain poorly understood. One of the key motivations for using memristors as network weights is, in addition to energy efficiency, their ability to learn (change conductivity) in real time by superimposing voltage pulses from pre- and postsynaptic signals. This article presents the results of numerical simulation of a spiking neural network (SNN) with memristive synaptic connections, approximately solving an optimal control problem using trace variables to change weights, allowing one to approach reinforcement learning in real time. The fundamental possibility of such training in the task of holding a pole on a moving platform is shown, a comparison of various reward functions is given, and assumptions are made about ways to increase the effectiveness of this approach.

An Erratum to this paper has been published: https://doi.org/10.1134/S0031918X2355001X

Comparative studies of the properties of metal/nanocomposite/metal (M/NC/M) memristive structures based on NC consisting of a LiNbO 3 matrix and various Co-Fe-B or CoFe metal granules have been carried out. The M/NC/M structures were obtained using ion-beam sputtering from composite targets Co 40 Fe 40 B 20 -LiNbO 3 and Co 50 Fe 50 -LiNbO 3 on glass-ceramic substrates. The same NC layers were synthesized on polyimide substrates to study the structural features by X-ray absorption fine structure spectroscopy (XAFS). The XAFS data show an identical crystal structure of granules in both types of NC, indicating that a significant part of the B atoms during the synthesis of NC is in an insulating matrix, forming an oxide of the B 2 O 3 type, which is confirmed by X-ray photoelectron spectroscopy data. In this case, no metallic state of niobium is observed in the layers. Both types of M/NC/M structures demonstrate resistive switching (RS), however, in the case of NC with boron, the RS effect is much stronger, which is explained by the significant role of oxygen vacancies formed during the oxidation of boron in the RS.

Предложена компактная поведенческая модель нанокомпозитного мемристора (Co 40 Fe 40 B 20 ) x (LiNbO 3 ) 100 – x , которая количественно описывает динамику изменения проводимости лабораторных образцов, а также реализует механизмы конечного времени хранения резистивных состояний, разбросов по напряжениям переключения от цикла к циклу и от устройства к устройству. Показана возможность реализации импульсной нейронной сети с синаптическими мемристорными связями на основе данной модели.

На примере импульсной нейросети в архитектуре «четыре входных нейрона - два выходных нейрона - два тормозных нейрона с мемристивными синаптическими связями» показана важная роль ингибирующих связей для реализации обучения с учителем на аппаратном уровне при решении задачи классификации простых графических образов.

Comparative studies of resistive switching (RS) effect of metal/nanocomposite/metal (M/NC/M), metal/nanocomposite/LiNbO3/metal (M/NC/LNO/M) structures based on NC (Co40Fe40B20)х(LiNbO3)100-x (x = 6–20 at.%) with CoFe nanogranules 2–4 nm in size, as well as structures without a NC layer (M/LNO/M), have been carried out. It was found that the percolation conductivity in NC and presence of a thin LNO layer play a key role in the RS effect. When the metal content approaches the percolation threshold of M/NC/M structures (xp ≈ 10 at.%), low-resistance percolation nanochannels of granules are formed in structures with an embedded LNO layer, which ensure their stable RS, which, however, are noticeably suppressed as x decreases relative to xp by Δх ≈ 1-2 at.%.

The memristive properties of Cu/nanocomposite/ LiNbO 3 /Cu capacitor structures based on a (Co 40 Fe 40 B 20 ) x (LiNbO 3 ) 100 – x nanocomposite and an amorphous LiNbO 3 interlayer with thicknesses of about 40 and 20 nm, respectively, have been studied. It was found that these structures have relatively low resistive switching voltages (~2 V) and are capable of withstanding more than 10 4 cyclic switchings due to the formation of conducting channels in LiNbO 3 in fixed regions specified by the position of percolation chains of CoFe nanograins in the nanocomposite. It is shown that the conductance of Cu/nanocomposite/LiNbO 3 /Cu memristors can vary according to local biosimilar rules. A simple neural network based on such memristors, trained by feeding a frequency-coded noise signal to its inputs, was implemented.

В работе исследованы мемристоры на базе нанокомпозита (НК) (CoFeB) x(LiNbO3)1-x и построена на их основе импульсная нейроморфная сеть (ИНС) с четырьмя входными и одним выходным нейроном. Было показано, что в ИНС на основе НК мемристивных синапсов, помимо частотного, возможно и временное кодирование образов.

Influence of oxygen and water vapor in the chamber during the deposition of thin-film nanocomposites (Co40Fe40B20)x(LiNbO3)100-x on the electrical properties of the heterogeneous system has been studied. A significant increase in the resistivity of (Co40Fe40B20)x(LiNbO3)100-x nanocomposites with an increase in the partial pressure of reactive gases (oxygen and water vapor) has been established. A significant shift of the percolation threshold towards higher metal phase concentration in the film plane and perpendicular to the film was found during the synthesis of composites with the addition of reactive gases, which is associated with an increase in the volume concentration of the dielectric phase. It is revealed that the percolation threshold when measured in the geometry perpendicular to the plane of the film corresponds to a significantly lower concentration of atoms of the Co40Fe40B20 alloy than in the case of measurements in the plane of the film, which is associated with the elongated shape of the metallic granules in the direction of film growth and the effects of Coulomb blockade suppression by a high transverse electric field.

Представлен оригинальный способ математического обоснования оптимальных локальных правил обучения спайковых нейронных сетей (СНС) с частотным кодированием информации и их возможная связь с правилами обучения типа STDP. Обсуждаются подходы к реализации системы ценностей интеллектуального агента на базе локальных правил обучения с подкреплением в СНС. Также демонстрируются возможные аппаратные решения для моделей нейронов и синаптических весов на базе оригинальных мемристоров, подходящих для указанных типов локального обучения. Таким образом, представлена попытка обоснования компонент, необходимых для будущих нейроморфных систем универсального искусственного интеллекта на основе СНС.