A wide range of scenarios, such as warehousing, and smart manufacturing, have used RFID mobile robots for the localization of tagged objects. The state-of-the-art RFID-robot based localization works are based on the premise of stable speed. However, in reality this assumption can hardly be guaranteed because Commercial-Off-The-Shelf (COTS) robots typically have inconsistent moving speeds, and a small speed inconsistency will cause a large localization error. To this end, we propose a Speed Inconsistency-Immune approach to mobile RFID robot Localization (SILoc) system, which accurately locates targets when the robot moving speed varies or is even unknown. We propose an optimized unwrapping method to maximize the use of data, and a lightweight algorithm to calculate the locations in both 2D and 3D spaces. By utilizing the characteristics of tag-antenna distance and combining the phase data from multiple antennas, SILoc can effectively eliminate the side effects of speed inconsistency. To increase the flexibility, we further optimize the system and propose SILoc $+$ , which enables the system to achieve localization with part of the data, keeping speed inconsistency-immune. Extensive experiments demonstrate that SILoc and SILoc $+$ can achieve a centimeter-level localization accuracy in the scenario with an inconsistent or unknown robot moving speed.

With the wide application of multi-modal fusion sensing in scenarios such as autonomous driving and human-computer interaction, the privacy security and communication burden caused by massive data uploading need to be solved urgently. Federated Learning (FL) has received significant attention as a privacy-preserving distributed machine learning paradigm. Recent Multi-Modal Federated Learning (MMFL) focuses on addressing modal heterogeneity to enhance accuracy and speed up convergence. However, it overlooks the huge communication overhead in updating complex multi-modal network models, especially in edge environments with limited bandwidth. At the same time, the state-of-the-art communication-efficient FL methods are not customized to the MMFL characteristics. In this paper, we propose the Adaptive Quantization framework for Multi-modal Federated Learning (AQMFL). AQMFL implements decision-level multi-modal fusion locally by using parallel training and model ensemble, supporting its adaptation to modal heterogeneity and flexible deployment. AQMFL can adaptively allocate the number of quantization levels of gradient according to the modal contribution and the heterogeneous communication ability of nodes, which speeds up the system convergence and achieves a better balance between accuracy and communication efficiency. Compared with the classical baselines, AQMFL can reduce the total communication overhead by up to 50.47% and the total training time by up to 52.11% while maintaining the accuracy.

Mobile robot-assisted book inventory such as book identification and book order detection has become increasingly popular in smart library, replacing the manual book inventory which is time-consuming and error-prone. The existing systems are either computer vision (CV)-based or RFID-based, however several limitations are inevitable. CV-based systems may not be able to identify books effectively due to low accuracy of detecting texts on book spine. RFID tags attached to books can be used to identify a book uniquely. However, in high tag density scenarios such as library, tag coupling effects of adjacent tags may seriously affect the accuracy of tag reading. To overcome these limitations, this paper presents a novel RFID and CV fusion system for Book Inventory using mobile robot (RC-BI). RFID and CV are first used individually to obtain book order, then the information will be fused by the sequence based matching algorithm to remove ambiguity and improve overall accuracy. Specifically, we address three technical challenges. We design a deep neural network (DNN) model with multiple inputs and mixed data to filter out interference of RFID tags on other tiers, and propose a video information extracting schema to extract book spine information accurately, and use strong link to align and match RFID- and CV-based timestamp vs. book-name sequences to avoid errors during fusion. Extensive experiments indicate that our system achieves an average accuracy of 98.4% for tier filtering and an average accuracy of 98.9% for book order, significantly outperforming the state-of-the-arts.

In this article, we study the problem of wireless human sensing, which refers to human activity recognition (HAR). HAR based on wireless signals plays an important role in security, human-computer interaction, and healthcare in the 5G era. Most state-of-the-art human activity recognition applications rely on deep learning approaches, which require a large amount of training data to achieve good performance. However, wireless signal data is difficult to collect and label, and it also carries private information, making it challenging to construct large-scale datasets.The recent advances in federated learning provide a chance to aggregate a wide range of users to collaboratively train a HAR model using decentralized datasets under data-preserving constraints. However, since a wireless signal is easily interrupted by the environment, the data across all participants is non-IID, thus decreasing the performance of an aggregated model. Additionally, due to the resource-constrained nature of edge devices, training the HAR model on an end user usually takes too long, resulting in straggler problems in federated learning training. In this article, we proposed a cross-domain federated learning framework (CDFL) to address the lack of labeled wireless data. A transfer learning approach was proposed to simulate wireless data by converting from widely available image datasets, and solving the distribution mismatch problem by domain adaption. Additionally, a customized federated learning approach was proposed to reduce the computational overhead of local model training. Using a case study of ultrasonic signal-based gesture recognition, we demonstrate the effectiveness of the proposed framework. Our method achieves over 90 percent accuracy on a 5-category task without real data, and 88 percent accuracy on a 10-category task when the user collects only one piece of data.

With the wide deployment of RFID in various scenarios such as warehouse management, freight transportation, and manufacturing, tag authentication is increasingly important due to the threat of counterfeit tags. Recent physical-layer authentication approaches have demonstrated that the subtle differences in the hardware features offer a unique fingerprint to authenticate a tag. Although the state-of-the-art approaches are effective in laboratory environments, they are difficult for practical deployment because they either require complex analysis of the raw signal propagation or restrict the geometrical positions of the tags. In this paper, we propose an RFID tag authentication based on frequency- and orientation-related phase fingerprints, called FopPrint, which does not require raw signal analysis or complex geometric relationship. FopPrint uses the phase values of tags at different frequencies and orientations to construct feature matrices as physical-layer fingerprints and uses a pair of adjacent tags as identifiers of each object. FopPrint can effectively eliminate the influence of environmental factors by using the feature matrix constructed by the phase difference. We implement a prototype of FopPrint using Commercial-Off-The-Shelf (COTS) RFID devices. Extensive experimental results show that FopPrint achieves high authentication accuracy of 94% in various experimental settings.

Radio Frequency Identification (RFID) has been adopted in various applications owning to its many attractive properties such as low cost, no requirement on line-of-sight, and battery-free. This paper studies the problem of RFID-based Handwriting recognition, which is practically important in Human-Computer Interaction (HCI) scenarios. To the best of our knowledge, the state-of-the-art works beget leaking user privacy, because the malicious attacker can eavesdrop on the RFID signals (e.g., tag phase) broadcast in the air and further analyze the user's handwriting activity. To address the privacy leakage issue, we propose a secure RFID handwriting recognition system named SecRFPen to enable privacy-preserving handwriting recognition. In SecRFPen, the legal reader switches the probing frequency and power, the phase angles of RF signals reflected by the tagged pen will change accordingly. Thus, the phase profile of the tagged pen is actually determined by both readertag hardware characteristics and handwriting movements. We propose an authentication matrix to quantify RFID device hardware characteristics, which can be measured by legal users in advance. Thus, the legal RFID reader can recognize the handwriting activity via analytics on the authentication matrix and tag phase profile. On the contrary, since the malicious attacker knows nothing about the hardware characteristics of legal RFID devices, it cannot understand handwriting even if it can hear the tag signals. We implement the SecRFPen system based on the Commercial-Off-The-Shelf (COTS) RFID devices. Extensive experimental results demonstrate that the recognition accuracy of legal users can reach 94.2%, while the recognition accuracy of the malicious attacker is as low as 35.1%.

This paper studies the problem of tag localization using RFID-augmented robots, which is practically important for promising warehousing applications, e.g., automatic item fetching and misplacement detection. Existing RFID localization systems suffer from one or more of following limitations: requiring specialized devices; only 2D localization is enabled; having blind zone for mobile localization; low scalability. In this paper, we use Commercial Off-The-Shelf (COTS) robot and RFID devices to implement a Mobile RF-robot Localization (MRL) system. Specifically, when the RFID-augmented robot moves along the straight aisle in a warehouse, the reader keeps reading the target tag via two vertically deployed antennas ( Z1 and Z2) and returns the tag phase data with timestamps to the server. We take three points in the phase profile of antenna Z1 and leverage the spatial and temporal changes inherent in this phase triad to construct an equation set. By solving it, we achieve the location of target tag relative to the trajectory of antenna Z1. Based on different phase triads, we can have candidate locations of the target tag with different accuracy. Then, we propose theoretical analysis to quantify the deviation of each localization result. A fine-grained localization result can be achieved by assigning larger weights to the localization results with smaller deviations. Similarly, we can also calculate the relative location of target tag with respect to the trajectory of antenna Z2. Leveraging the geometric relationships among target tag and antenna trajectories, we eventually calculate the location of target tag in 3D space. We perform various experiments to evaluate the performance of the MRL system and results show that the proposed MRL system can achieve high accuracy in both 2D and 3D localization.

Mobile RFID robots have been increasingly used in warehousing and intelligent manufacturing scenarios to pinpoint the locations of tagged objects. The accuracy of state-of-the-art RFID robot localization systems depends much on the stability of robot moving speed. However, in reality this assumption can hardly be guaranteed because a Commercial-Off-The-Shelf (COTS) robot typically has an inconsistent moving speed, and a small speed inconsistency will cause a large localization error. To this end, we propose a Speed Inconsistency-Immune approach to mobile RFID robot Localization (SILoc) system, which can accurately locate RFID tagged targets when the robot moving speed varies or is even unknown. SILoc employs multiple antennas fixed on the mobile robot to collect the phase data of target tags. We propose an optimized unwrapping method to maximize the use of the phase data, and a lightweight algorithm to calculate the locations in both 2D and 3D spaces based on the unwrapped phase profile. By utilizing the characteristics of tag-antenna distance and combining the phase data from multiple antennas, SILoc can effectively eliminate the side effects of moving speed inconsistency. Extensive experimental results demonstrate that SILoc can achieve a centimeter-level localization accuracy in the scenario with an inconsistent or unknown robot moving speed.

Localization of target tagged objects on the shelf is of great significance in RFID-enabled warehousing scenarios. Compared with the RFID localization systems that use fixed reader antennas or mobile RFID-robot, the portable reader-based methods are much more cost-effective. Hence, this paper focuses on reader-portable RFID localization. However, the existing reader-portable localization systems suffer from the following limitations: (i) reader antenna is required to pass by the target tags. Thus, the tags in the corner can never be located; (ii) many reference tags need to be deployed on the shelf in advance, which considerably increases the manpower; (iii) specialized antenna is required, which limits the promotion potential. To this end, this paper proposes a Waving action-driven RFID Localization (WRL) system, which enables tag localization with a portable camera-augmented reader. In the WRL system, a user only needs to wave the camera-augmented reader before locating the target tags. Specifically, we first use a classical camera pose estimation method named PnP to recover the antenna’s movement trajectory in a pixel coordinate system. Then, WRL constructs a gridded hologram, in which camera data and RFID phase data are jointly used to calculate a probability for each grid. Intuitively, the higher probability a grid has, the more possible the target tag lies in the corresponding grid. Based on this idea, WRL calculates the target tag’s location on the shelf. We use the Commercial-Off-The-Shelf (COTS) RFID and camera devices to implement the WRL system. Extensive experiments have been conducted, and the results demonstrate that the mean localization error of WRL is less than 20cm with a confidence of about 95%.

Recognition of human-object interactions is practically important in various human-centric sensing scenarios such as smart supermarket, factory, and home. This paper proposes an RF-Camera system by fusing RFID and Computer Vision (CV) techniques, which is the first work to recognize the human gestural interactions with physical objects in multi-subject and multi-object scenarios. In RF-Camera, we first propose a dimension reduction method to transform the subject's 3D hand trajectory captured by depth camera to a 2D image, using which the subject's gesture can be recognized. We also propose a method to extract the facial image of target subject from an image that may contain irrelevant subjects, thereby further recognizing his/her identity. Finally, we model the physical movements of the held object's tag and further predict the tag phase data, by comparing which with real phase data of each tag human-object matching can be discovered. When implementing RF-Camera, three technical challenges need to be addressed. (i) To remove noisy data corresponding to irrelevant actions from raw sensing data, we propose a state transition diagram to determine the boundary of effective data. (ii) To predict phase data of the held target tag with unknown hand-tag offset, we quantify target tag trajectory by adding a variable hand-tag vector to captured hand trajectory. (iii) To ensure high reading rates of target tags in tag-dense scenarios, we propose a CV-assisted RFID scheduling method, in which analytics on CV data can help schedule RFID readings. We conduct extensive experiments to evaluate the performance of RF-Camera. Experimental results demonstrate that RF-Camera can recognize the gestural actions, human identity and human-object matching with an average accuracy higher than 90% in most cases.

In recent years, there is an increasing demand for indoor localization services with the aim to locate people and objects inside buildings. However, localization accuracy is susceptible to inaccurate and high variant sensor measurements due to the unpredictable fluctuations of received wireless signals and the sensitivity of hardware devices. To address this issue, in this paper, we establish a new Bluetooth indoor localization system, whose architecture can be basically decomposed into two parts: the internet-of-things (IoT) framework and the localization module. Concretely, the IoT platform uses the state-of-the-art light weight Spring Boot microservice framework consisting of multi-layer structure. In the localization module, it follows the general process of trilateration but significantly distinguished from it. A set of measures are adopted to strengthen the system’s robustness when obtained measurements cannot be fully trusted. Specifically, in the first place, rather than using conventional propagation model to predict the distance between Bluetooth transmitter and receiver, we design a bran-new LSTM-based distance estimator which can better depict the nonlinearity of attenuation characteristics of radio signal. Moreover, we also employ a series of self-adaptive mechanisms, including elastic radius intersecting, multiple weighted centroid localization and self-adaptive Kalman tracking, to make the system robust against inaccurate measurements and unpredictable sudden variation of received wireless signal. A bunch of tests are conducted in both ideal lab environment and Alibaba’s large-scale warehouse, and experimental results show our indoor localization system outperforms the state-of-the-art benchmarks by a large margin in both localization accuracy and stability.

Wireless human sensing plays a crucially important role in the human-computer interaction context, in which human activities and even emotions can be recognized and understood by computers. A batch of academic efforts have been proposed to use wireless techniques such as WiFi, RFID, Bluetooth, Radar, and Zigbee to address the human sensing problem. Each kind of sensing technology has its own characteristics and advantages and thus is suitable for specific application scenarios. For example, WiFi-based solutions can achieve non-intrusive human sensing, and RFID-based solutions can enable individual human sensing for multi-person scenarios. To let users better understand the existing wireless human sensing solutions and choose the most suitable one according to their demands, this article presents a comprehensive survey of the existing wireless human sensing approaches. Specifically, we discuss promising human sensing applications and partition them into three categories: vital sign monitoring, gesture recognition, and activity recognition. For each category of applications, we further conduct a taxonomy of the existing solutions and summarize their ideas, characteristics, pros, and cons from various perspectives, such as design approaches, system configuration, wireless technology, and used information. Finally, we discuss some technical challenges and problems that have not been noticed yet and point out the potential opportunities in the future study of wireless human sensing.