BackgroundA Just-in-Time Adaptive Intervention (JITAI) recognizes the dynamic nature of individuals’ states and contexts, predicts support needs, and sends tailored support at more opportune, actionable times. ObjectiveThis paper outlines the application architecture and protocol for the pilot “Walking With Just-in-Time Adaptive Interventions” (WWJ) study, which uses a JITAI approach to improve walking behavior—duration, speed, and distance—and reduce stationary time, defined as idle sitting or standing. MethodsThis study targets 20 adults who are physically inactive and leverages the Apple Watch to deliver fully automated tailored intervention notifications to “walk faster,” “walk longer,” or “stand up and move around” based on real-time data and contextual factors, including time-of-day activity patterns, geographic locations (eg, home, work, park, and gymnasium), weather conditions (eg, precipitation, wind speed, and humidity), and receptiveness. The protocol involves a preintervention assessment of demographics, behavior change constructs, anthropometrics, and resting vital signs; a 2-week observation period to establish walking behavior and stationary time baselines; a 2-week just-in-time learning period to evaluate receptiveness to untailored prompts at all applicable times; the 2-week JITAI intervention phase; and a postintervention assessment. Feasibility will be evaluated through protocol fidelity, participant adherence, Apple Watch wear-time compliance, user burden, acceptability ratings, and perceptions of benefits and preferences. ResultsThe WWJ architecture development began in spring 2021 and concluded in fall 2022. Participant recruitment and enrollment began in fall 2022. A total of 18 participants were recruited. Upon completion of the analyses, the results of this study are expected to be submitted for publication. ConclusionsDistinctively, the WWJ just-in-time learning period aims to train the learner based on user receptiveness within contexts by sending interventions whenever a participant meets the predetermined thresholds regardless of the likelihood that the user will be receptive to the notification to prune out nonopportune or “nonactionable” times. This approach may allow for greater customization during the JITAI period. International Registered Report Identifier (IRRID)DERR1-10.2196/79022

Objectives:To improve nudge outcome classification accuracy in a context-aware personalized nudging framework using wearable sensor data targeted to reduce sedentary behavior using Just- in-Time Adaptive Interventions (JITAIs). Methods:Data were collected using a custom smartwatch application in a free-living observational study conducted at the University of Delaware (Newark, Delaware, USA) between Spring 2021 and Fall 2022. A total of 18 participants were enrolled. The system continuously recorded motion, physiological, and contextual data and delivered adaptive behavioral prompts. A decision- tree model was trained using sitting and walking bouts enriched with contextual features such as time, location, physiological state, and prior intervention outcomes. Behavioral responses were automatically evaluated using sensor-derived outcomes. Results:The proposed model improved classification accuracy for nudge outcomes from 0.42 to 0.78 across 787 sitting bouts. A walking-nudge model achieved an accuracy of 0.70 on 207 walking bouts. Nudged walking bouts were longer in duration, covered greater distances, and exhibited higher average speeds than non-nudged bouts. Conclusions:Context-aware adaptive nudging can improve both the timing and behavioral effectiveness of wearable-based interventions. Incorporating contextual and historical features enables personalized and behaviorally meaningful intervention delivery. Policy Implications:Wearable-based adaptive interventions offer a scalable and cost-effective strategy to reduce sedentary behavior and support population-level health promotion.

Background: Physical activity is an effective modifiable behavior for preventing recurrent strokes. This study aims to determine the adherence to physical activity recommendations among stroke survivors in the United States. We further compared our findings with the adherence observed among myocardial infarction (MI) survivors. Methods: We utilized data from the 2011-2019 Behavioral Risk Factor Surveillance System, a nationally representative survey. To establish benchmarks, we referenced the physical activity recommendations outlined in 2011, 2014, and 2021 American Heart Association stroke prevention guidelines. Adherence to recommendations was determined by the respondents' self-reported intensity, duration, and frequency of physical activity. Multivariate logistic regression was used to compare adherence in stroke survivors, MI survivors, and healthy adults. Results: Among 48,222 stroke survivors in the United States, the overall adherence rates to 2011, 2014, and 2021 physical activity guidelines were 75.4%, 40.2%, and 69.2%. For independently mobile stroke survivors, the adherence rates increased to 78.1%, 42.1%, and 69.9%. When 2021 recommendations were used as a benchmark, older (≥65) stroke survivors were more likely to adhere to recommendations than younger survivors (71.9%vs.62.3%; p<0.0001). However, when the benchmark was changed to the 2011 and 2014 guidelines, which recommended longer exercise durations, the difference between younger and older stroke survivors dissipated. After adjusting for sociodemographic factors and comorbidities, non-Hispanic Black survivors were less likely to adhere to recommendations (aOR,0.81[95%CI,0.7–0.94]), whereas older and higher educated stroke survivors were more likely to adhere to recommendations. Geographically, stroke belt and non-rural residents were less likely to adhere to recommendations [(63.5%vs.67.9%;p<0.0001), (53.8%vs.58.7%;p<0.0001)]. Stroke and MI survivors were less likely to adhere to the latest recommendations than healthy adults (aOR,0.74[95%CI,0.69-0.8], (aOR,0.24[95%CI,0.22-0.26])). Conclusion: A substantial number of stroke survivors do not meet physical activity recommendations. Tailored interventions should be designed for at-risk populations, e.g., non-Hispanic Black survivors and lower educated stroke survivors.

Background: Self-management among stroke survivors is effective in mitigating the risk of a recurrent stroke. This study aims to determine the prevalence of self-management and its associated factors among stroke survivors in the United States. Methods: We analyzed the Behavioral Risk Factor Surveillance System (BRFSS) data from 2016 to 2021, a nationally representative health survey. A new outcome variable, stroke self-management (SSM = low or SSM = high), was defined based on five AHA guideline-recommended self-management practices, including regular physical activity, maintaining body mass index, regular doctor checkups, smoking cessation, and limiting alcohol consumption. A low level of self-management was defined as adherence to three or fewer practices. Results: Among 95,645 American stroke survivors, 46.7% have low self-management. Stroke survivors aged less than 65 are less likely to self-manage (low SSM: 56.8% vs. 42.3%; p < 0.0001). Blacks are less likely to self-manage than non-Hispanic Whites (low SSM: 52.0% vs. 48.6%; p < 0.0001); however, when adjusted for demographic and clinical factors, the difference was dissipated. Higher education and income levels are associated with better self-management (OR: 2.49, [95%CI: 2.16–2.88] and OR: 1.45, [95%CI: 1.26–1.67], respectively). Further sub-analysis revealed that women are less likely to be physically active (OR: 0.88, [95%CI: 0.81–0.95]) but more likely to manage their alcohol consumption (OR: 1.57, [95%CI: 1.29–1.92]). Stroke survivors residing in the Stroke Belt did not self-manage as well as their counterparts (low-SSM: 53.1% vs. 48.0%; p < 0.001). Conclusions: The substantial diversity in self-management practices emphasizes the need for tailored interventions. Particularly, multi-modal interventions should be targeted toward specific populations, including younger stroke survivors with lower education and income.

The genetic code determines how the precise amino acid sequence of proteins is specified by genomic information in cells. But what specifies the precise histologic organization of cells in plant and animal tissues is unclear. We now hypothesize that another code, the tissue code , exists at an even higher level of complexity which determines how tissue organization is dynamically maintained. Accordingly, we modeled spatial and temporal asymmetries of cell division and established that five simple mathematical laws ("the tissue code") convey a set of biological rules that maintain the specific organization and continuous self-renewal dynamics of cells in tissues. These laws might even help us understand wound healing, and how tissue disorganization leads to birth defects and tissue pathology like cancer.

Health coaching is an evidence-based approach to help individuals adopt health behaviors such as physical activity (PA). However, human health coaching is limited by a lack of real-time data and scalability. In this paper, we present the architecture and functionality of a novel semi-automated health coach that we call BeSMART that automates the goal modification and dialogue functionalities of human health coaching using a commercial health tracker and mobile phone. Results from an acceptability study (N=10) indicate that users have difficulty fully completing all actions as initially planned. To build planning systems that can dynamically adapt action plans to such changes, we show that user preferences are vital, and that users find re-planning acceptable. Towards solving this problem, we extracted and categorized user preferences from health coaching sessions and propose extensions to Hierarchical Task Network (HTN) representations. When taking users personally re-planned activities into account, post-hoc analysis shows that users are much more likely to complete their overall PA goals.

Wearable technology opens opportunities to reduce sedentary behavior; however, commercially available devices do not provide tailored coaching strategies. Just-In-Time Adaptive Interventions (JITAI) provide such a framework; however most JITAI are conceptual to date. We conduct a study to evaluate just-in-time nudges in free-living conditions in terms of receptiveness and nudge impact. We first quantify baseline behavioral patterns in context using features such as location and step count, and assess differences in individual responses. We show there is a strong inverse relationship between average daily step counts and time spent being sedentary indicating that steps are steadily taken throughout the day, rather than in large bursts. Interestingly, the effect of nudges delivered at the workplace is larger in terms of step count than those delivered at home. We develop Random Forest models to learn nudge receptiveness using both individualized and contextualized data. We show that step count is the least important identifier in nudge receptiveness, while location is the most important. Furthermore, we compare the developed models with a commercially available smart coach using post-hoc analysis. The results show that using the contextualized and individualized information significantly outperforms non-JITAI approaches to determine nudge receptiveness.

For software applications in health coaching domains to be effective, it is vital that they address issues of privacy, modularity, scalability, individualization, data integration, transferability, coordination and flexibility. In this paper, we propose a novel generic multi-agent architecture which serves as a template for health coaching applications involving wearable sensors. Analyzer and communication modules allow different functionalities like goal formation, planning, scheduling, event detection, learning, inter-agent + human communication and long-term data collection, based on the capabilities of the underlying sensor platforms. To show the flexibility of our proposed architecture, we have successfully built two different health coaching systems with the proposed architecture: (1) a static system based on the Fitbit platform where the coaching is done at specific preset times to encourage increased physical activity, and (2) a dynamic system based on the Apple Watch platform where the smart coach adapts and learns when to intervene to encourage physical activity and reduce sedentary behavior.

Branching patterns occur throughout nature and are often described by the Fibonacci numbers. While the regularity of these branching patterns in biology can be described by the Fibonacci numbers, the branches (leaves, petals, offshoots, limbs, etc.) are often variegated (size, color, shape, etc.). To begin to understand how these patterns arise, we considered different branching patterns based on p-Fibonacci sequences. In our model, different branching patterns were created based on a specific number of decreasing-sized branches that arise from a main branch (termed the degree of branching). It was assumed that the ratio between the sizes of pairs of consecutive branches (ordered by size) equals the ratio of the largest branch size to the sum of the largest and smallest branch sizes. Generation of these branching structures illustrates that pattern self-similarities occur across different degrees of branching and multiple dimensions. Conclusion: studying geometric branching patterns based on p-Fibonacci sequences begins to show how the regularity in branching patterns might occur in biology.

We build a profitable electronic trading agent with Reinforcement Learning that places buy and sell orders in the stock market. An environment model is built only with historical observational data, and the RL agent learns the trading policy by interacting with the environment model instead of with the real-market to minimize the risk and potential monetary loss. Trained in unsupervised and self-supervised fashion, our environment model learned a temporal and causal representation of the market in latent space through deep neural networks. We demonstrate that the trading policy trained entirely within the environment model can be transferred back into the real market and maintain its profitability. We believe that this environment model can serve as a robust simulator that predicts market movement as well as trade impact for further studies.

This work studies mixed-autonomy traffic optimization at a network level with Deep Reinforcement Learning (DRL). In mixed-autonomy traffic, a mixture of connected autonomous vehicles (CAVs) and human driving vehicles is present on the roads at the same time. We hypothesize that controlling distributed CAVs at a network level can outperform the individually controlled CAVs. Our goal is to improve traffic fluidity in terms of the vehicle's average velocity and collision avoidance. We propose three distributed learning control policies for CAVs in mixed-autonomy traffic using Proximal Policy Optimization (PPO), a policy gradient DRL method. We conduct the experiments with different traffic settings and CAV penetration rates on the Flow framework, a new open-source microscopic traffic simulator. The experiments show that network-level RL policies for controlling CAVs outperform the individual-level RL policies in terms of the total rewards and the average velocity.

Recent multi-agent actor-critic methods have utilized centralized training with decentralized execution to address the non-stationarity of co-adapting agents. This training paradigm constrains learning to the centralized phase such that only pre-learned policies may be used during the decentralized phase, which performs poorly when agent communications are delayed, noisy, or disrupted. In this work, we propose a new system that can gracefully handle partially-observable information due to communication disruptions during decentralized execution. Our approach augments the multi-agent actor-critic method's centralized training phase with generative modeling so that agents may infer other agents' observations when provided with locally available context. Our method is evaluated on three tasks that require agents to combine local and remote observations communicated by other agents. We evaluate our approach by introducing both partial observability during decentralized execution, and show that decentralized training on inferred observations performs as well or better than existing actor-critic methods.

In cooperative game theory, the two foremost problems are determining what coalitions will form and how a coalition's payoff should be divided. The Shapley value, a proven fair and unique payoff distribution, has become a central solution concept in the field. Computing the Shapley value is exponential in the number of agents, however, and has motivated many practical approximation methods, only two of which apply to arbitrary cooperative games. We propose a Shapley value approximation method using hierarchical clustering that partitions coalitions based on agent feature similarity and then interpolates the subcoalitions' Shapley values. Additionally, the approximation is guaranteed to satisfy the Shapley value's desirable fairness properties of symmetry, efficiency, and often null player. With a low runtime, experimental error, and a tuning parameter for error-runtime trade-off, this algorithm is the most practical for cooperative games requiring fast, near-optimally fair payoff distributions.

One of the biggest obstacles of Reinforcement Learning (RL) is its slow convergence rate in large state spaces or with sparse rewards. It has been shown that single-agent RL can be accelerated within a cooperative multi-agent scenario with information sharing, however the speedup depends on how well the agents' information can be used together. We demonstrate in this paper that state-space partitioning among agents can be realized by reward design without hard coded rules. The partitioning-associated reward directs agents to focus on different partitions and thus share information more efficiently. This approach has two advantages: (1) agents' actions are not diminished and remain relatively independent from one another; (2) it can be used to accelerate learning in both structured state domains (where partitions can be pre-determined) and arbitrarily-structured state domains (where partitions may be developed dynamically by agent teams as they explore the environment). Finally, we validate the method's efficacy by comparing it to previous related work in a simplified soccer domain.

We examine Probabilistic Partial Policy Reuse (PPR) for the purposes of developing tailored coaching strategies in the Coach-Trainee Problem (CTP). Policy reuse (PR) aims to improve a reinforcement learning agent by guiding exploration with past similar problems’ learned policies. PPR extends probabilistic policy reuse that transfers only relevant parts of a policy for new problems. We explore PPR in the context of a human CTP where a coaching agent must develop a coaching strategy for the human trainee in order for the trainee to efficiently solve their problem (e.g. lose weight). In human CTPs, coach training data is limited because collecting too much data may annoy/discourage/harm the human trainee. In this paper, we present a decision tree-based algorithm, DTpartition, to identify partitions in the state space based on the problem’s features, and also examine the effects of grouping problem meta-data (i.e. pruning the decision tree) on CTP performance. Particularly, we demonstrate that PPR improves task library generation and expert policy utilization compared

To serve large user populations, autonomous intervention systems (i.e. intelligent agents) are being developed to play more active roles such as fitness coaches and clinical disease prevention aids. Although generic user models have been developed, users may require extensive individualization to meet their personal needs. Machine learning techniques may be applied to learn tailored intervention policies for users. However, traditional machine learning requires significant amounts of data to learn an optimal policy. For wearable technology, this may mean probing the user to perform some activity and gauging user response. This paper presents a feasible intervention system model and discusses learners for tailoring user intervention policies. We examine how similar the general user model has to be with respect to the tailored model in order for our learner to perform well.