Similarity-based geocoding pipelines are highly sensitive to lexical noise in address data. This work evaluates the potential of language models to enhance and standardize address inputs during preprocessing, thereby improving geocoding accuracy in CIDACS-RL, which relies on Jaro-Winkler similarity and is sensitive to string length and prefix agreement. We extended CIDACS-RL with a preprocessing step that removes street types (e.g., street, avenue, lane) and stop words and expands numeric and abbreviated address components. Two pre-trained language models were included in the experiments. Records from nine Northeastern Brazilian states in the cohort baseline (N = 54,985,455) were geocoded by linking addresses to census tracts or coordinates from the Brazilian National Register of Addresses (CNEFE; N = 30,545,117). After linkage, we manually assessed the accuracy of our extension against previous CIDACS-RL version using stratified samples (N = 2,000) from each run to determine the cutoff points. The geocoding was evaluated using accuracy and precision. Overall, the proposed extension achieved competitive performance, with higher mean precision (0.86, SD = 0.01) and accuracy (0.94, SD = 0.024) compared to the original CIDACS-RL, which achieved a mean precision of 0.84 (SD = 0.02) and a mean accuracy of 0.93 (SD = 0.024). The largest gains were observed in states with shorter average street name lengths, such as Alagoas and Pernambuco, where precision increased by approximately 3% to 10%. Our findings support that geocoding tasks may benefit from pre-trained language models for preprocessing steps, addressing the limitation of well-posed similarity measures.

This work aims to construct a dataset of unique addresses from Brazilian administrative, socioeconomic, and health databases. We assess the potential of iterative deduplication to generate a parent-duplicate data structure that supports address management and downstream information retrieval tasks. Our approach followed three steps: local deduplication within each database to establish parent-duplicate relationships; cross-database deduplication to identify global parent records and propagate their identifiers; and iterative deduplication of global parent records, updating assignments across databases. Graph-based modeling was used to identify shared connected components, enhancing geocoding coverage. Deduplication accuracy was evaluated through manual review of state-stratified samples (N = 2,000), with independent assessment by at least two reviewers and disagreements resolved by a third. Five data sources from Northeastern Brazil were used: the National Register of Addresses from 2010 (CNEFE 2010; N = 17,839,562), CNEFE 2022 (N = 30,545,117), the 100 Million Brazilian Cohort baseline (CadÚnico; N = 54,985,455), Mortality (SIM; N = 6,431,199), and Live Birth (SINASC; N = 17,199,287). Local deduplication substantially reduced data volume, identifying 8.25M, 531 thousand, and 583 thousand parent records for the CadÚnico, SIM, and SINASC, corresponding to reductions of 77.84%, 91.74%, and 96.61%. After iterative cross-database deduplication. The resulting database comprised 28.6M records, including 24.8M unique parent addresses and 5.9M duplicates. Average deduplication precision across all steps exceeded 0.9. Iterative deduplication is an effective strategy for enhancing data management and quality across heterogeneous datasets, facilitating large-scale spatial analysis for public health and climate change research.

Record linkage is widely used to integrate administrative health databases, but its performance depends on appropriate parameterization and decision thresholds. We propose optimizing CIDACS-RL parameters using genetic algorithms to efficiently explore the parameter space. Experiments linking Brazilian live birth (SINASC) and mortality (SIM) records using a labeled dataset reduced false positives by about 90% while maintaining high recall. Precision increased from 0.70 to 0.96 and accuracy from 0.89 to 0.99, with consistent improvements across all southern Brazilian states analyzed. These results suggest that parameter optimization can improve linkage quality and the reliability of large-scale health data integration.

The integration of administrative data for population-based studies and research faces many challenges, e.g., integrity, quality, privacy, security, and availability of this data. The governance of integrated administrative data for research purposes is context-dependent and involves the articulation of methodological, technical, ethical, legal, and social issues. Brazil, the largest country in Latin America, has an estimated population size of approximately 213 million inhabitants. Despite the country’s diverse array of national information systems to support public administration, among them health data from the universal health system, the Sistema Único de Saúde (SUS), these systems are presented and made available in an isolated manner. Present and discuss CIDACS’ approach to provide requirements and recommendations for a national health data policy for scientific research and studies aimed at producing knowledge to support evidence-based public health in Brazil. Reporting experience on a collaboration between the Center for Data and Knowledge Integration for Health (CIDACS/Fiocruz Bahia) and the Information Technology Department of the Unified Health System (DATASUS) to situate proposed strategies to support requirements and recommendations for safe and secure access to, linkage and analysis of existing health data produced by the SUS. Results and Considerations We delineate an approach encompassing a literature review on technical, ethical, legal, and societal issues related to health data usage and reuse for scientific and public health research purposes, articulated with the mapping and analysis of the national and international regulatory landscape, legal consultancy, technical visits, as well as interviews and workshops with data stakeholders.

Privacy-preserving record linkage (PPRL) indexing techniques typically organize Bloom Filters (BF) into data structures to reduce unnecessary comparisons. However, widely used solutions like Multibit Trees (MTB) often face scalability issues with large datasets or high-dimensional BFs, requiring parallel or distributed computation. This study explores the integration of the MTB algorithm into Atyimo, a publicly available Brazilian PPRL tool for merging large-scale administrative databases. We used both simulated and real-world data in our experiments to evaluate Atyimo’s effectiveness with MTB in linking routinely collected health records in Brazil. The results show that our Spark DataFrame-based solution builds robust index structures that preserve the linkage accuracy and significantly reduce execution time compared to the baseline.

BACKGROUND Linked datasets that enable longitudinal assessments are scarce in low and middle-income countries. OBJECTIVES We aimed to assess the linkage of administrative databases of live births and under-five child deaths to explore mortality and trends for preterm, small (SGA) and large for gestational age (LGA) in Mexico. METHODS We linked individual-level datasets collected by National statistics from 2008 to 2019. Linkage was performed based on agreement on birthday, sex, residential address. We used the Centre for Data and Knowledge Integration for Health software to identify the best candidate pairs based on similarity. Accuracy was assessed by calculating the area under the receiver operating characteristic curve. We evaluated completeness by comparing the number of linked records with reported deaths. We described the percentage of linked records by baseline characteristics to identify potential bias. Using the linked dataset, we calculated mortality rate ratios (RR) in neonatal, infants, and children under-five according to gestational age, birthweight, and size. RESULTS For the period 2008-2019, a total of 24,955,172 live births and 321,165 under-five deaths were available for linkage. We excluded 1,539,046 records (6.2%) with missing or implausible values. We succesfully linked 231,765 deaths (72.2%: range 57.1% in 2009 and 84.3% in 2011). The rate of neonatal mortality was higher for preterm compared with term (RR 3.83, 95% confidence interval, CI 3.78, 3.88) and for SGA compared with appropriate for gestational age (AGA) (RR 1.22 95% CI, 1.19, 1.24). Births at <28 weeks had the highest mortality (RR 35.92, 95%CI, 34.97, 36.88). LGA had no additional risk vs AGA among children under five (RR 0.92, 95%CI, 0.90, 0.93). CONCLUSIONS We demonstrated the utility of linked data to understand neonatal vulnerability and child mortality. We created a linked dataset that would be a valuable resource for future population-based research.

Background:Public health research frequently requires the integration of information from different data sources. However, errors in the records and the high computational costs involved make linking large administrative databases using record linkage (RL) methodologies a major challenge.Methods:We present Tucuxi-BLAST, a versatile tool for probabilistic RL that utilizes a DNA-encoded approach to encrypt, analyze and link massive administrative databases. Tucuxi-BLAST encodes the identification records into DNA. BLASTn algorithm is then used to align the sequences between databases. We tested and benchmarked on a simulated database containing records for 300 million individuals and also on four large administrative databases containing real data on Brazilian patients.Results:Our method was able to overcome misspellings and typographical errors in administrative databases. In processing the RL of the largest simulated dataset (200k records), the state-of-the-art method took 5 days and 7 h to perform the RL, while Tucuxi-BLAST only took 23 h. When compared with five existing RL tools applied to a gold-standard dataset from real health-related databases, Tucuxi-BLAST had the highest accuracy and speed. By repurposing genomic tools, Tucuxi-BLAST can improve data-driven medical research and provide a fast and accurate way to link individual information across several administrative databases.

on leprosy and child mortality. Other studies are now being conducted that are of utmost relevance to the health inequalities of Brazil and many low- and middle-income countries, and many research opportunities are being opened up with the linkage of a range of health outcomes.

Background: Public health research frequently requires the integration of information from different data sources.However, errors in the records and the high computational costs involved make linking large administrative databases using record linkage (RL) methodologies a major challenge. Methods:We present Tucuxi-BLAST, a versatile tool for probabilistic RL that utilizes a DNA-encoded approach to encrypt, analyze and link massive administrative databases.Tucuxi-BLAST encodes the identification records into DNA.BLASTn algorithm is then used to align the sequences between databases.We tested and benchmarked on a simulated database containing records for 300 million individuals and also on four large administrative databases containing real data on Brazilian patients. Results:Our method was able to overcome misspellings and typographical errors in administrative databases.In processing the RL of the largest simulated dataset (200k records), the state-of-the-art method took 5 days and 7 hours to perform the RL, while Tucuxi-BLAST only took 23 hours.When compared with five existing RL tools applied to a gold-standard dataset from real health-related databases, Tucuxi-BLAST had the highest accuracy and speed.By repurposing genomic tools, Tucuxi-BLAST can improve data-driven medical research and provide a fast and accurate way to link individual information across several administrative databases.

Background Record linkage is the process of identifying and combining records about the same individual from two or more different datasets. While there are many open source and commercial data linkage tools, the volume and complexity of currently available datasets for linkage pose a huge challenge; hence, designing an efficient linkage tool with reasonable accuracy and scalability is required. Methods We developed CIDACS-RL (Centre for Data and Knowledge Integration for Health – Record Linkage), a novel iterative deterministic record linkage algorithm based on a combination of indexing search and scoring algorithms (provided by Apache Lucene). We described how the algorithm works and compared its performance with four open source linkage tools (AtyImo, Febrl, FRIL and RecLink) in terms of sensitivity and positive predictive value using gold standard dataset. We also evaluated its accuracy and scalability using a case-study and its scalability and execution time using a simulated cohort in serial (single core) and multi-core (eight core) computation settings. Results Overall, CIDACS-RL algorithm had a superior performance: positive predictive value (99.93% versus AtyImo 99.30%, RecLink 99.5%, Febrl 98.86%, and FRIL 96.17%) and sensitivity (99.87% versus AtyImo 98.91%, RecLink 73.75%, Febrl 90.58%, and FRIL 74.66%). In the case study, using a ROC curve to choose the most appropriate cut-off value (0.896), the obtained metrics were: sensitivity = 92.5% (95% CI 92.07–92.99), specificity = 93.5% (95% CI 93.08–93.8) and area under the curve (AUC) = 97% (95% CI 96.97–97.35). The multi-core computation was about four times faster (150 seconds) than the serial setting (550 seconds) when using a dataset of 20 million records. Conclusion CIDACS-RL algorithm is an innovative linkage tool for huge datasets, with higher accuracy, improved scalability, and substantially shorter execution time compared to other existing linkage tools. In addition, CIDACS-RL can be deployed on standard computers without the need for high-speed processors and distributed infrastructures.

Background Research using linked routine population-based data collected for non-research purposes has increased in recent years because they are a rich and detailed source of information. The objective of this study is to present an approach to prepare and link data from administrative sources in a middle-income country, to estimate its quality and to identify potential sources of bias by comparing linked and non-linked individuals. Methods We linked two administrative datasets with data covering the period 2001 to 2015, using maternal attributes (name, age, date of birth, and municipally of residence) from Brazil: live birth information system and the 100 Million Brazilian Cohort (created using administrative records from over 114 million individuals whose families applied for social assistance via the Unified Register for Social Programmes) implementing an in house developed linkage tool CIDACS-RL. We then estimated the proportion of highly probably link and examined the characteristics of missed-matches to identify any potential source of bias. Results A total of 27,699,891 live births were submited to linkage with maternal information recorded in the baseline of the 100 Million Brazilian Cohort dataset of those, 16,447,414 (59.4%) children were found registered in the 100 Million Brazilian Cohort dataset. The proportion of highly probably link ranged from 39.3% in 2001 to 82.1% in 2014. A substantial improvement in the linkage after the introduction of maternal date of birth attribute, in 2011, was observed. Our analyses indicated a slightly higher proportion of missing data among missed matches and a higher proportion of people living in an urban area and self-declared as Caucasian among linked pairs when compared with non-linked sets. Discussion We demonstrated that CIDACS-RL is capable of performing high quality linkage even with a limited number of common attributes, using indexation as a blocking strategy in larg e routine databases from a middle-income country. However, residual records occurred more among people under worse living conditions. The results presented in this study reinforce the need of evaluating linkage quality and when necessary to take linkage error into account for the analyses of any generated dataset.

K-modes, an extension of K-means, aims to cluster categorical data by using the mode to update clusters' centroids, as well an overlap measure to define the distance between objects and centers. This work presents K-fact, an extension of K-modes, which proposes the frequency factor, a new probability-based measure to update the centroids, and the use of several similarity measures suitable to categorical values. In our validity scheme, we ran both K-modes and K-fact over real-world and synthetic datasets in order to compare them with an index from the external validity criteria. K-modes is always outperformed by some parameter set used in K-fact, especially on datasets with higher or moderate variance.

Health technology assessment (HTA) is the systematic evaluation of the properties and impacts of health technologies and interventions. In this article, we presented a discussion of HTA and its evolution in Brazil, as well as a description of secondary data sources available in Brazil with potential applications to generate evidence for HTA and policy decisions. Furthermore, we highlighted record linkage, ongoing record linkage initiatives in Brazil, and the main linkage tools developed and/or used in Brazilian data. Finally, we discussed the challenges and opportunities of using secondary data for research in the Brazilian context. In conclusion, we emphasized the availability of high quality data and an open, modern attitude toward the use of data for research and policy. This is supported by a rigorous but enabling legal framework that will allow the conduct of large-scale observational studies to evaluate clinical, economical, and social impacts of health technologies and social policies.

Data linkage refers to the process of identifying and linking records that refer to the same entity across multiple heterogeneous data sources. This method has been widely utilized across scientific domains, including public health where records from clinical, administrative, and other surveillance databases are aggregated and used for research, decision making, and assessment of public policies. When a common set of unique identifiers does not exist across sources, probabilistic linkage approaches are used to link records using a combination of attributes. These methods require a careful choice of comparison attributes as well as similarity metrics and cutoff values to decide if a given pair of records matches or not and for assessing the accuracy of the results. In large, complex datasets, linking and assessing accuracy can be challenging due to the volume and complexity of the data, the absence of a gold standard, and the challenges associated with manually reviewing a very large number of record matches. In this paper, we present AtyImo, a hybrid probabilistic linkage tool optimized for high accuracy and scalability in massive data sets. We describe the implementation details around anonymization, blocking, deterministic and probabilistic linkage, and accuracy assessment. We present results from linking a large population-based cohort of 114 million individuals in Brazil to public health and administrative databases for research. In controlled and real scenarios, we observed high accuracy of results: 93%-97% true matches. In terms of scalability, we present AtyImo's ability to link the entire cohort in less than nine days using Spark and scaling up to 20 million records in less than 12s over heterogeneous (CPU+GPU) architectures.

Record or data linkage is a technique frequently used in diverse domains to aggregate data stored in different sources that presumably pertain to the same real world entity. Deterministic (key-based) or probabilistic (rulebased) linkage methods can be used to implement data linkage, being the second approach suitable when no common link attributes exist amongst the data sources involved. Depending on the volume of data being linked, indexing (or blocking) techniques should be used to reduce the number of pairwise comparisons that need to be executed to decide if a given pair of records match or not. In this paper, we discuss a new indexing scheme, based on term-frequency counts, deployed in our data linkage tool (AtyImo). We present our algorithm design and some metrics related to accuracy and efficiency (reduction ratio achieved during blocking construction), as well a comparative analysis with a predicatebased technique also used in AtyImo. Our results shows a very high level of accuracy and reduction in terms of pairwise comparison tasks.

ABSTRACT Background and aims A cooperation Brazil-UK was set in mid-2013 aiming at to build a huge cohort comprised by individuals registered in CadastroÚnico (CADU), a socioeconomic database used in social programmes of the Brazilian government. Epidemiologists and statisticians wish to assess the impact of Bolsa Família (PBF), a conditional cash transfer programme, on the incidence of several diseases (tuberculosis, leprosy, HIV etc). The cohort must contain all individuals who received at least one payment from PBF between 2007 and 2012, which results in a 100-million records according to our preliminary analysis. These individuals must be probabilistically linked with databases from the Unified Health System (SUS), such as hospitalization (SIH), notifiable diseases (SINAN), mortality (SIM), live births (SINASC), to produce data marts (domain-specific data) to the proposed studies. Within this cooperation, our first goal was to design and evaluate probabilistic methods to routine link the cohort, PBF, and SUS outcomes. Approach We implemented two probabilistic linkage methods: a full probabilistic, based on the Dice similarity (Sorensen index) of Bloom filters; and an hybrid approach, based on rules to deterministic and probabilistic matching. We performed linkages involving CADU (2011 extraction) and SUS outcomes (SIH, SINAN, and SIM) with samples from 3 states (Sergipe, Santa Catarina and Bahia) with an increasing size (from 1,447,512 to 12,036,010). Results Using a Dice between 0.90 and 0.92, our methods retrieved more than 95% of true positive pairs amongst the linked pairs. For Sergipe, we obtained as : <23,22>, <315,300>, <32,32>, respectively for SIH, SINAN, and SIM. For Bahia: <771,593>, <2677,2626>, <208,207>. Another linkage between CADU (1,447,512 records) and SINAN (624 records), for tuberculosis in Sergipe, returned 397 (full probabilistic) and 311 (hybrid) linked pairs, being 306 and 300 true positives. Another execution considering CADU (1,988,599 records) and SINAN (2,094 records), for tuberculosis in Santa Catarina, returned 791 (full probabilistic) and 500 (hybrid) linked pairs, with 667 and 472 true positives. Linking CADU (1.685,697 records) and SIM, for mortality of children under-4, returned 18 linked pairs, all of them true positives, for a Dice between 0.90 and 0.92 and with 100% of sensitivity, specificity, and positive predictive value. Conclusion Due to the absence of gold standards, we use samples with increasing sizes and manual review when adequate. Our results are quite accurate, although obtained with an unique extraction of CADU. We are starting to run linkages with the entire cohort.