A transfer learning environment is characterized by not having sufficient labeled training data from the domain of interest (target domain) to build a high-performing machine learner. Transfer learning algorithms use labeled data from an alternate domain (source domain), that is similar to the target domain, to build high-performing learners. The design of a transfer learning algorithm is typically comprised of a domain adaptation step following by a learning step. The domain adaptation step attempts to align the distribution differences between the source domain and the target domain. Then, the aligned data from the domain adaptation step is used in the learning step, which is typically implemented with a traditional machine learning algorithm. Our research studies the impact of the learning step on the performance of various transfer learning algorithms. In our experiment, we use five unique domain adaptation methods coupled with seven different traditional machine learning methods to create 35 different transfer learning algorithms. We perform comparative performance analyses of the 35 transfer learning algorithms, along with the seven stand-alone traditional machine learning methods. This research will aid machine learning practitioners in the algorithm selection process for a transfer learning environment in the absence of reliable validation techniques.

In machine learning applications, there are scenarios of having no labeled training data, due to the data being rare or too expensive to obtain. In these cases, it is desirable to use readily available labeled data, that is similar to, but not the same as, the domain application of interest. Transfer learning algorithms are used to build high-performance classifiers, when the training data has different distribution characteristics from the testing data. For a transfer learning environment, it is not possible to use validation techniques (such as cross validation or data splitting) to set the desired performance of a classifier, due to the lack of labeled training data from the test domain. As a result, the area under the receiver operating characteristic curve (AUC) performance measure may not be predictive of the actual classifier performance. In an environment where validation techniques are not possible, the relationship between AUC and classification accuracy is needed to better characterize transfer learning algorithm performance. This paper provides relative performance analysis of state-of-the-art transfer learning algorithms and traditional machine learning algorithms, addressing the correlation between AUC and classification accuracy under domain class imbalance conditions with statistical analysis provided.

Transfer learning is a subclass of machine learning, which uses training data (source) drawn from a different domain than that of the testing data (target). A transfer learning environment is characterized by the unavailability of labeled data from the target domain, due to data being rare or too expensive to obtain. However, there exists abundant labeled data from a different, but similar domain. These two domains are likely to have different distribution characteristics. Transfer learning algorithms attempt to align the distribution characteristics of the source and target domains to create high-performance classifiers. This paper provides comparative performance analysis between stateof- the-art transfer learning algorithms and traditional machine learning algorithms under the domain class imbalance condition. The domain class imbalance condition is characterized by the source and target domains having different class probabilities, which can create marginal distribution differences between the source and target data. Statistical analysis is provided to show the significance of the results.

The detection of phishing websites using traditional machine learning methods has been demonstrated in previous studies. Traditional machine learning methods assume that the input feature space is the same between the training and testing data. There are scenarios in machine learning, where the available labeled training data has a different input feature space than the testing data. In cases where the input feature space between the testing and training data are different, traditional machine learning methods cannot be used. Heterogeneous transfer learning methods are used to transform the different input feature spaces between the testing and the training data into a unique and common set of input features. For our experiment, we construct numerous scenarios for the application of phishing website detection, where the features of the testing and training data are different. Our experiment starts with a baseline dataset for the detection of phishing websites. This baseline dataset is used to create separate training and testing datasets by splitting the features, such that the features in the training and testing data are mutually exclusive. Then, a heterogeneous transfer learning technique called Canonical Correlation Analysis is used to align the input feature space between the training and testing data. The feature aligned training and testing data is used with various traditional machine learning methods and homogeneous transfer learning methods to predict phishing websites. The performance results of the different scenarios and algorithms are reported and analyzed.

In the field of supervised machine learning, a transfer learning environment is defined as the training data having different distribution characteristics than the testing data. This is due to the lack of available labeled data for the domain of interest, which prompts an alternate domain to be used as the training data. Because there is insufficient labeled data from the domain of interest, validation techniques cannot be reliably used for the algorithm selection process in a transfer learning environment. A transfer learning algorithm is typically comprised of a domain adaptation step followed by a learning step. The learning step is usually implemented using a traditional machine learning algorithm. In this paper, we examine and analyze the impact that the traditional machine learning algorithm (the learning step) has on the overall performance of a transfer learning algorithm. Using the transfer learning test framework, we test five state-of-the-art transfer learning algorithms coupled with seven different traditional learning algorithms for a total of 35 unique transfer learning algorithms. For our experiment, no labeled data from the domain of interest is available for the training process. Since validation techniques cannot be reliably used for the algorithm selection process in a transfer learning environment, it is important for machine learning researchers and practitioners to understand the impact of a traditional machine learner on the overall performance of a transfer learning algorithm.

A transfer learning environment is characterized by a machine learning algorithm being trained with data from one domain (the source domain) and being tested on data from a different domain (the target domain). In a transfer learning scenario, the class probability of the source domain may be different from the class probability of the target domain, which is referred to as "domain class imbalance". Domain class imbalance is different from "class imbalance". Class imbalance refers to the condition of a single domain having unequal class probabilities. In traditional machine learning, the training and testing data are drawn from a single domain. The effects of class imbalance in traditional machine learning are well studied; however, the issue of domain class imbalance in the field of transfer learning has received little research attention. This paper provides a comparative performance test of state-of-the-art transfer learning algorithms, using a wide-range of domain class imbalance combinations. A detailed discussion on the relative performances of the different algorithms, with statistical validation, is presented for the different domain class imbalance scenarios.

Most works covering the topic of transfer learning propose an algorithm to solve a given domain adaptation problem, then test the algorithm using real-world datasets. A test with a real-world dataset represents a single transfer learning test condition, which partially measures an algorithm's performance. Previous research has placed little emphasis on developing a comprehensive and uniform test for transfer learning algorithms. With this in mind, a test framework is proposed, comprising of distortion profiles which define a comprehensive test suite. The unique contribution of this paper is the definition of a test framework that measures a more complete profile of a transfer learning algorithm's capability, facilitating the identification of relative poor and good performance areas. As a proof of concept, the test framework is used to test a homogeneous transfer learning algorithm. The test framework will be the basis for a number of future applications.

Previous research focusing on the evaluation of transfer learning algorithms has predominantly used real-world datasets to measure an algorithm's performance. A test with a real-world dataset exposes an algorithm to a single instance of distribution difference between the training (source) and test (target) datasets. These previous works have not measured performance over a wide-range of source and target distribution differences. We propose to use a test framework that creates many source and target datasets from a single base dataset, representing a diverse-range of distribution differences. These datasets will be used as a stress test to measure an algorithm's performance. The stress test process will measure and compare different transfer learning algorithms and traditional learning algorithms. The unique contributions of this paper, with respect to transfer learning, are defining a test framework, defining multiple distortion profiles, defining a stress test suite, and the evaluation and comparison of different transfer learning and traditional machine learning algorithms over a wide-range of distributions.

Machine learning and data mining techniques have been used in numerous real-world applications. An assumption of traditional machine learning methodologies is the training data and testing data are taken from the same domain, such that the input feature space and data distribution characteristics are the same. However, in some real-world machine learning scenarios, this assumption does not hold. There are cases where training data is expensive or difficult to collect. Therefore, there is a need to create high-performance learners trained with more easily obtained data from different domains. This methodology is referred to as transfer learning. This survey paper formally defines transfer learning, presents information on current solutions, and reviews applications applied to transfer learning. Lastly, there is information listed on software downloads for various transfer learning solutions and a discussion of possible future research work. The transfer learning solutions surveyed are independent of data size and can be applied to big data environments.