We have endeavored to classify the commonly seen data movement needs, as observed in data-intensive institutions (both commercial and non-profit), into four categories. Knowing how to map a data movement task into one of the four categories helps select proper data mover tools. For each category, how the data storage is involved, high-level examples and the nature of typical solutions are described. Finally, some general remarks are provided to help further orient readers new to this field - the 4th IT pillar.

PingER (Ping End-to-End Reporting) is a tool developed by SLAC National Accelerator Laboratory for the purpose of Internet End-to-end Performance Monitoring (IEPM). The aim of this research work is to develop a mobile application for Android mobile devices using Firebase for storing the data, obtained from pinging the beacons, and authenticating the users. The Measuring Agent (MA) pings the beacon list, the data obtained is formatted with the help of a Regular Expression library before being pushed to Firebase. In addition, the location of the MA, latitude and longitude, is also tracked with the help of Google's Geolocation API. This data is also stored in the database.

This research work has studied the internet network performance in India and Pakistan over the years. The comparison is made on various factors which were prevailing in the country during that time, and how it affected the network in those areas. PingER is acronym for Ping end-to-end reporting. This project was started in the year 1995 and aimed at measuring the internet performance over various reasons. It uses a simple ping command to find the Round-trip Travel Time(RTT) between two nodes of the internet.

PingER was developed by the Stanford Linear Accelerator Center's (SLAC) National Accelerator Laboratory as a tool for Internet End-to-end Performance Monitoring (IEPM). It monitors over 700 sites worldwide, and aims to measure the round-trip time, loss jitter etc. for packets travelling between nodes on the internet. The PingER MeasurementAgent can be deployed on servers running Linux, however these servers have limitations. The fixed-line servers currently in use are not mobile and require a continuous power source. The extension of the PingER project to the Android ecosystem brings advantages like greater power efficiency, ease of installation, maintenance, and better affordability to the table. The Android application is planned to supplement the existing PingER Measurement Agent Linux application set up at about 40 locations around the globe.

This paper aims towards analyzing the changes in the Internet Performance in India over a span of 3 years by using the clustering algorithm on PingER data. The paper analyzed the enhanced performance of the internet after major changes in the Telecom industry in India in the year 2016 and some insights into the level of improvement. PingER which is short for Ping End-to-end Reporting is a project started with an aim to monitor end-to-end performance of between Internet hosts by SLAC National Accelerator Laboratory, Stanford, California. And over the last decade or so they have collected a huge amount of data which is stored as space separated flat files along with sophisticated methods in order to enable searching for data fast.

This paper presents a case study of the 10-Gigabit Ethernet (10GbE) adapter from Intel R . Specifically, with appropriate optimizations to the configurations of the 10GbE adapter and TCP, we demonstrate that the 10GbE adapter can perform well in local-area, storage-area, system-area, and wide-area networks. For local-area, storage-area, and system-area networks in support of networks of workstations, network-attached storage, and clusters, respectively, we can achieve over 7-Gb/s end-to-end throughput and 12-µs end-to-end latency between applications running on Linux-based PCs. For the wide-area network in support of grids, we broke the recently-set Internet2 Land Speed Record by 2.5 times by sustaining an end-to-end TCP/IP throughput of 2.38 Gb/s between Sunnyvale, California and Geneva, Switzerland (i.e., 10,037 kilometers) to move over a terabyte of data in less than an hour. Thus, the above results indicate that 10GbE may be a cost-effective solution across a multitude of computing environments.

PingER was developed by Stanford Linear Accelerated Center as a tool for internet End-to-End monitoring. The PingER measurement tool can be deployed on host servers running UNIX based systems, however these servers do have their limitations. The Extension of the PingER project to the Android ecosystem would have the advantages like greater power efficiency, ease of installation, maintenance, reduction in storage dimensions and better affordability. In order to develop this extension, a monitoring site was set up at Amity University, India for understanding the data collection process, which was further used to develop a mirror implementation on Android through native Android development tools and strategies. In this research work, two models have been proposed that explore the feasibility and implementation of the concept of extension of PingER on Android. We compared the two models and chose to implement a model that used the native Android tools rather than mimicking the original Perl based implementation.

PingER was developed by Stanford Linear Accelerated Center as a tool for internet End-to-End monitoring. The PingER measurement tool can be deployed on host servers running UNIX based systems, however these servers do have their limitations. The Extension of the PingER project to the Android ecosystem would have the advantages like greater power efficiency, ease of installation, maintenance, reduction in storage dimensions and better affordability. In order to develop this extension, a monitoring site was set up at Amity University, India for understanding the data collection process, which was further used to develop a mirror implementation on Android through native Android development tools and strategies. In this research work, two models have been proposed that explore the feasibility and implementation of the concept of extension of Pin gER on Android. We compared the two models and chose to implement a model that used the native Android tools rather than mimicking the original Perl based implementation

With the huge amount of data continuously accumulated and shared by individuals and organizations, it has become necessary to meet the emerging processing and information retrieval requirements associated with these large volumes of data. This could be achieved by indexing the data sets and reducing heavy computational overhead accustomed to most current indexing strategies during processing of very large amounts of data sets. This study proposes a novel Indexing strategy called Big Data INDexing Strategy (BIND), using a concept of high performance parallel computing. BIND supports parallel distribution of data and performs processing in a MapReduce fashion. To develop the BIND strategy, Ian Foster's task-scheduling concept for parallel processing is applied. The proposed indexing strategy was first tested on a 2-node cluster environment where varying sizes of datasets were used to note if the performance improves or declines as the size of the data increases. Subsequently, it was tested on a 3-node cluster to note the performance when the number of computation resources are increased. The results demonstrates that BIND minimizes the processing and query time as compared to the current strategy. The findings have significant implication in efficiently managing Big Data and facilitating data processing and information retrieval for users and organizations that manage Big Data.

Nowadays, the Internet has turned into a crucial piece of our cutting edge society. It is a stage of exploration, financial development, democratic participation and speech. The operations of the Internet have prompted a huge development and collection of information known as Big Data. Therefore, it is important to monitor and measure the Quality of Service (QoS) of Internet traffic. The SLAC National Accelerator Laboratory started the PingER project in 1995 to measure the End-to-End Internet performance history of servers and routers worldwide. The project involves measurements of the 700 monitored sites in over 160 countries. PingER Monitoring Agents (MAs) ping a list of monitored sites after every 30 minutes to obtain Round Trip Time (RTT) values revealing interesting information about Internet performance (e.g., RTT, jitter, packet loss and unreachability) major events (e.g., fiber cuts, earthquakes, and social upheavals). Thus, the project has collected a vast amount of historical Internet Performance data worldwide since 1995. Currently, the data is stored in flat text files, making it difficult to analyze collectively. In addition, this simplistic format limits the analytical potential of this data. In this paper, we propose an approach to process, store, analyze and visualize PingER data. A Data warehouse is created which combines Hadoop Big Data techniques. The data are processed by using Sci-cumulus MR workflow, stored in HDFS, analyzed by Impala queries and visualized by using Google API's. This approach makes PingER data more accessible and enhances its potential contribution to ongoing research and application development.

The PingER project was started by the SLAC National Accelerator Laboratory, Stanford, California for the purpose of monitoring end to end network performance. For the last eighteen years PingER has generated an enormous amount of data that has been stored in space separated files. However due to the difficulties faced in retrieving data efficiently, it has been proposed that all the data be put into the form of RDF triples. Interpreting and analyzing such large volumes of data becomes a primary concern. By making using of clustering algorithms new and interesting patterns can be observed in the data sets. Outlier analysis can be performed giving insight to the exceptions occurring in the dataset and analyzing the probable causes of such. Patterns could be observed based on the country to which the data belongs and comparisons can be drawn between the patterns between the different countries.

Geolocation is a leading Internet trend due to its significance in the commercial and services realm. So far, the network infrastructural dependency of IP geolocation has restricted its use to North America and Europe. However, the real potential of geolocation is in being a global rather than just a regional application. Since the last decade world-wide Internet usage and performance have increased exponentially. This directly urges the need to re-evaluate the performance of geolocation in more of the world. In this paper we first analyze various parameters in different regions. These are parameters that we believe affect geolocation performance and include: average distance and delay between landmarks and target, density of landmarks around a target, correlation between delay and distance around a target and the stability of proportionality between round trip times (RTT) and distance (referred to as alpha, α) around the target. We then test the performance of several well known geolocation techniques in different regions. These techniques include: Shortest Ping, TULIP, Constraint Based Geolocation with Speed of Internet, Constraint Based Geolocation and Topology Based Geolocation. We then quantify the estimates of each technique in relation to the values of the aforementioned parameters. Finally we analyze the impact of selecting a geometric technique (trilateration, multilateration and apollonius) on the performance of a geolocation solution.

Adoption of ICT and Internet access is expanding in Africa. But, due to more rapid growth elsewhere, a digital divide exists between Africa and the rest of the world. Recent investment in fibre cable networks in and around Africa opens the possibility of slowing the growth of this divide, and reducing it. Education and research in many sub-Saharan countries still suffer serious Internet deficiencies, despite progress in the development of National Research and Education Networks (NRENs). It is widely acknowledged in policy statements from the African Union and others that strength in this sector is key to meeting and sustaining Millennium Development Goals. Countries with advanced cyber-capabilities proclaim the benefits to rich and poor alike arising from the information revolution. This is but a dream for many scientists in African institutions. As world science becomes Internet-dependent, they become increasingly isolated. eGY-Africa is a bottom-up initiative by African scientists and collaborators to address this problem by a campaign of advocacy for better institutional capabilities. The present status of Internet services, problems, and plans are being mapped via direct measurement of Internet performance via the PingER Project and a questionnaire-based survey. Information is gathered on relevant policy statements and initiatives and used to argue for better Internet facilities. Action groups of concerned scientists are being formed at the national and regional levels in Africa and opinion in the international science community is being mobilized. Finally, eGY-Africa is engaging with other programs, initiatives, and bodies with the goal of reducing the digital divide - either directly or indirectly as a step towards other national development goals. The expectation is that informed opinion from the scientific community at the institutional, national, and international levels can be used to influence the decision makers and donors who are in a position to enabl e better Internet capabilities.

The Energy Sciences Network (ESnet) is the primary provider of network connectivity for the US Department of Energy Office of Science, the single largest supporter of basic research in the physical sciences in the United States. In support of the Office of Science programs, ESnet regularly updates and refreshes its understanding of the networking requirements of the instruments, facilities, scientists, and science programs that it serves. This focus has helped ESnet to be a highly successful enabler of scientific discovery for over 20 years. In August 2009 ESnet and the Office of High Energy Physics (HEP), of the DOE Office of Science, organized a workshop to characterize the networking requirements of the programs funded by HEP. The International HEP community has been a leader in data intensive science from the beginning. HEP data sets have historically been the largest of all scientific data sets, and the communty of interest the most distributed. The HEP community was also the first to embrace Grid technologies. The requirements identified at the workshop are summarized below, and described in more detail in the case studies and the Findings section: (1) There will be more LHC Tier-3 sites than orginally thought, and likely more Tier-2 to Tier-2 traffic than was envisioned. It it not yet known what the impact of this will be on ESnet, but we will need to keep an eye on this traffic. (2) The LHC Tier-1 sites (BNL and FNAL) predict the need for 40-50 Gbps of data movement capacity in 2-5 years, and 100-200 Gbps in 5-10 years for HEP program related traffic. Other key HEP sites include LHC Tier-2 and Tier-3 sites, many of which are located at universities. To support the LHC, ESnet must continue its collaborations with university and international networks. (3) While in all cases the deployed 'raw' network bandwidth must exceed the user requirements in order to meet the data transfer and reliability requirements, network engineering for trans-Atlantic connectivity is more complex than network engineering for intra-US connectivity. This is because transoceanic circuits have lower reliability and longer repair times when compared with land-based circuits. Therefore, trans-Atlantic connectivity requires greater deployed bandwidth and diversity to ensure reliability and service continuity of the user-level required data transfer rates. (4) Trans-Atlantic traffic load and patterns must be monitored, and projections adjusted if necessary. There is currently a shutdown planned for the LHC in 2012 that may affect projections of trans-Atlantic bandwidth requirements. (5) There is a significant need for network tuning and troubleshooting during the establishment of new LHC Tier-2 and Tier-3 facilities. ESnet will work with the HEP community to help new sites effectively use the network. (6) SLAC is building the CCD camera for the LSST. This project will require significant bandwidth (up to 30Gbps) to NCSA over the next few years. (7) The accelerator modeling program at SLAC could require the movement of 1PB simulation data sets from the Leadership Computing Facilities at Argonne and Oak Ridge to SLAC. The data sets would need to be moved overnight, and moving 1PB in eight hours requires more than 300Gbps of throughput. This requirement is dependent on the deployment of analysis capabilities at SLAC, and is about five years away. (8) It is difficult to achieve high data transfer throughput to sites in China. Projects that need to transfer data in or out of China are encouraged to deploy test and measurement infrastructure (e.g. perfSONAR) and allow time for performance tuning.

The Digital Divide prevents Africa from taking advantages of new information technologies. One of the most urgent priorities is to bring the Internet in African Universities, Research, and Learning Centers to the level of other regions of the world. eGY-Africa, and the Sharing Knowledge Foundation are two bottom-up initiatives by scientists to secure better cyber-infrastructure and Internet facilities in Africa. Recommendations by the present scientific communities are being formulated at national, regional and international levels. The Internet capabilities are well documented at country level overall, but this is not the case at the University level. The snapshot of the Internet status in universities in 17 African countries, obtained by a questionnaire survey, is consistent with measures of Internet penetration in the corresponding country. The monitoring of Internet performance has been proposed to those African universities to provide an information base for arguing the need to improve the coverage for Africa. A pilot program is recommended that will start scientific collaboration with Europe in western Africa using ICT. The program will lay the foundations for the arrival of new technologies like Grids.

The Digital Divide prevents Africa from taking advantages of new information technologies. One of the most urgent priorities is to bring the Internet in African Universities, Research, and Learning Centres to the level of other regions of the world. eGY-Africa, and the Sharing Knowledge Foundation are two bottom-up initiatives by scientists to secure better cyber-infrastructure and Internet facilities in Africa. Recommendations by the present scientific communities are being formulated at national, regional and international levels. The Internet capabilities are well documented at country level overall, but this is not the case at the University level. The snapshot of the Internet status in universities in 17 African countries, obtained by a questionnaire survey, is consistent with measures of Internet penetration in the corresponding country. The monitoring of Internet performance has been proposed to those African universities to provide an information base for arguing the need to improve the coverage for Africa. A pilot programme is recommended that will start scientific collaboration with Europe in western Africa using ICT. The programme will lay the foundations for the arrival of new technologies like Grids.

Adoption of information and communication technologies and access to the Internet is expanding in Africa, but because of the rapid growth elsewhere, a Digital Divide between Africa and the rest of the world exists, and the gap is growing. In many sub-Saharan African countries, education and research sector suffer some of the worst deficiencies in access to the Internet, despite progress in development of NRENs National Research and Education (cyber) Networks. By contrast, it is widely acknowledged in policy statements from the African Union, the UN, and others that strength in this very sector provides the key to meeting and sustaining Millennium Development Goals. Developed countries with effective cyber-capabilities proclaim the benefits to rich and poor alike arising from the Information Revolution. This is but a dream for many scientists in African institutions. As the world of science becomes increasingly Internet-dependent, so they become increasingly isolated. eGY-Africa is a bottom-up initiative by African scientists and their collaborators to try to reduce this Digital Divide by a campaign of advocacy for better institutional facilities. Four approaches are being taken. The present status of Internet services, problems, and plans are being mapped via a combination of direct measurement of Internet performance (the PingER Project) and a questionnaire-based survey. Information is being gathered on policy statements and initiatives aimed at reducing the Digital Divide, which can be used for arguing the case for better Internet facilities. Groups of concerned scientists are being formed at the national, regional levels in Africa, building on existing networks as much as possible. Opinion in the international science community is being mobilized. Finally, and perhaps most important of all, eGY-Africa is seeking to engage with the many other programs, initiatives, and bodies that share the goal of reducing the Digital Divide either as a direct policy objective, or indirectly as a means to an end, such as the development of an indigenous capability in science and technology for national development. The expectation is that informed opinion from the scientific community at the institutional, national, and international levels can be used to influence the decision makers and donors who are in a position to deliver better Internet capabilities.

The future of Computing in High Energy Physics (HEP) applications depends on both the Network and Grid infrastructure. South Asian countries such as India and Pakistan are making significant progress by building clusters as well as improving their network infrastructure However to facilitate the use of these resources, they need to manage the issues of network connectivity to be among the leading participants in Computing for HEP experiments. In this paper we classify the connectivity for academic and research institutions of South Asia. The quantitative measurements are carried out using the PingER methodology; an approach that induces minimal ICMP traffic to gather active end-to-end network statistics. The PingER project has been measuring the Internet performance for the last decade. Currently the measurement infrastructure comprises of over 700 hosts in more than 130 countries which collectively represents approximately 99% of the world's Internet-connected population. Thus, we are well positioned to characterize the world's connectivity. Here we present the current state of the National Research and Educational Networks (NRENs) and Grid Infrastructure in the South Asian countries and identify the areas of concern. We also present comparisons between South Asia and other developing as well as developed regions. We show that there is a strong correlation between the Network performance and several Human Development indices.