This study proposes strategies to reduce the financial cost of using cloud clusters, through Amazon Web Services (AWS) ParallelCluster, to run the weather forecast model Brazilian developments on the Regional Atmospheric Modeling System (BRAMS). We developed an instance selection algorithm that obtains and compares the costs of various instance types in different regions and markets, recommending those with the lowest costs. If the suggested instance is a Spot instance and is revoked by the cloud provider, the proposed strategy resumes the application execution from a pre-recorded checkpoint by rescheduling it on On-Demand instances. This study also presents a detailed analysis of BRAMS execution across various instance architectures and proposes a novel three-queue architecture for managing BRAMS execution on On-Demand and Spot instances within AWS ParallelCluster. The results obtained from small and large spatial domains executed in AWS ParallelCluster using the proposed strategies show that adopting a cloud cluster is a promising alternative for this type of High-Performance Computing application, compared with execution on a supercomputer.

High Performance Computing (HPC) is increasingly moving to the cloud, where Spot instances offer significant cost savings compared to On-demand price models. However, selecting instances remains challenging due to a large variety of instance types and costs across different regions and providers. This work investigates strategies for allocating a set of Spot instances (a Spot fleet), simultaneously, in commercial clouds, focusing exclusively on raw pricing. AWS and Azure are first analyzed independently to highlight their particularities in pricing models and instance offerings. Building on this analysis, a multi-cloud solution (MC-Fleet) is proposed to support cost-efficient selection of a set of instances across providers. Extensive experiments demonstrate MC-Fleet effectiveness, achieving cost savings of 23% against a fully provisioned AWS single region us-east-1 and up to 47% over an incomplete provisioned sa-east-1.

Traditional static computational resource allocation in cloud or on-premises clusters often results in inefficient overprovisioning. Users frequently lack precise knowledge of the memory and processors their applications require, leading them to request excess resources. This causes wasted capacity, higher costs, and, in shared environments, longer queue waiting times. Dynamic resource allocation through autoscaling addresses this issue by adjusting resources at runtime. Kubernetes, a widely used container orchestration platform, supports autoscaling via Horizontal and Vertical Pod Autoscalers. However, its default restart-based scaling can disrupt stateful, long-running workloads without checkpointing. This work leverages Kubernetes' new in-place scaling, which resizes resources without restarts, to propose the Dynamic Resizing Strategy (DRS), a novel autoscaling approach that proactively manages contention by temporarily throttling co-located pods to prioritize a bursting application. We evaluate it with NAS Parallel Benchmarks and synthetic workloads in co-execution scenarios, showing improved efficiency and stability, increasing success rates and reducing global average wait time by over 18% compared to the Burstable QoS class.

Modeling petroleum field behavior provides crucial knowledge for risk quantification regarding extraction prospects. Since their processing requires significant computational and storage capabilities, oil companies run reservoir simulation jobs on high-performance computing clusters managed by job managers, for example, Slurm. In this scenario, efficiently using machine learning algorithms to predict the runtime of incoming jobs can improve the effectiveness of cluster resources, such as enhancing the resource usage rate and reducing the jobs queue time. This work analyses diverse machine learning-based predictors built from a real-world Slurm jobs log from Petrobras, a globally renowned Brazilian energy company. Furthermore, a two-step estimation strategy that predicts the duration time interval of reservoir simulation jobs is proposed and assessed, indicating that such estimated runtimes, when employed by job managers in their scheduling decisions, can positively impact the throughput of a real-world batch system.

High Performance Computing (HPC) is increasingly transitioning to the cloud, although cost remains a significant barrier. While On-demand instances and Committed Use Discounts provide predictable pricing, the Spot market offers an appealing opportunity for substantial cost savings - though it does not guarantee resource availability. Effectively managing multiple instances for parallel HPC applications is essential. Services like AWS EC2 Fleet or Spot Fleet help with this, but they come with limitations, notably being constrained to a single region. Furthermore, simply selecting the lowest-priced instance often leads to suboptimal performance and, surprisingly, higher overall costs. To truly economize, a more sophisticated approach is required: one that involves profiling applications and instances to understand their intricate cost-performance trade-offs. The most cost-effective execution prioritizes instances that strike a better balance between the price per hour being charged and the actual performance they offer the application, even if its current Spot price is not the lowest available. This paper addresses these challenges by conducting a thorough analysis of existing EC2 (Spot) Fleet policies and introducing Fleet-MR, a novel multi-region instance selection framework. Fleet-MR aims to improve execution times or reduce costs, and its effectiveness is validated through experimental evaluations.

Web scraping is a widely used technique for decision-making, collecting, and structuring public data from the internet. As the volume of data continues to grow, the need for more efficient methods of data extraction becomes crucial. This article introduces a novel web scraping framework that utilizes Burstable virtual machines (VMs) on Amazon Web Services with the objective of reducing the monetary cost of execution while ensuring compliance with service level agreements (SLAs). To achieve this, the framework utilizes a combination of fixed and temporary Burstable VMs in a mixed cluster, which can be elastically scaled up to fulfill the SLA and scaled down to minimize monetary costs. Two strategies for handling VM allocation are proposed and evaluated: (i) a queue and SLA-based strategy that employs queue size information and SLA criteria to determine the required number of VMs for the current scraping requests, and (ii) a credit-based strategy that incorporates information about Burstable VM credits to effectively manage instance creation and termination. Experimental tests show that the proposed framework meets the defined SLA while achieving cost reductions of up to 74% compared to an approach that executes on fixed-size clusters of Burstable instances.

Cloud computing platforms are essential for a wide range of applications, including High-Performance Computing (HPC) and artificial intelligence. However, the performance variability of virtual machines (VMs) in these shared environments presents significant challenges. This paper provides an extensive month-long analysis of the performance variability of C family VMs on Amazon Web Services (AWS) across two regions (us-east-1 and sa-east-1), various instance generations, and market types. Our findings indicate that Graviton processors (c6g.12xlarge and c7g.12xlarge) exhibit minimal performance variability and high cost-effectiveness, with the c7g.12xlarge instance, in particular, offering significantly reduced execution times and lower costs. Intel and AMD instances, while showing performance improvements from generation c6 to c7, exhibited up to 20% variability.

Federated Learning systems allow training machine learning models distributed across multiple clients, each one using private local data. Iteratively, the clients send their training contributions to a server, which performs a merge to produce an enhanced global model. Due to resource and data heterogeneity, client selection is crucial to optimize the system efficiency and improve the global model generalization. Selecting more clients is likely to increase the overall energy consumption, while a small number of clients may decline the performance of the trained model or require longer training time. We propose two time- and energy-aware client selection algorithms, MEC and ECMTC, which are proven regarding their optimality and evaluated against state-of-the-art algorithms on an extensive series of experiments in both simulation and HPC platform scenarios. The results indicate the benefits of jointly optimizing the time and energy consumption metrics using our proposals.

This paper presents the framework SIM@ ClOUD that optimizes cost-related resource allocation decisions for simulation jobs in cloud environments. SIM@ CLOUD offers comprehensive management of simulations throughout their execution life-cycle in the cloud, including the selection of Virtual Machine (VM) types across different regions and markets. By leveraging Spot VMs and application checkpointing, the framework transparently reduces the monetary costs associated with the execution without client intervention. Historical data analysis enables the prediction of simulation execution times, which is refined further by a dynamic predictor for adaptive VM selection. SIM@ CLOUD is being deployed in an industrial setting and employs a cachebased storage solution to improve access latency to in-house data by VMs located in geographically distinct regions. An evaluation carried out on AWS EC2, using real oil reservoir simulations, demonstrates the effectiveness of the framework.

Federated Learning (FL) is a distributed technique that allows multiple users to train models collaboratively without accessing private and sensitive data. Iteratively, each user trains a “local” model in a specific machine consuming private data and then sends the model updates to a server for their fusion into a centralized one. Although FL represents a step forward, the training duration in each iteration directly depends on the several configurations set, e.g., hyperparameters. Analyzing hyperparameters during the FL workflow allows for dynamic fine-tuning that can improve the performance of FL regarding training time and quality of results. However, due to its exploratory nature, the user may lose track of which configurations have been used to train the model with the best accuracy if the choices are not correctly registered. Provenance is the natural choice to represent data derivation traces to help hyperparameters fine-tuning by providing a global data-oriented picture of the FL workflow. Yet, the existing FL frameworks do not provide dynamic fine-tuning nor support provenance capturing. Therefore, this paper introduces an FL framework named Flower-PROV that uses provenance data for tracking configurations and evaluation metrics during the FL execution to allow for dynamic fine-tuning of hyperparameters, thus saving training time. We show a use case with Cross-Silo FL where Flower-PROV dynamic fine-tuning reduced the FL training time up to 94.24% when compared with the fine-tuning using grid-search.

Este artigo tem como objetivo avaliar o impacto da utilização do tempo de execução de jobs, previstos pelos modelos de aprendizado de máquina J48, Linear Regression e Random Forest, no escalonamento em sistemas computacionais de alto desempenho. Os tempos previstos por esses modelos foram usados pela política SJF (Shortest Job First) em uma simulação de escalonamento baseada em um conjunto de milhares de jobs de aplicações reais de alto desempenho que foram executados em um ambiente de produção da Petrobras. As métricas de desempenho de escalonamento throughput e tempo médio de espera foram examinadas adicionalmente às tradicionais métricas teóricas de modelos preditores. Demonstramos que o efeito prático das predições pode divergir do resultado teórico dos preditores, destacando a importância de avaliações empíricas para a otimização do escalonamento de jobs.

O Aprendizado Federado (AF) é uma técnica descentralizada que possibilita que vários usuários treinem modelos de Aprendizado de Máquina de forma colaborativa, sem precisar acessar dados privados ou sensíveis. O treinamento de um modelo pode exigir várias iterações, e a duração de cada iteração depende diretamente das configurações definidas, como os valores dos hiper-parâmetros. Analisar os hiperparâmetros durante o treinamento permite uma melhor compreensão do modelo treinado e abre oportunidades para melhorias. Este artigo apresenta o arcabouço Flower-PROV que tem como objetivo capturar dados de proveniência durante o treinamento para rastrear configurações e métricas de avaliação, possibilitando a análise dos hiperparâmetros em tempo real. O banco de dados de proveniência segue a recomendação W3C PROV, facilitando a comparação, explicação e reprodução desses experimentos.

Cloud providers offer several types of Virtual Machines (VMs) in diverse markets, with different guarantees in terms of availability and reliability. Among them, the most popular market models are the on-demand and the spot. On-demand VMs are allocated for a fixed cost per time, and their availability is ensured during the whole execution. On the other hand, in the spot market, VMs are offered with a huge discount, but their availability fluctuates according to cloud's current demand that can terminate or hibernate a spot VM at any time. Furthermore, to cope with workload variations, cloud providers have also introduced the concept of burstable VMs, which can burst up their CPU performance during a limited period of time. In this work, we present the Burst Hibernation-Aware Dynamic Scheduler (Burst-HADS), a framework that executes Bag-of-Tasks applications with deadline constraints by exploiting both spot and on-demand burstable VMs, aiming at minimizing both the monetary cost and the execution time. Performance results on Amazon EC2 show that Burst-HADS reduces the monetary cost and meets the application deadline even in spot hibernation scenarios, when compared to other approaches from the related literature which uses only spot and non-burstable on-demand instances.

Petroleum reservoir simulation uses computer models to predict fluid flow in porous media, aiding to forecast oil production. Engineers execute numerous simulations with different geological realizations to refine the accuracy of the model. These experiments require considerable computational resources, which are not always available within the on-premises infrastructure. Commercial public cloud platforms can offer many advantages, such as virtually unlimited scalability and pay-per-use pricing. This paper introduces MScheduler, a meta scheduler framework for reservoir simulations at Petrobras, a Brazilian energy company. It efficiently executes jobs in the cloud, utilizing spot Virtual Machines (VMs) to reduce costs and ensure job completion even with VM termination. Contributions include a novel methodology for reservoir simulation checkpointing, a cost-based scheduler, and an analysis of the strategy using real production jobs from Petrobras.

Provedores de nuvens públicas oferecem uma grande variedade de serviços e recursos computacionais. A utilização de serviços mais especializados e automaticamente gerenciados pelos provedores, tal como o modelo de Plataforma como Serviço (PaaS), é uma das causas para o aumento de custos monetários cobrados aos usuários. Neste trabalho apresentamos uma ferramenta de implantação e execução de clusters Spark que utiliza o modelo de Infraestrutura como Serviço (IaaS). Os resultados obtidos a partir de diversos casos de uso apontam que a ferramenta proposta, comparada ao PaaS, é capaz de reduzir razoavelmente os custos de execução de aplicações Spark na nuvem.

Federated Learning (FL) is a distributed Machine Learning (ML) technique that can benefit from cloud environments while preserving data privacy. We propose Multi-FedLS, a framework that manages multi-cloud resources, reducing execution time and financial costs of Cross-Silo Federated Learning applications by using preemptible VMs, cheaper than on-demand ones but that can be revoked at any time. Our framework encloses four modules: Pre-Scheduling, Initial Mapping, Fault Tolerance, and Dynamic Scheduler. This paper extends our previous work \cite{brum2022sbac} by formally describing the Multi-FedLS resource manager framework and its modules. Experiments were conducted with three Cross-Silo FL applications on CloudLab and a proof-of-concept confirms that Multi-FedLS can be executed on a multi-cloud composed by AWS and GCP, two commercial cloud providers. Results show that the problem of executing Cross-Silo FL applications in multi-cloud environments with preemptible VMs can be efficiently resolved using a mathematical formulation, fault tolerance techniques, and a simple heuristic to choose a new VM in case of revocation.

A utilização de nuvens computacionais em computação de alto desempenho tem levantado questões sobre a vantagem desse tipo de sistema em relação a sistemas tradicionais (on premise). Porém, comparar esses dois sistemas não é trivial, devido a diversos fatores como a quantidade de soluções que os provedores de nuvens oferecem e também do comportamento da aplicação. Este estudo visou analisar o modelo numérico de previsão de tempo e clima BRAMS nos dois sistemas. Observou-se que para um estudo de caso pequeno, a aplicação possui desempenho semelhantemente nos sistemas. Também foi analisado o custo da execução da aplicação em diferentes mercados ofertados pela AWS, que para o problema utilizado, é aconselhado o uso do mercado spot.

Este trabalho teve como objetivo analisar o desempenho do modelo de previsão numérica do tempo BRAMS em execução em um cluster AWS criado com o AWS ParallelCluster em diferentes mercados de instâncias, comparandoo com a execução no supercomputador Santos Dumont. Foi proposta uma metodologia para executar uma versão tolerante a falhas do BRAMS no mercado de Spot, onde as instâncias podem ser revogadas, embora ofereçam custos mais baixos. Os tempos de execução na nuvem foram satisfatórios quando comparados ao Santos Dumont. Em geral, a solução Spot reduziu o custo financeiro quando comparado ao uso de instâncias regulares On-Demand. Apenas em um cenário com muitas revogações, o que consequentemente aumenta o tempo de execução e o custo, a opção de usar o mercado On-Demand foi mais adequada.