High-performance computing (HPC) is becoming mainstream for organizations, spurred by their increasing use of artificial intelligence (AI) and data analytics. A 2021 study by Insect360 Search found that 81% of organizations that use HPC said they either use AI and machine learning or plan to implement them soon. This is happening globally and contributing to global HPC spending that is expected to exceed $59.65 billion in 2025, according to Grandview Search.
Simultaneously, the intersection of HPC, AI, and analytics workflows puts pressure on system administrators to support ever more complex environments. Administrators are required to perform tedious manual configurations and reconfigurations of servers, storage, and networking when moving nodes between clusters to provide the resources needed for different workload demands. The resulting cluster sprawl consumes excessive amounts of computing resources.
The answer? For many organizations, this is a greater reliance on open source software.
Reaping the Benefits of Open Source Software and Communities
Developers in some organizations have found open source software to be an effective way to advance the HPC software stack beyond the limitations of a single vendor. Apache Ignite, Open MPI, OpenSFS, OpenFOAM, and OpenStack are examples of open source software used for HPC. Almost every major original equipment manufacturer (OEM) participates in the OpenHPC community, as well as leading independent software vendors (ISVs) HPC and top
Organizations like Arizona State University Research Computing have turned to open source software like Omniaa set of tools to automate the deployment of open source or publicly available Slurm and Kubernetes workload management, as well as libraries, frameworks, operators, services, platforms and applications .
The Omnia software stack was created to simplify and speed up the process of creating environments for mixed workloads by removing manual steps that can slow down provisioning and lead to misconfigurations. It was designed to speed up and simplify the process of deploying and managing environments for mixed workloads, including simulation, high-throughput computing, machine learning, deep learning, and data analytics. data.
Members of the open source software community contribute updates to code and documentation for feature requests and bug reports. They also provide open forums for conversations about feature ideas and potential implementation solutions. As the open-source project grows and develops, so does the technical governance committee, with representation from key contributors and stakeholders.
“We have ASU engineers on my team who work directly with Dell engineers on the Omnia team,” said Douglas Jennewein, senior director of research computing at Arizona State University (ASU). “We’re working on the code and providing feedback and guidance on what we should look at next. It’s been a very rewarding effort…We’re not only paving the way for ASU, but also paving the way for advanced computing.
ASU teams also use Open on demand, an open-source HPC portal that allows users to connect to an HPC cluster through a traditional Secure Shell Protocol (SSH) terminal or through a web interface that uses Open OnDemand. Once connected, they can upload and download files; create, edit, submit and monitor jobs; run applications; and more via a web browser in a cloud-like experience with no client software to install and configure.
Some New Features of Open Source Software for HPC
Here is a sampling of some of the latest open source software features available to HPC application developers.
- Dynamically modify a user’s environment by adding or removing directories in the PATH environment variable. This makes it easy to run specific software in specific folders without updating the PATH environment variable or rebooting. This is especially useful when third-party applications point to conflicting versions of the same libraries or objects.
- Choice of host operating system (OS) provisioned on bare metal. Application speed and accuracy are inherently affected by the host operating system installed on the compute node. This provides bare metal options of different operating systems in the lab to be able to choose the one that performs optimally at any given time and best suited for an HPC application.
- Provide low-cost block storage that natively uses the Network File System (NFS). This adds flexible scalability and is ideal for long-term persistent storage.
- Use telemetry and visualization on Red Hat Enterprise Linux. Red Hat Enterprise Linux users can take advantage of telemetry and visualization features to view power consumption, temperatures, and other operational metrics. BOSS RAID controller support. Redundant bay of independent disks (RAID) use multiple disks to distribute the I/O load and are often preferred by HPC developers.
The benefits of open source software for HPC are considerable. They include the ability to deploy faster, leverage fluid pools of resources, and integrate full lifecycle management for unified data analytics, AI, and HPC clusters.
For more information and to contribute to the Omnia Communitywhich includes Dell, Intel, University Research Environments and many more, visit the Github Omnia.
Intel® Technologies Advance Analytics
Data analytics is the key to unlocking the maximum value you can extract from your organization’s data. To create a productive, cost-effective scanning strategy that gets results, you need high-performance hardware that’s optimized to work with the software you’re using.
Modern data analytics covers a range of technologies, from dedicated analytics platforms and databases to deep learning and artificial intelligence (AI). New to analytics? Ready to evolve your analytics strategy or improve your data quality? There’s always room to grow, and Intel is ready to help. With a broad ecosystem of analytics technologies and partners, Intel accelerates the efforts of data scientists, analysts, and developers across industries. Learn more about Intel Advanced Analytics.