Cloud, HPC And Open Technologies Converge To Fuel Research, Innovation

With a mission-based focus on scientific research, universities and research institutes have long been at the forefront of technology innovation that embraces open standards and ease of collaboration. By nature, the academy well understands that when hundreds of researchers contribute to a shared purpose and solve a shared problem in open and transparent ways, everyone benefits. The pace of innovation is accelerated and the diversity of solutions and approaches ensures that good solutions persist and not so good ones are quickly identified. Some might argue that the relative success of open-sourced platforms suggests that proprietary technologies often preclude necessary innovation. The growth of cloud has further challenged both researchers and industry. The scale economies of cloud-based solutions brings with it attendant challenges for the research community – beginning with the question of how to scale a conversation that brings both sides of an open community together to ensure that the full benefits of the cloud can be realized.

Recognizing the potential role of cloud-based open computing technologies to the research community, a group of 30 key stakeholders and decision-makers from academia and industry got together last week to share their views on how open computing solutions can best support existing and emerging use cases in a range of research disciplines and high performance workloads. The event was hosted by Argonne National Labs and jointly sponsored by Notre Dame, Internet2 and Rackspace; with attendees representing 20 organizations including participants from top research institutes, major research universities and key industry partners that provide technology supporting the research community.

The group discussed how big data and high-performance computing can introduce new challenges and new frustrations. Say you’re an academic or researcher who needs time on a supercomputer; oftentimes you’ll have to wait months to get approved, and even then you likely only get a limited window. So if something with your software is not working at that time, you’re out of luck.

The cloud changes the computing equation and redefines the experience and service by adding on-demand, utility and self-service capabilities to computing infrastructure. The cloud is quickly evolving into a premium model for scientific computation and big data, and the face of high-performance computing is changing faster than ever. We’ve seen the change happen over the past couple of years, as open technologies like the Open Compute Project and OpenStack, in particular, democratize access to mass commodity hardware and software.  Now, top research institutes such as CERN, Argonne National Laboratory, Notre Dame, University of Texas at San Antonio and MIT have chosen to build their high-performance clouds on OpenStack. By embracing open standards and collaboration, university researchers are at the forefront of innovation and contribute to a shared purpose that benefits everyone.

The event organizers – Narayan Desai (Argonne National Lab) , Paul Brenner (Notre Dame University), Khalil Yazdi (Internet2)  and Paul Rad (Rackspace) – welcomed participants and laid out a vision and an agenda for the workshop. Here are some of the key points that emerged during discussions in the morning:

  • Public clouds were not designed to address the requirements of research communities.
  • Public clouds appear inexpensive until you factor in the costs of networking and I/O.  Consequently, technical computing often requires hybrid and community clouds.
  • Provisioning a private cloud for average loads and bursting for peak loads provides the most economic model, but access to data sets can be critical for big data workflows, e.g. in high-energy physics.
  • By working together as community, we can scale up more appropriately. e.g. a community-owned open cloud might include a number of federated universities and research labs.
  • Within five to 10 years, large super computer systems will converge with cloud provider systems.
Click image to enlarge

In the afternoon sessions, as part of a “Cloud Best Practices” discussion, Notre Dame’s Brenner, Argonne’s Desai and UTSA’s Raj Boppana delivered in-depth presentations on a wide range of research and university OpenStack use cases for High Performance Cloud.

  • Case Study No. 1: Lessons Learned Running a Technical Cloud – Narayan Desai (Argonne National Lab)
  • Case Study No. 2: Bridging campus, lab, and commercial research infrastructure with an open cloud for high energy physics – Dr. Paul Brenner (Notre Dame University)
  • Case Study No. 3: OpenStack-based High Performance Cloud – Dr. Rajendra Boppana (UT, San Antonio)

They shared lessons learned and presented the findings and gaps that point the way forward for compute- and data-intensive applications.

Internet2 CTO Stephen Wolff said, “Workshops like the one held at ANL play an important role in bringing together stakeholders to explore technology trends and to cooperatively accelerate their development.  Internet2 is excited to be involved in this cutting edge work.

At the end of the session, the community identified two immediate incubation projects (with several other possible projects noted):

  • Big Data Reachback for Cloud Bursting Scientific Applications such as high energy physics led by Notre Dame, Internet2, Rackspace, UTSA, MIT and Cycle Computing
  • Big Data Scale out storage architecture led by Argonne, University of Chicago and Nimbis Services

The teams are planning to develop blue prints, detailed service descriptions and plans for a continuing collaborative effort and identifying regular communication channels for these projects. They will likely get back together at supercomputing 2013 and WCSC 2013 in San Antonio Texas.

Previous articleBuilding The Cloud Monitoring Dashboard
Next articleAccelerate Your Career With OpenStack: Upcoming LPI Webinar
Paul Rad is an Open Cloud Research Strategist for Rackspace. Hestarted his career as a computer architect by founding Data Processing Corp, overseas before moving to the United States, and later held product and services leadership roles at Data Concepts and Dell Inc. He has numerous published articles on enterprise solutions and holds several U.S. patents in the fields of virtualization, clustering, software engineering and quality assurance. Paul holds a Master of Computer Architecture from Sharif University and a Master of Computer Science from University of Texas at San Antonio. Paul has been a strong supporter of connecting university and industry in order to build the future workforce. He is a standing committee member of Quantitative Literacy at University of Texas at San Antonio.