Tal Lavian, Randy H. Katz; Doctoral Thesis, University of California at Berkeley. January 2006.
As we navigate the 21st century, the practice of science is undergoing significant shifts, particularly in the realm of e-Science applications. The exponential increase in network capacity, facilitated by the rapid development of agile optical transport, is ushering in a new era. This evolution is crucial for the burgeoning field of e-Science, which necessitates the transfer of immense data volumes, ranging from dozens to hundreds of TeraBytes and even PetaBytes.
The invention of the transistor in 1947 at Bell Labs was the triggering event that led to the technology revolution of the 20th century. The completion of the Human Genome Project (HGP) in 2003 was the triggering event for the life science revolution of the 21st century. Understanding the genome, DNA, proteins, and enzymes is a prerequisite to modifying their properties and advancing systematic biology. Grid Computing has become the fundamental platform for conducting this e-science research. Vast increases in data generation by e-science applications, along with advances in computation, storage, and communication, affect the nature of scientific research. During this decade, crossing the "Peta" line is expected: Petabyte in data size, Petaflop in CPU processing, and Petabit/s in network bandwidth.
Numerous challenges arise from a network with a capacity millions of times more remarkable than the public Internet. Currently, the distribution of large amounts of data is restricted by the inherent bottleneck nature of today's public Internet architecture, which employs packet-switching technologies. Bandwidth limitations of the Internet inhibit the advancement and utilization of new e-science applications in Grid Computing. These emerging e-science applications are evolving in data centers and clusters; however, the potential capability of a globally distributed system over long distances is yet to be realized. Today's network orchestration of resources and services is done manually via multi-party conference calls, emails, yellow sticky notes, and reminder communications, all of which rely on human interaction to get results. The work in this thesis automates the orchestration of networks with other resources, better utilizing all resources time-efficiently. Automation allows for a vastly more comprehensive use of all components and removes human limitations from the process. We demonstrated automatic Lambda setting-up and tearing-down as part of application servers over MEMs testbed in the Chicago metro area in a matter of seconds and across domains over transatlantic links in around a minute.
The central aim of this thesis is to construct a novel grid-computing paradigm that fully exploits the available communication infrastructure. An optical network acts as the third leg in orchestration with computation and storage. This tripod architecture forms the basis for the global distribution of vast data volumes in emerging e-science applications, emphasizing this research's practical benefits and efficiency.
One of the key areas of investigation in this thesis is the potential of Lambda on demand technology to revolutionize e-Science applications in Grid Virtual Organization (VO). This innovative technology provides crucial networking fundamentals that are currently absent from the Grid Computing environment. By overcoming current bandwidth limitations, it paves the way for the realization of VO, thereby eliminating some fundamental barriers to the growth of this new big science branch and instilling a sense of optimism about the future of e-Science applications.
Within this thesis, the Lambda Data Grid serves as the knowledge plane that enables e-science applications to transfer enormous data volumes over a dedicated Lightpath. This practical application of the research enhances science research by facilitating the efficient collaboration of large distributed teams, utilizing simulations and computational science as a third branch of research.