The project finished on April 2023 and the scientific results were accepted by the SNSF.

The past few years have witnessed a rise in demand for real-time intelligence, as businesses from a wide-range of application domains increasingly require interactive data analytics for making insightful decisions in fast-breaking situations. As the amount of data accumulated by applications continues to grow faster than Moore’s law, achieving real-time intelligence requires a computational infrastructure that can sift through billions of rows in milliseconds. These demanding requirements have led to the development of a new breed of analytical database engines that are designed to exploit the massive parallelism and high memory bandwidth of General-Purpose Graphics Processing Units (GPU) to provide as much as 700× performance improvement over traditional CPU-based analytical engines in some situations [2, 3, 4].

For all the performance benefit provided by these new GPU-based engines, they significantly lag behind CPU-based engines with respect to versatility. All current GPU-based engines make one or more simplifying assumptions about several aspects like data size, workload partitionability, expected concurrency levels, or SQL compatibility, to name a few. For instance, in order to achieve the advertised 700× improvement in performance, GPU-based engines typically require all data to be loaded in the limited device memory of GPUs and co-partitioned in such a way that queries can be executed without any data movement. While certain niche applications, like interactive geo-visualization, fit these assumptions, most others require the performance of these GPU-based engines without sacrificing the versatility of traditional CPU-based analytical engines.

Our goal is to build an in-memory, analytical engine that enables real-time business intelligence by supporting concurrent execution of ad-hoc SQL queries over Terabyte-sized databases using multiple GPUs.

To achieve this goal, we will revisit the three central design aspects that govern the functioning of an analytical engine, namely, data layout and storage models, query execution and processing models, run-time optimization and scheduling models. For each aspect, we will investigate the applicability of current CPU-based techniques in the new multi-GPU context, research new techniques that can exploit GPU-specific properties, and perform a systematic analysis to establish rules of thumb for designing versatile, GPU-based data analytics engines.

The DIAS lab has a legacy of developing open-source database engines, like Shore-MT, that have been used as a platform by several researchers from both academia and industry for advancing state-of-the-art in database research. Our intention is to continue this tradition by developing an open-source GPU-based analytics engine that can be used by the data management community as a foundational platform for building more advanced analytics applications. Given the growing popularity of data analytics in the Swiss industrial sector, we also expect the results from our research to directly impact society by transforming several application domains like finance, security, and e-commerce.

The project is funded by the Swiss National Science Foundation, Project No.: 200021_178894

This project is part of Proteus DBMS (proteusdb.com).