Zap-In Technology

ultra-high-speed database technology

Overview

The greatest strength of Zap-In Technology is its overwhelming ultra-high speed processing, realizing speeds of 20 or more times those of Spark, said to be an exemplar of ultra-high-speed databases. This advantage increases by leaps and bounds when processing Big Data. The data structures and algorithms of the linear filtering technology developed by Turbo Data Laboratories make possible database processing at unprecedented ultra-high speeds.

1. Overwhelming ultra-high speeds

Zap-In Technology database processing using the data structures and algorithms of linear filtering has the following properties.

– Ultra-high-speed database processing

Enables processing speeds 10-1000 times faster than those of an ordinary database system, or 100,000 times faster in JOIN processing. In real-world examples, shortened the time needed for processing that had taken day and night on an ordinary database system to one minute or less.

<To benchmarks>

– A proportional relation between data volume and processing time

A basic property of an ordinary relational database system is the way the time processing takes will increase rapidly as the volume of data increases (o(n*log(n)), where n is the volume of data). For this reason, Big Data, which involves massive data volumes, takes a very long time to process. In contrast, with Zap-In Technology processing time is proportional to the volume of data (o(n), where n is the volume of data). This difference in speeds is very pronounced when using Big Data.

– High-speed data loading

Loads data files in .csv or other formats at speeds an order of magnitude faster than those of ordinary databases. These are the results of unmatched technologies including high-speed indexing using a proprietary data structure and parallel processing on multicore CPUs.

2. Speed comparisons

<To speed benchmarks>

Processing RDB (disk) database RDB (in-memory) database NoSQL database Full-text retrieval database Turbo Zap-In (Note: Distinctive features of Zap-In)
Loading CSV data 1 10 50 0.05 100
JOIN operation 1 10 1000-100,000 High speeds even with high cardinality
SORT 1 10 100-100,000 High speeds even with high cardinality
SEARCH 1 10 100-1000
(perfect match of key term)
10-1000 High speeds even with large data volumes
BOM deployment 1 10 500-700
Categorizing 1 10 10-1000,000 Particularly high speed O(n)
Totaling 1 10 1000-100,000 High speeds even with high cardinality
Calculation, updating 1 10 0.1-10,000 Updating one item at a time is slow, bulk updating is ultra-high-speed
EXPORT 1 10 1
(when one data item fully matches the key term)
1-10
(pulling hit documents)
100-1000 Particularly high-speed when the number of hits is high
Full-text retrieval 1 10 100-10000 10-1000 High-speed even when the number of hits is high

Note: Cardinality: Number of types of values. If the item has the value “male” or “female,” the cardinality is two. It becomes much larger with data such as names. It becomes massive with numerical and similar data. In an ordinary database, speed decreases as cardinality increases.

<To speed benchmarks>

3. Reasons for ultra-high speeds

This is an in-memory database that takes advantage of the higher speeds possible through storing data in main memory instead of on a hard disk. Various vendors offer in-memory databases. Use of an in-memory database can result in a high-speed database (up to 10 times faster) even if the data structures and processing algorithms remain unchanged. However, this alone cannot be said to be putting the speed advantages of main memory to their full use.

– Linear filtering (LFM)

 Turbo Data has developed its own proprietary linear filtering method to take even greater advantage of the efficacy of in-memory processing. Furthermore, using new data structures and processing algorithms based on this has resulted in the ultra-high-speed database technology of Zap-In Technology. Linear filtering and its algorithms are patented around the world.

– Putting multicore CPUs to efficient use

 Today’s PCs use multicore CPUs with from four to 16 computing cores. Putting these multiple computing cores to effective use makes it possible to increase the speed of database processing massively. Doing so requires software capable of parallel database processing. However, in general it is difficult to use parallel processing with ordinary database systems due to their data structures and algorithms. Linear filtering, in contrast, enables high-speed parallel processing of most database processing thanks to the nature of its data structures and algorithms. Zap-In Technology takes maximum advantage of multicore CPUs, realizing ultra-high speeds not only in database processing but in processing such as loading of data as well.

– Indexing of all items

 Index design is key to increasing database speeds when developing a system. In a traditional database system, time needs to be taken in designing an index in order to enable efficient, high-speed processing. It is important to decide which items of a database to index in order to increase speed. In many cases it is not possible to achieve sufficiently high speeds even with the optimal index design, and when new functions are needed after the start of operation sometimes it’s not possible to secure sufficient processing speeds without a thorough redesign of the index. Thanks to the basic properties of its linear filtering, 
Zap-In Technology indexes all items, so that there is no need for additional processing to add index items This enables high-speed database processing for any item in the database. Also, since there is no need to design an index during system integration, it shortens the time required for the development process, and it also makes it possible to upgrade systems quickly because there is no need to redesign the index even if new functions need to be added after the start of operation.

– Algorithms

 While Zap-In Technology uses linear filtering to realize ultra-high-speed database processing, 、to put this theory to use in actual software products it employs a wide range of algorithms, peripheral technologies, and software technologies to put linear filtering to full use. We will continue patenting linear filtering technologies in countries around the world.

4. Theoretical background

See here for more on Linear Filtering Method (LFM), the core of Zap-In Technology:<Technical details: About Linear Filtering Method>

<To next page: Automated programming feature>