ultra-high-speed database technology
The greatest strength of Zap-In Technology is its overwhelming ultra-high speed processing, realizing speeds of 20 or more times those of Spark, said to be an exemplar of ultra-high-speed databases. This advantage increases by leaps and bounds when processing Big Data. The data structures and algorithms of the linear filtering technology developed by Turbo Data Laboratories make possible database processing at unprecedented ultra-high speeds.
1. Overwhelming ultra-high speeds
Zap-In Technology database processing using the data structures and algorithms of linear filtering has the following properties.
– Ultra-high-speed database processing
Enables processing speeds 10-1000 times faster than those of an ordinary database system, or 100,000 times faster in JOIN processing. In real-world examples, shortened the time needed for processing that had taken day and night on an ordinary database system to one minute or less.
– A proportional relation between data volume and processing time
A basic property of an ordinary relational database system is the way the time processing takes will increase rapidly as the volume of data increases (o(n*log(n)), where n is the volume of data). For this reason, Big Data, which involves massive data volumes, takes a very long time to process. In contrast, with Zap-In Technology processing time is proportional to the volume of data (o(n), where n is the volume of data). This difference in speeds is very pronounced when using Big Data.
– High-speed data loading
Loads data files in .csv or other formats at speeds an order of magnitude faster than those of ordinary databases. These are the results of unmatched technologies including high-speed indexing using a proprietary data structure and parallel processing on multicore CPUs.
2. Speed comparisons
|Processing||RDB (disk) database||RDB (in-memory) database||NoSQL database||Full-text retrieval database||Turbo Zap-In||(Note: Distinctive features of Zap-In)|
|Loading CSV data||1||10||50||0.05||100|
|JOIN operation||1||10||–||–||1000-100,000||High speeds even with high cardinality|
|SORT||1||10||–||–||100-100,000||High speeds even with high cardinality|
(perfect match of key term)
|–||10-1000||High speeds even with large data volumes|
|Categorizing||1||10||–||–||10-1000,000||Particularly high speed O(n)|
|Totaling||1||10||–||–||1000-100,000||High speeds even with high cardinality|
|Calculation, updating||1||10||–||–||0.1-10,000||Updating one item at a time is slow, bulk updating is ultra-high-speed|
(when one data item fully matches the key term)
(pulling hit documents)
|100-1000||Particularly high-speed when the number of hits is high|
|Full-text retrieval||1||10||–||100-10000||10-1000||High-speed even when the number of hits is high|
Note: Cardinality: Number of types of values. If the item has the value “male” or “female,” the cardinality is two. It becomes much larger with data such as names. It becomes massive with numerical and similar data. In an ordinary database, speed decreases as cardinality increases.
3. Reasons for ultra-high speeds
This is an in-memory database that takes advantage of the higher speeds possible through storing data in main memory instead of on a hard disk. Various vendors offer in-memory databases. Use of an in-memory database can result in a high-speed database (up to 10 times faster) even if the data structures and processing algorithms remain unchanged. However, this alone cannot be said to be putting the speed advantages of main memory to their full use.
– Linear filtering (LFM)
Turbo Data has developed its own proprietary linear filtering method to take even greater advantage of the efficacy of in-memory processing. Furthermore, using new data structures and processing algorithms based on this has resulted in the ultra-high-speed database technology of Zap-In Technology. Linear filtering and its algorithms are patented around the world.
– Putting multicore CPUs to efficient use
Today’s PCs use multicore CPUs with from four to 16 computing cores. Putting these multiple computing cores to effective use makes it possible to increase the speed of database processing massively. Doing so requires software capable of parallel database processing. However, in general it is difficult to use parallel processing with ordinary database systems due to their data structures and algorithms. Linear filtering, in contrast, enables high-speed parallel processing of most database processing thanks to the nature of its data structures and algorithms. Zap-In Technology takes maximum advantage of multicore CPUs, realizing ultra-high speeds not only in database processing but in processing such as loading of data as well.
– Indexing of all items
Index design is key to increasing database speeds when developing a system. In a traditional database system, time needs to be taken in designing an index in order to enable efficient, high-speed processing. It is important to decide which items of a database to index in order to increase speed. In many cases it is not possible to achieve sufficiently high speeds even with the optimal index design, and when new functions are needed after the start of operation sometimes it’s not possible to secure sufficient processing speeds without a thorough redesign of the index. Thanks to the basic properties of its linear filtering, Zap-In Technology indexes all items, so that there is no need for additional processing to add index items This enables high-speed database processing for any item in the database. Also, since there is no need to design an index during system integration, it shortens the time required for the development process, and it also makes it possible to upgrade systems quickly because there is no need to redesign the index even if new functions need to be added after the start of operation.
While Zap-In Technology uses linear filtering to realize ultra-high-speed database processing, ､to put this theory to use in actual software products it employs a wide range of algorithms, peripheral technologies, and software technologies to put linear filtering to full use. We will continue patenting linear filtering technologies in countries around the world.
4. Theoretical background
See here for more on Linear Filtering Method (LFM), the core of Zap-In Technology:<Technical details: About Linear Filtering Method>