Zap-In Technology
Speed benchmark testing

Overview

Benchmark 1: Speed comparison among Zap-In and Spark and BigQuery
Benchmark 2: Zap-In speed measurement: categorizing processing
Benchmark 3: Zap-In speed measurement: searching, sorting, totaling, overwriting

Processing-speed benchmark 1

Speed comparison among Zap-In and Spark and BigQuery

We compared processing speeds with Spark and BigQuery, well-known high-speed processing engines for large-scale data.

Zap-In recorded 25 times faster than Spark and 27 times faster than BigQuery in processing of loading data, searching, joining, totaling, and writing data.
This speed advantage becomes even more dramatic the larger the data being used.

Benchmark Result
Process Zap-In Spark BigQuery
Time[ms] Time[ms] Ratio Time[ms] Ratio
1. Import Data A 13 2,352 x181 1,000 x77
1. Import Data B 2 150 x75 1,000 x500
1. Import Data C 1,788 24,424 x14 71,100 x40
2. Search 19 495 x26 15,700 x826
3. Join 658 234 x0.4 (*) 10,200 x15
4. Summary 125 208 x1.7 3,000 x24
5. Export Data D 1,586 78,973 x50 (*) 12,000 x8
Total Time 4,192 106,839 x25 114,000 x27

(*) In join processing, Spark conducts only preparatory processing when the join request is issued. Actual processing takes place when exporting the data.

Benchmark Condition
Data

Data A: SalesmanMaster.csv          1,000 lines
Data B: ProductMaster.csv                100 lines
Data C: SalesLog.csv              6,070,000 lines
Data D: Export.csv                      73,000 lines

Benchmark Tasks

1. Data Import
2. Search
3. Join
4. Summary
5. Data Export

Hardware (Zap-In, Spark)

— CPU ————————————-
Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz
cpu clock : 1201.148 MHz
cache size : 10240 KB
fpu : yes
fpu_exception : yes

— Memory ————————————
MemTotal:      32726272 kB
MemFree:       28515008 kB
MemAvailable: 30948488 kB
Buffers:                   884 kB
Cached:            2568612 kB
SwapCached:               0 kB

— OS ——————————————–
CentOS Linux release 7.1.1503 (Core)

Hardware (BigQuery)

Google Cloud

Processing-speed benchmark 2

Zap-In Technology speed measurement

We measured Zap-In’s speed of processing categorizing.  It completed categorizing 1 billion records of data into 10 categories in only 187 ms.
Result

Process Time
      187 ms    (535,000,000 overwrite/sec)

Benchmark Condition
Data

Mass of data              1,000,000,000 lines
Columns in each line: 8 columns

Benchmark Tasks

Categorized into 10 categories

Hardware

— CPU  ————————————-
AMD Phenom2 CPU x4  925(4core) @ 2.80GHz
cpu clock : 1201.148 MHz
L1 : 2MB
L2 : 6MB

— Memory ————————————
MemTotal:      8 MB (1333MHz)

— OS ——————————————–
Windows 7 Ultimate 64bit

Processing-speed benchmark 3

Zap-In Technology speed measurement

We measured Zap-In’s speed of processing searching, sorting, totaling, and overwriting.  As an example, it completed sorting data containing 100 million text strings in only 5,659 ms.
Result
Processing Time
 Process Zap-In
1. Search (hit 10 items) 0 ms
1. Search (hit 10,000 items) 0 ms
1. Search (hit 1,000,000 items) 7 ms
2. Sort 5,659 ms
3. Summary 9,312 ms
4. Overwrite (1,000,000 items) 10,784 ms

 

Benchmark Conditions
Data

Mass of Data          100,000,000 lines
Colums in each line 8 columns

column 1: integer, 100 million types of values, sequential
column 2: integer, 10,000 types of values, random
column 3: integer, 100 types of values, random
column 4: integer, 100 types of values, random
column 5: text string, 100 million types of values, random
column 6: text string, 100 types of values, random
column 7: double-precision floating point, 100 million types of values, random
column 8: decimal fixed point (38 digits), 100 million types of values, random

Benchmark Tasks

Process 1. Searching: Search a text string (unique and random)
Process 2. Sort: Sort item 5 (unique and random text string)
Process 3. Totaling: One-dimensional (100 types of values), one-measure (unique, random floating point) totaling
Process 4. Overwriting: Overwriting 1 million items

Hardware

— CPU  ————————————-
Intel Xeon x2 (2core) @ 3.16GHz
L1 : 64kB
L2 : 1MB/1core

— Memory ————————————
MemTotal:      32 MB (1333MHz)

— OS ——————————————–
Windows Server 2008 Standard

<To Zap-In Technical Documents>