Benchmark Hello World

Bagaimana benchmark dilakukan?

Benchmark Hello World dibuat untuk melihat beban kerja terkecil pada tiap framework. Banyak orang tidak menyukai benchmark ini karena penerapan aplikasi sebenarnya membutuhkan fitur & struktur yg lebih kompleks. Bagaimanapun jg, tes ini mengidentifikasikan waktu minimum yg dibutuhkan oleh tiap framework untuk melakukan tugas yg sederhana. Yaitu sebuah tugas yg merepresentasikan kebutuhan minimum untuk tiap framework untuk memroses suatu request.

Lebih spesifik, benchmark hanya mengukur waktu yg dibutuhkan frameworok untuk mulai bekerja, menjalankan action dan membersihkan sumber daya yg terpakai di akhir request. Berbagai aplikasi PHP berbasis arsitektur MVC akan membutuhkan waktu untuk melakukan tugas ini. Karena kesederhanaan benchmark, dapat dipastikan bahwa waktu yg dibutuhkan untuk request yg kompleks akan lebih lama.

Sebuah controller dan sebuah view dibuat untuk masing-masing framework. Controller “Say” dan action “hello”. Action hanya mengirimkan data ke view untuk menampilkannya (“Hello!”). Menggunakan tool benchmark “ab”‘ dikirimkan 2000 (dua ribu) request menggunakan 10 koneksi bersamaan (concurent) pada tiap framework.

Pengukuran apa saja yg dicatat?

Ini adalah daftar pengukuran yg dicatat untuk mengidentifikasi performa keseluruhan dr tiap framework:

  • Requests per detik
  • Waktu di semua request yg konkuren/bersamaan
  • Jumlah file PHP yg di-include pada satu request (diukur menggunakan fungsi get_included_files)
  • Penggunaan memori per request (diukur menggunakan fungsi memory_get_usage)

Hasil

Yii (YII_DEBUG=false) Version yii-1.1.13

# ab -n 2000 -c 10 http://localhost/bench/helloworld/yii/index.php?r=say/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            80

Document Path:          /bench/helloworld/yii/index.php?r=say/hello
Document Length:        61 bytes

Concurrency Level:      10
Time taken for tests:   2.081 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      508000 bytes
HTML transferred:       122000 bytes
Requests per second:    961.28 [#/sec] (mean)
Time per request:       10.403 [ms] (mean)
Time per request:       1.040 [ms] (mean, across all concurrent requests)
Transfer rate:          238.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   10   4.3      9      42
Processing:     0    0   1.0      0      24
Waiting:        0    0   0.8      0      17
Total:          3   10   4.3      9      42

Percentage of the requests served within a certain time (ms)
  50%      9
  66%     11
  75%     13
  80%     14
  90%     15
  95%     17
  98%     21
  99%     26
 100%     42 (longest request)

Symfony Version 2.1.6

# ab -n 2000 -c 10 http://localhost/bench/Symfony/web/app.php/say/hello/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            80

Document Path:          /bench/Symfony/web/app.php/say/hello/
Document Length:        16 bytes

Concurrency Level:      5
Time taken for tests:   1.848 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      249000 bytes
HTML transferred:       16000 bytes
Requests per second:    541.01 [#/sec] (mean)
Time per request:       9.242 [ms] (mean)
Time per request:       1.848 [ms] (mean, across all concurrent requests)
Transfer rate:          131.55 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    9   4.8      8      61
Processing:     0    0   0.6      0      15
Waiting:        0    0   0.6      0      15
Total:          4    9   4.8      8      61

Percentage of the requests served within a certain time (ms)
  50%      8
  66%      9
  75%     11
  80%     12
  90%     15
  95%     18
  98%     22
  99%     30
 100%     61 (longest request)

CodeIgniter 2.1.0

# ab -n 2000 -c 10 http://localhost/bench/codeigniter/index.php/say/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            80

Document Path:          /bench/helloworld/codeigniter/index.php/say/hello
Document Length:        16 bytes

Concurrency Level:      10
Time taken for tests:   1.888 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      418000 bytes
HTML transferred:       32000 bytes
Requests per second:    1059.05 [#/sec] (mean)
Time per request:       9.442 [ms] (mean)
Time per request:       0.944 [ms] (mean, across all concurrent requests)
Transfer rate:          216.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    9   4.1      9      33
Processing:     0    0   0.8      0      19
Waiting:        0    0   0.7      0      16
Total:          3    9   4.2      9      33

Percentage of the requests served within a certain time (ms)
  50%      9
  66%     10
  75%     11
  80%     12
  90%     14
  95%     16
  98%     21
  99%     24
 100%     33 (longest request)

Kohana 3.2.0

# ab -n 2000 -c 10 http://localhost/bench/helloworld/kohana/index.php/say/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            80

Document Path:          /bench/helloworld/kohana/index.php/say/hello
Document Length:        15 bytes

Concurrency Level:      10
Time taken for tests:   2.324 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      446446 bytes
HTML transferred:       30030 bytes
Requests per second:    860.59 [#/sec] (mean)
Time per request:       11.620 [ms] (mean)
Time per request:       1.162 [ms] (mean, across all concurrent requests)
Transfer rate:          187.60 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   11   5.1     10      64
Processing:     0    0   1.9      0      39
Waiting:        0    0   1.4      0      35
Total:          3   11   5.3     11      64

Percentage of the requests served within a certain time (ms)
  50%     11
  66%     13
  75%     15
  80%     15
  90%     17
  95%     18
  98%     24
  99%     31
 100%     64 (longest request)

Fuel 1.2.1

# ab -n 2000 -c 10 http://localhost/bench/helloworld/fuel/public/say/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            80

Document Path:          /bench/helloworld/fuel/public/say/hello
Document Length:        16 bytes

Concurrency Level:      10
Time taken for tests:   2.742 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      418000 bytes
HTML transferred:       32000 bytes
Requests per second:    729.42 [#/sec] (mean)
Time per request:       13.709 [ms] (mean)
Time per request:       1.371 [ms] (mean, across all concurrent requests)
Transfer rate:          148.88 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   13   6.0     12      79
Processing:     0    0   1.3      0      22
Waiting:        0    0   0.8      0      21
Total:          4   14   6.1     13      80

Percentage of the requests served within a certain time (ms)
  50%     13
  66%     15
  75%     17
  80%     17
  90%     19
  95%     24
  98%     30
  99%     38
 100%     80 (longest request)

Zend Framework 1.11.11

# ab -n 2000 -c 10 http://localhost/bench/helloworld/zendfw/public/index.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            80

Document Path:          /bench/helloworld/zendfw/public/index.php
Document Length:        16 bytes

Concurrency Level:      10
Time taken for tests:   5.641 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      418000 bytes
HTML transferred:       32000 bytes
Requests per second:    354.55 [#/sec] (mean)
Time per request:       28.205 [ms] (mean)
Time per request:       2.820 [ms] (mean, across all concurrent requests)
Transfer rate:          72.36 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   27   9.6     25      89
Processing:     0    1   3.0      0      70
Waiting:        0    0   2.9      0      70
Total:          9   28   9.6     26      90

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     28
  75%     32
  80%     34
  90%     41
  95%     46
  98%     55
  99%     62
 100%     90 (longest request)

Laravel 3.2.5

# ab -n 2000 -c 10 http://localhost/bench/helloworld/laravel/public/say/hello

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            80

Document Path:          /bench/helloworld/laravel/public/say/hello
Document Length:        15 bytes

Concurrency Level:      10
Time taken for tests:   4.090 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      1665162 bytes
HTML transferred:       30045 bytes
Requests per second:    489.03 [#/sec] (mean)
Time per request:       20.449 [ms] (mean)
Time per request:       2.045 [ms] (mean, across all concurrent requests)
Transfer rate:          397.61 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   20   7.6     19      92
Processing:     0    0   2.5      0      53
Waiting:        0    0   2.5      0      53
Total:          6   20   7.6     19      93

Percentage of the requests served within a certain time (ms)
  50%     19
  66%     21
  75%     23
  80%     24
  90%     29
  95%     34
  98%     42
  99%     48
 100%     93 (longest request)

Phalcon Version 0.8.0

# ab -n 2000 -c 10 http://localhost/bench/helloworld/phalcon/index.php?_url=/say/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            80

Document Path:          /bench/helloworld/phalcon/index.php?_url=/say/hello
Document Length:        16 bytes

Concurrency Level:      10
Time taken for tests:   0.789 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      418000 bytes
HTML transferred:       32000 bytes
Requests per second:    2535.82 [#/sec] (mean)
Time per request:       3.943 [ms] (mean)
Time per request:       0.394 [ms] (mean, across all concurrent requests)
Transfer rate:          517.56 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4   1.7      3      23
Processing:     0    0   0.2      0       6
Waiting:        0    0   0.2      0       6
Total:          2    4   1.7      3      23

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      4
  75%      4
  80%      4
  90%      5
  95%      6
  98%      8
  99%     14
 100%     23 (longest request)

Grafik

Grafik pertama menunjukan berapa banyak request per detik dari tiap framework yg dapat diterima. Yang kedua menunjukkan rata-rata waktu dr keseluruhan request yg bersamaan/konkuren.

Framework / Request per second (#/sec)
[lebih banyak lebih baik]

rps

Framework / Time per Request (mean, across all concurrentrequests)
[less is better]

tpr

Framework / Number of included PHP files (mean, number on asingle request)
[less is better]

ipr

Framework / Memory Usage (mean, megabytes per request)
[lessis better]

mpr

Kesimpulan

Sifat alamiah dari Phalcon yg compiled menawarkan performa luarbiasa yg jauh melebihi semua framework lainnya yg mengikuti benchmark ini.

Terjemah dr Benchmark Hello World
http://docs.phalconphp.com/en/latest/reference/benchmark/hello-world.html