Tag Archives: comparison

Dalvik versus ART benchmarks on a Nexus 5 running Android 4.4.2

Updated benchmark data for Dalvik and ART running apps on Android 4.4.2 Kit Kat on a Google Nexus 5

This article provides an update to the performance results from benchmarks running on the new Android ART runtime on Android 4.4.2.  This is not a review.  ART is only meant for developers at this time, so don’t use it if you have critical apps on your phone.

My previous article is still available: Dalvik versus ART benchmarks on a Nexus 5.

I also made a 40-minute video of all of the benchmarks running on the Nexus 5:

Test setup and brief introduction to ART:

If you have a smartphone running Android 4.4.2 “Kit Kat”, and you have activated the developer options in settings, you may have noticed a new option to select the runtime.  In all previous versions of Android, the runtime was “Dalvik”.  Google now offers “ART”.  ART will one day let you run all of your apps faster and with less power by using ahead-of-time compiling.  As of May 2014, ART is still not at that point, but Google wants developers to try out the new runtime and give feedback.

ART is not meant for the general public yet.  There are still bugs, and some apps won’t work.

Once again, I wanted to see how ART does at running benchmarks with the updated Android 4.4.2 release.  I used two Google Nexus 5 smartphones running Android 4.4.2.  This time, I ran all of the tests “unplugged” (on the battery). The specific hardware and firmware is shown in the screenshot below.

Tested firmware and hardware versions

Details of the tested phone

When the “Developer options” are activated, they show up in “Settings” just above “About phone”. You can switch to ART under the “Select runtime” option. Without “Developer options”, the phone defaults to “Dalvik”. The screenshot below shows one phone running Dalvik and one running ART.

Dalvik and ART test phones

Dalvik and ART test phones

All of the benchmark apps are available for free on the Play Store.  I wanted apps that people could try on their current phones for comparison.

I ran all of the tests on the Dalvik and ART phones.  I only ran each test one time, and the phones were not reinitialized, so I may have some incorrect results.

Quadrant Standard Edition by Aurora Softworks:

According to the developer, “Quadrant is a CPU, I/O and 3D graphics benchmark.”

I did not use airplane mode for this benchmark, as it requires a network connection.

Quadrant Standard Edition results

Quadrant Standard Edition results

Quadrant Standard Edition is available on Google Play:

https://play.google.com/store/apps/details?id=com.aurorasoftworks.quadrant.ui.standard

 

GFXBench 2 by Kishonti Ltd.:

According to the developer, “Using console-quality gaming content, GFXBench (formerly GLBenchmark 2.7) provides a tool for unified cross-platform graphics benchmarking on major consumer platforms including Android, iOS and Windows.”

I did not use airplane mode for this benchmark, as it requires a network connection.

GFXBench 3 results (concatenated)

GFXBench 3 results (concatenated)

GFXBench 3 is available on Google Play;

https://play.google.com/store/apps/details?id=com.glbenchmark.glbenchmark27

AndEBench by EEMBC:

According to the developer, “The AndEBench™ benchmark provides a standardized, industry-accepted method of evaluating Android platform performance.”

I ran this benchmark in airplane mode.

AndEBench results

AndEBench results

AndEBench is available on Google Play;

https://play.google.com/store/apps/details?id=com.eembc.coremark

3DMark by Futuremark Oy:

According to the developer, 3DMark is “the world’s most popular performance test”.

I ran this benchmark in airplane mode.

3DMark results

3DMark results (I accidentally took the Dalvik screenshot in landscape mode.)

3DMark is available on Google Play:

https://play.google.com/store/apps/details?id=com.futuremark.dmandroid.application

AnTuTu Benchmark by Antutu:

According to the developer, “AnTuTu Benchmark is free benchmarking app that helps users know deeper into their Android devices through hard numbers and solid facts.”

I ran this benchmark in airplane mode.

AnTuTu Benchmark results

AnTuTu Benchmark results

AnTuTu Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=com.antutu.ABenchMark

Basemark X by Rightware:

According to the developer, “Basemark® X by Rightware® is professional performance evaluation tool for objective cross platform comparisons of game graphics performance between Android, iOS and Windows Phone 8 devices.”

I ran this benchmark in airplane mode.

Basemark X results

Basemark X results

Basemark X is available on Google Play:

https://play.google.com/store/apps/details?id=com.rightware.BasemarkX_Free

Vellamo Mobile Benchmark by Qualcomm Connected Experiences:

According to the developer, “Vellamo is designed to be an accurate, easy-to-use suite of system-level benchmarks for devices based on Android 2.3 forward.”

I ran this benchmark in airplane mode.

Vellamo Mobile Benchmark results

Vellamo Mobile Benchmark results

Vellamo Mobile Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=com.quicinc.vellamo

Pi Benchmark by Sasa D. Markovic:

According to the developer, “Pi Benchmark is a tool to measure the real speed of CPU and memory of your phone.”

I ran this benchmark in airplane mode.

Pi Benchmark results

Pi Benchmark results

Pi Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=rs.in.luka.android.pi

Prime Number Generator by Adrian Campos:

According to the developer, Prime Number Generator is “Just a prime number generator. Nothing special. It will allow you to enter any number to start with and to end with, and then share the results.”

I ran this app in airplane mode.  It was not designed as a benchmark, so I counted video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I generated prime numbers from 2 to 100 000. Also, you may have noticed that these are old screenshots from the earlier tests. I forgot to take new screenshots for the Prime Number Generator, but I did insert the updated times.

Prime Number Generator results

Prime Number Generator results

Prime Number Generator is available on Google Play;

https://play.google.com/store/apps/details?id=com.adriancampos.primenumbers

Lame4Android by Intervigil:

According to the developer, “Lame4Android brings the LAME MP3 encoder to the Android smartphone platform.”

I ran this app in airplane mode.  It was not designed as a benchmark, so I counted video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I encoded a ten minute mono 44 KHz audio file (local French news program) at the “Extreme” setting.

Lame4Android results

Lame4Android results

Lame4Android is available on Google Play;

https://play.google.com/store/apps/details?id=com.intervigil.lame

ASTRO File Manager with Clouds by Metago:

According to the developer, “ASTRO Cloud & File Manager has over 70 million downloads worldwide!”

I ran this app in airplane mode.  It was not designed as a benchmark, so I counted video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I used the built-in zip function to compress a 380 MB file.  The test file was the map of France in Mappy GPS Free by Mappy S.A.

ASTRO File Manager with Clouds zip function results

ASTRO File Manager with Clouds zip function results

ASTRO File Manager with Clouds is available on Google Play:

https://play.google.com/store/apps/details?id=com.metago.astro

Summary of results:

BenchmarkEditorSub-testDalvik resultART result
Quadrant Standard EditionAurora Softworks893212583
GFXBench 3Kishonti Ltd.T-Rex HD Onscreen1361 (24 FPS)1365 (24 FPS)
AndEBenchEEMBCNative1412313267
AndEBenchEEMBCJava506295
3DMarkFuturemark Oy1715216610
AnTuTu BenchmarkAnTuTu2377421999
Basemark XRightwareOn-Screen17.24217.147
Basemark XRightwareOff-Screen13.16513.569
Vellamo Mobile BenchmarkQualcomm Connected ExperiencesHTMl 512571189
Vellamo Mobile BenchmarkQualcomm Connected ExperiencesMetal11781143
Pi BenchmarkSasa D. Markovic100000 digits18.04 seconds11.51 seconds
Prime Number GeneratorAdrian CamposPrimes from 2-1000008.1 seconds6.6 seconds
Lame4AndroidIntervigilCompression of a 10 minute audio file16.5 seconds15.9 seconds
ASTRO File Manager with CloudsMetagozip function on GPS map file122.2 seconds121.0 seconds

Dalvik versus Art benchmark results on Android 4.4.2

All of the apps in this study ran flawlessly on the new ART runtime.  Most of the benchmarks ran at similar times, but some did show a significant difference in performance.  Google will certainly continue to work on improving ART before possibly making it the main runtime.  Until then, it would be a good idea for developers to test their apps on ART.  You can let me know what you think of the results in your comments below.

Dalvik versus ART benchmarks on a Nexus 5

Early benchmark data for Dalvik and ART running apps on Android 4.4 Kit Kat on a Google Nexus 5

NOTE: AN UPDATE TO THIS ARTICLE IS AVAILABLE WITH ANDROID 4.4.2 RESULTS.

The new article provides updated results with two separate phones running Android 4.4.2 Kit Kat. Also, the updated tests are all run “unplugged” (using the battery) and use the latest versions of the benchmark apps. The new article is available here: Dalvik versus ART benchmarks on a Nexus 5 running Android 4.4.2.

Here’s the original article (running Android 4.4.0):

This article covers performance results from benchmarks running on the new Android ART runtime.  This is not a review.  ART is only meant for developers at this time, so don’t use it if you have critical apps on your phone.

I also made a 30-minute video of all of the benchmarks running on the Nexus 5:

Test setup and brief introduction to ART:

If you have a smartphone running Android 4.4 “Kit Kat”, and you have activated the developer options in settings, you may have noticed a new option to select the runtime.  In all previous versions of Android, the runtime was “Dalvik”.  Google now offers “ART”.  ART will one day let you run all of your apps faster and with less power by using ahead-of-time compiling.  As of November 2013, ART is still not at that point, but Google wants developers to try out the new runtime and give feedback.

ART is not meant for the general public yet.  There are still bugs, and some apps won’t work.

I wanted to see how ART does at running benchmarks.  I used a Google Nexus 5 smartphone running Android 4.4.  The specific hardware and firmware is shown in the screenshot below.

Tested firmware and hardware versions

Details of the tested phone

All of the benchmark apps are available for free on the Play Store.  I wanted apps that people could try on their current phones for comparison.

I ran all of the tests on Dalvik.  Then, I set the runtime to Art and waited about 12 minutes for my 158 apps to be prepared.  Then I ran all of the same tests.  I only ran each test one time, so I may have some incorrect results.

Quadrant Standard Edition by Aurora Softworks:

According to the developer, “Quadrant is a CPU, I/O and 3D graphics benchmark.”

I did not use airplane mode for this benchmark, as it requires a network connection.

Quadrant Standard Edition results

Quadrant Standard Edition results

Quadrant Standard Edition is available on Google Play:

https://play.google.com/store/apps/details?id=com.aurorasoftworks.quadrant.ui.standard

 

GFXBench 2 by Kishonti Ltd.:

According to the developer, “Using console-quality gaming content, GFXBench (formerly GLBenchmark 2.7) provides a tool for unified cross-platform graphics benchmarking on major consumer platforms including Android, iOS and Windows.”

I did not use airplane mode for this benchmark, as it requires a network connection.

GFXBench 2 results (scrolled up)

GFXBench 2 results (scrolled up)

GFXBench 2 results (scrolled down)

GFXBench 2 results (scrolled down)

GFXBench 2 is available on Google Play;

https://play.google.com/store/apps/details?id=com.glbenchmark.glbenchmark27

AndEBench by EEMBC:

According to the developer, “The AndEBench™ benchmark provides a standardized, industry-accepted method of evaluating Android platform performance.”

I ran this benchmark in airplane mode.

AndEBench results

AndEBench results

AndEBench is available on Google Play;

https://play.google.com/store/apps/details?id=com.eembc.coremark

3DMark by Futuremark Oy:

According to the developer, 3DMark is “the world’s most popular performance test”.

I ran this benchmark in airplane mode.

3DMark results

3DMark results (I accidentally took the Dalvik screenshot in landscape mode.)

3DMark is available on Google Play:

https://play.google.com/store/apps/details?id=com.futuremark.dmandroid.application

AnTuTu Benchmark by Antutu:

According to the developer, “AnTuTu Benchmark is free benchmarking app that helps users know deeper into their Android devices through hard numbers and solid facts.”

I ran this benchmark in airplane mode.

AnTuTu Benchmark results

AnTuTu Benchmark results

AnTuTu Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=com.antutu.ABenchMark

Basemark X by Rightware:

According to the developer, “Basemark® X by Rightware® is professional performance evaluation tool for objective cross platform comparisons of game graphics performance between Android, iOS and Windows Phone 8 devices.”

I ran this benchmark in airplane mode.

Basemark X results

Basemark X results

Basemark X is available on Google Play:

https://play.google.com/store/apps/details?id=com.rightware.BasemarkX_Free

Vellamo Mobile Benchmark by Qualcomm Connected Experiences:

According to the developer, “Vellamo is designed to be an accurate, easy-to-use suite of system-level benchmarks for devices based on Android 2.3 forward.”

I ran this benchmark in airplane mode.

Vellamo Mobile Benchmark results

Vellamo Mobile Benchmark results

Vellamo Mobile Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=com.quicinc.vellamo

Pi Benchmark by Sasa D. Markovic:

According to the developer, “Pi Benchmark is a tool to measure the real speed of CPU and memory of your phone.”

I ran this benchmark in airplane mode.

Pi Benchmark results

Pi Benchmark results

Pi Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=rs.in.luka.android.pi

Prime Number Generator by Adrian Campos:

According to the developer, Prime Number Generator is “Just a prime number generator. Nothing special. It will allow you to enter any number to start with and to end with, and then share the results.”

I ran this app in airplane mode.  It is not designed as a benchmark, so I had to count video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I generated prime numbers from 2 to 100 000.

Prime Number Generator results

Prime Number Generator results

Prime Number Generator is available on Google Play;

https://play.google.com/store/apps/details?id=com.adriancampos.primenumbers

Lame4Android by Intervigil:

According to the developer, “Lame4Android brings the LAME MP3 encoder to the Android smartphone platform.”

I ran this app in airplane mode.  It is not designed as a benchmark, so I had to count video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I encoded a ten minute mono 44 KHz audio file (local French news program) at the “Extreme” setting.

Lame4Android results

Lame4Android results

Lame4Android is available on Google Play;

https://play.google.com/store/apps/details?id=com.intervigil.lame

ASTRO File Manager with Clouds by Metago:

According to the developer, “ASTRO Cloud & File Manager has over 70 million downloads worldwide!”

I ran this app in airplane mode.  It is not designed as a benchmark, so I had to count video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I used the built-in zip function to compress a 333 MB file.  The test file was the map of France in Mappy GPS Free by Mappy S.A.

ASTRO File Manager with Clouds zip function results

ASTRO File Manager with Clouds zip function results

ASTRO File Manager with Clouds is available on Google Play:

https://play.google.com/store/apps/details?id=com.metago.astro

Summary of results:

BenchmarkEditorSub-testDalvik resultART result
Quadrant Standard EditionAurora Softworks944412349
GFXBench 2Kishonti Ltd.T-Rex HD Onscreen1348 (24 FPS)1351 (24 FPS)
AndEBenchEEMBCNative1389011485
AndEBenchEEMBCJava391300
3DMarkFuturemark Oy1663915012
AnTuTu BenchmarkAnTuTu2267022215
Basemark XRightwareOn-Screen17.15717.187
Basemark XRightwareOff-Screen13.60913.748
Vellamo Mobile BenchmarkQualcomm Connected ExperiencesHTMl 5997985
Vellamo Mobile BenchmarkQualcomm Connected ExperiencesMetal11151156
Pi BenchmarkSasa D. Markovic100000 digits17.92 seconds11.58 seconds
Prime Number GeneratorAdrian CamposPrimes from 2-1000008.4 seconds6.6 seconds
Lame4AndroidIntervigilCompression of a 10 minute audio file16.2 seconds16.6 seconds
ASTRO File Manager with CloudsMetagozip function on 333 MB file104.2 seconds107.2 seconds

Dalvik versus Art benchmark results

All of the apps in this study ran flawlessly on the new ART runtime.  Most of the benchmarks ran at similar times, but some did show a significant difference in performance.  Google will certainly continue to work on improving ART before possibly making it the main runtime.  Until then, it would be a good idea for developers to test their apps on ART.  You can let me know what you think of the results in your comments below.