Benchmarking - what do you want to test

[Guest muff]

Hi all

I'm working on some stuff atm, and one of the projects is a benchmarker for handsets.

So I thought I'd ask you all for input on what u think it should test.

At the moment it's primarily designed to test my new gfx library, but I'm planning on extending it to test a whole range of stuff.

Things I already have on the list :-

Free memory vs Total memory

SD Card speeds

Memory Speeds

Record OS details

Gapi details (hires or not, pixel format)

Any other suggestions?

muff

[Guest fraser]

Add the options for a custom binary perhaps? Then folk can add their own custom benchmarks, or get benchmarks on an existing app.

Another useful feature would be to create artificial load. E.g. have an option to reserve a chunk of memory and produce 10/20/50/100% cpu usage in order to simulate loaded devices. Then you can run your test and see how things are impacted.

I kinda do this stuff for a living, but not the Pocket PC. If you really want to do a full-featured benchmarker, you'd want to add some form of reporting system that can be plugged into. E.g. a CSV file for transaction logging could allow other apps to log other benchmarking details. There are two ways to do this I can think of; mark app start/stop times and post-process for the data, or just mark the tranaction lengths. Of course, appending to a CSV file is a bit crap and will have file locking issues, so some kind of API/DLL to log to would be better.

Finally, you want to log the memory and cpu usage of the process under test against time. Handy for thread/memory leaks and the like. CSV logs are ultimately king as they can be easilly processed by many different tools. Believe it or not, I do all of my analysis in Excel.

Depends how far you want to take it... ;-)

[Guest]

Battery life

[Guest muff]

Add the options for a custom binary perhaps? Then folk can add their own custom benchmarks, or get benchmarks on an existing app.

Another useful feature would be to create artificial load. E.g. have an option to reserve a chunk of memory and produce 10/20/50/100% cpu usage in order to simulate loaded devices. Then you can run your test and see how things are impacted.

I can see the line of thought here

atm I'm just thinking of something that gives raw benchmarks of the device itself, I'm not really aiming for something to aid the benchmarking of 3rd party products - I'd really hope that they'd do those tests themselves ;)

Simulation of memory loads and CPU loads - that's an interesting one, but again I think it falls into the field of a secondary project - but I'll add it to my wishlist for the future ;)

and I agree that CSV's are always a nice easy format for analysis - just copy and paste into the 'summary' spreadsheet :(

atm I'm planning on doing output in a visual graphic on screen, summary text file, and maybe a CSV purely for pasting

gerryr:

there are several existing apps out there for battery life tests, and most of them run over long periods to get more accurate results / predictions - what I'm thinking of here is something that can be run on a new device to give everyone an idea of it's performance / capabilities

[Guest]

gotcha muff, good idea! I'll be sure to test it for you when you get to beta stage.

atb

gerryR

[Guest fraser]

I'm not really aiming for something to aid the benchmarking of 3rd party products - I'd really hope that they'd do those tests themselves ;)

LOL. :( If the j2ee world is anything to go by, they don't!! I have even lesser expectations on the PPC world. Loadtesting/benchmarking is quite a niche field really; most people don't do much of it.

Simulation of memory loads and CPU loads - that's an interesting one, but again I think it falls into the field of a secondary project - but I'll add it to my wishlist for the future ;)

A seldom found PPC app, "Pocket PC Testsuite 2.0" does that, that's where I got the idea. This tool is a GUI record/playback tool for automated testing. Cool, but I've not used it in anger yet...

[Guest muff]

yeah, had a quick look at the "Pocket PC Testsuite" and that's WAY beyond what I'm doing atm

atm I'm testing the first version before releasing it to the general public