There are myths, damn myths, and benchmarks!
- Unknown
|
"The Seven Myths of Benchmarking"
|
The first myth: its easy to do.
Facts:
Benchmarking is no place for amateurs - not if you want to make a decision based on the results.
If you are "fooling around" and investigating things for yourself,
you're very likely to get very different results even by altering what appears to be very minor factors.
This is especially true in embedded benchmarking, where variable such as board bus speed, clock speed,
memory speed, memory set-up, memory wait-states, types of memory, compiler used, compiler flags used,
link map defined, incorrect programming of the on-chip timer,
and literally dozens of other factors can make a huge difference in performance.
|
The second myth:
|
"Just send me the binary or the source code and I can get the same results as you do."
|
Facts:
Well, we wish this were true.
Sadly, its not that simple.
Because of the First myth, you can see that re-creating the benchmark environment can be difficult, time-consuming,
and exacting.
If you do not have the exact configuration, exact set of the same software tools (including the same version!),
and so on, it is usually very difficult to get the same scores as someone else.
You need complete disclosure - something
EEMBC
insists upon.
|
The third myth:
|
"Benchmarks will tell me what to buy."
|
Facts:
We wish this were true - it would make ECL's mission in life all the more important.
However, over-stating benchmark scores and drawing unwarranted conclusions based on simplistic benchmarks
is more "bench-marketing" than benchmarking.
Dhrystone is an excellent example.
Its an integer only, synthetic, tiny benchmark that was written long ago by our friend,
Dr. Reinhold Weicker (of Siemens-Nixdorf).
Reinhold himself has disavowed Dhrystone (which explains why he was and is a key member of
SPEC,
a reputable workstation and server benchmark organization).
Dhrystone is easily defeated by even primitive compilers, can be defeated easily if the rules are not followed
(for example, if you in-line the code, the compiler can easily see all of the code and optimized most of it away!),
and is an integrer-only benchmark.
It is so small it does not take into account memory subsystems, nor floating point, nor
SIMD
instructions like AltiVec, 3DNow,
MMX,
and so on and so forth.
This is a major reason why
EEMBC
was formed - to bring real-world application-based benchmarks and certified,
trustworthy scores to customers.
|
The fourth myth:
|
MIPS or MOPS means something.
|
Facts:
Here's a hint - if someone is quoting benchmark scores in "MIPS" (millions of instructions per second,
or "meaningless indicator of performance!) or "MOPS" (millions of operations per second), ask youself this:
what kind of instructions?
How "wide" are the instructions? And who certified the results to prevent cheating?
|
The fifth myth:
|
"de facto" benchmarks are industry standards.
|
Facts:
Others may claim to have "de facto industry standards",
but all that really means is that they created the benchmarks themselves
- without an industry-standards organization behind them!
"De Facto" is another term for "proprietary."
Then they have the nerve to "self-certify" the benchmark scores!
<smile>
This is like a form of recursion, but its not very funny if you have to make a major business
or technology decision based on "standards" dreamed up by one company.
Only the Embedded Microprocessor Benchmark Consortium (EEMBC) has over 50 members.
Only EEMBC uses public meetings with peer review of the benchmark source code, agreed-upon rules,
and an outside third party, ECL, who certifies the benchmark scores.
Do not be mislead - only EEMBC's benchmarks are derived from real application algorithms instead
of being simply fragments of code.
And only EEMBC has the full faith and trust of the industry, its customers, and the Press.
|
The sixth myth:
|
embedded benchmarking means writing the code in assembly language. Only that way can you compare apples to apples.
|
Facts:
Back in the old days, engineers and software programmers did write most
DSP, code, and
indeed most embedded code, in Assembly language. Compilers were inefficient,
often emitted bad code, applications were simpler, and processors were frankly
not all that powerful. The developer had to get the most from the processor,
so Assembly language was the way to do it.
Well, these days modern C compilers are generating, for the most part,
very good code. In fact, many companies are extremely proud of the fact that
you can get a fair fraction of the maximum performance from a processor just
using their C compiler and other software tools (commonly called the Software
Development Kit, or SDK).
As applications have increased in complexity, it makes more sense to
focus on good software engineering principles (to enable re-use, to allow
for collaborative team development, and to improve quality) and to use the
SDK's (software tools) to do so. Besides, the shear size of high-end embedded
applications makes the thought of coding it all in assembly language painful
to even think!
Face it - you may be a great programmer, but as the embedded world has
expanded to take up MOST of computing (its true!), the number of hot-shot,
bare-metal, code for 100 hours straight to bare silicon programmers is not
infinite. And besides - those folks are now CTO's of companies,
and don't have time to code themselves. Productivity depends upon quality
software tools to help the programmer extract performance from the part.
Modern DSP
architectures, especially
VLIW
and mixed
RISC/VLIW
architectures, are tremendously complicated. Having the C compiler do the
"dirty work" makes sense - you can always tune the algorithm in the hotspots
in assembly language (or just wait for a slightly faster clock speed of the
same part and save the months of pain).
This situation will likely increase as
DSP's
are blended with microcontrollers in modern SoC ("system on a chip") or SiP ("system in package") designs.
If you want to use configurable/scalable IP (intellectual property) cores for your next SoC,
you're not doing it in Assembler - the tools themselves emit the architecture, the instruction set, the Verilog or RTL,
and so on.
And Java? Well, embedded Java is taking off, too. That requires software
tools that emit Java bytecodes (usually), which means the programmer is writing
in Java, not Assembler.
The revolution on archtitecture in the late 1980's and 1990's was this: compiler is part of archtitecture.
RISC (reduced instruction set computers) and
VLIW's demand
excellent compilers to do more of the work. Benchmarks can also help compare
one compiler with another, just as you compare one processor with another!
Other companies may write the benchmarks themselves in an architecture's
assembly language, and then benchmarks it. How do you know that the programmer
on one architecture is as good as that from another architecture?
EEMBC's industry-standard, certified benchmarks offer a better way:
both "out of the box" (C language benchmarks) and "full fury" (full optimized)
benchmark scores are offered. If a company wants to let the compiler do the
work (or if a customer wants to SEE the effects of the C compiler), you go
to the "out of the box" scores. If you want to see how good the compiler
does compared with "full fury", you get both sets of scores and compare them.
If you want to see ultimate, theoretical performance, you just get the full-fury
numbers. Certified by ECL, of course - independently. And ECL NEVER, EVER
optimizes the benchmarks for the EEMBC members or a cmythnt. That would be
- unfair!
|
The seventh myth:
|
companies don't want to compete based on real numbers. They all want to cheat.
|
Facts: Well, maybe some. But the members of EEMBC have pledged their
support for fair, honest, above-board, certified benchmarking. They have
signed EEMBC License Agreements that are very strict. They abide by the EEMBC
Bylaws. They adhere to ECL's Certification Rules (or else they can't publish
benchmark scores, period). If you don't see benchmark scores for a particular
processor, demand to see certified benchmark scores from any EEMBC member.
If they still don't offer them, or benchmark and have those scores certified
within short order, contact ECL and we can do a Benchmark Study for you.
EEMBC members (compiler companies, processor companies, and
IP/core vendors) all
want your business, and are willing to compete on performance, price/performance,
performance/milliwatt, and quality of their software tools. |
| |
|
|
Check out our News page for more about about ECL and our Customers, Clients, and People…
 |
|
See how we are working for our customers by checking out the News
|
|
| Latest Benchmarks! |
 |
|
Compare benchmarks for: Telecom, Consumer, Networking, Office Automation, Automotive/Industrial, or Microcontrollers!
|
|
|
Home
