Futuremark's Approved NVIDIA Drivers For 3DMark03

After posting the NVIDIA GeForce FX Go5200 GPU review, I received an e-mail from Futuremark. They asked me to remove the 3DMark03 results I posted because the NVIDIA driver (Rev. 53.03) I used was not approved by Futuremark. I was to rebenchmark using Futuremark-approved drivers instead. Currently, that would be the Rev. 52.16 driver.

For purpose of transparency, here is the e-mail I received :-

Hi Adrian,

I am in a somewhat ambiguous situation, where on the other hand I am very pleased to see that you use 3DMark03 in your reviews (http://www.techarp.com/default.aspx?location=3&var1=74&var2=0), but on the other hand, I noticed that you use it with drivers that we have not approved for 3DMark03 usage.

For us it is of paramount importance to make sure that when you use our products, you will get results that you and your readers know they can trust.

We want to support your work by helping you to continue producing high-quality reviews as you have been doing.

To view which drivers we have approved to be used with 3DMark03, please look here:

http://www.futuremark.com/community/drivers/?approved

Only by using the drivers we have listed (+ build 340 of 3DMark03), you get a fully valid and comparable 3DMark03 result.

We also posted a "Reviewer's Guide" how to use 3DMark03 the correct way. It can be viewed here:

http://www.futuremark.com/companyinfo/reviewers_guide_3dmark03.pdf

As you see, we kindly need to ask you to take down the 3DMark03 results obtained with drivers that we have not verified and instead use drivers that are in the approved drivers list.

If you have any questions/comments regarding this, please feel free to email me about it!

My Doubts

When I went to check Futuremark's website, I found that the latest (and ONLY) driver approved by them was the NVIDIA Rev. 52.16 driver. Curiously, there was this little note posted just below the links to the driver. For your convenience, I pasted it below and highlighted the pertinent part :-

The NVIDIA ForceWare 52.16 Drivers have been tested with the GeForce3 series, GeForce4 series and the GeForceFX series. The drivers have not been tested with the MX-series! There are no Win9x or WinME WHQL drivers available supporting all the GeForce series. The 52.16 drivers have 3DMark03 specific optimization for the Pixel Shader 2.0 test and that score is solely comparable between nvidia cards.

Let's recall what Futuremark wrote to me :-

Only by using the drivers we have listed (+ build 340 of 3DMark03), you get a fully valid and comparable 3DMark03 result.

Did I miss something here or do you also see the discrepancy?

Futuremark asked me to remove the Rev. 53.05 results because they were not from a certified driver and presumably open to hanky-panky by NVIDIA's driver team. This apparently makes them unsuitable for comparisons against results from other GPUs.

But why recommend the Rev. 52.16 driver as the only certified NVIDIA driver? As they have stated on their website, the Rev. 52.16 drivers have "3DMark03 specific optimization for the Pixel Shader 2.0 test and that score is solely comparable between NVIDIA cards"? The Rev. 52.16 drivers clearly cannot be used to provide a valid and comparable 3DMark03 result.

But if 3DMark03-specific optimizations are kosher, then why isn't the new Rev. 53.03 driver allowed? Or any other driver for that matter. Surely NVIDIA would have included the same Pixel Shader 2.0 optimization into the newer Rev. 53.03 driver. So, what exactly are Futuremark's criteria for drivers to be approved?

I wrote to Futuremark asking them those questions. Maybe they know something we don't. Let's see what they have to say...