Acid3 receptions and misconceptions and do we have a winner?

By Lars Gunther | October 2nd, 2008 | Filed in Acid3, Browsers, Bugs

Acid3 progress and what it really means.

Acid3 is probably the most visible thing that WaSP has done the last year. When Google Chrome was launched almost every review included our little test as an indicator of standards support. It is often mentioned in blogs and articles. Now the Surfin Safari blog has announced that the team behind Webkit considers that they have passed the test in every aspect. And no doubt this is a great achievement. Congratulations to the Webkit team, but even more we would like to congratulate the average web user – who in a few years thanks to our test we hope will get a better experience!

What exactly does it mean to pass the Acid3 test?

There has been some confusion about the test and its importance. Some people have been saying things like ”my browser does not pass the test and I have no problems using it”. Quite a few other people seem to think that Webkit and Gogi (Opera’s internal build) passed the test already in March – despite the fact that neither team has made this claim.

To answer these misconceptions we need to address the issue of what exactly is being tested and how. The main part of test is automated through JavaScript, a sort of test harness that runs 100 subtests. Getting a score of 100 is not the same as passing Acid3 – a common misconception, or perhaps an oversimplification.

Many subtests are high on a developer’s wish list: Full CSS 3 selectors support, media queries, SVG fonts. Admittedly a few others test edge cases and more esoteric features – but the test was supposed to be a significant challenge!

The second part is a rendering test. Some of the scripted subtests produce results that affect the rendering, but there are also rendering issues that come in addition to these. Some of them are high on many designers’ wish list: Text shadow, downloadable fonts, and display: inline-block.

The third test is the so called “smoothness” criterion. It is basically a speed test. No subtest may take too long – and especially subtest 26 is challenging. Compared to Slickspeed, Sun Spider, the V8 test suite or Dromaeo Acid3 is not so thorough. It will give some indication of a browsers speed, though.

This is exactly as planned. Acid3 was not meant to be the one and only indication of a browser’s performance. In fact many other test suites are far more important. (We provide links to some of them below.)

Testing is really important. Without tests that check how well a certain browser follows standards, i.e. applies mark up and displays the result correctly, we can never guarantee an open, fully interoperable web.

A highly visible test like Acid 3 hopefully helps to promote such interoperability. One can also hope that all the other tests will receive the attention they deserve. Writing them is not a glamorous task, but highly essential.

Apart from improving its support for CSS in its browser, Microsoft has contributed 2524 test cases to the CSS 2.1 test suite. For that they deserve credit!

We all know that Internet Explorer currently lag a bit behind the other browsers in standards compliance. Indeed they are last of the big ones to pass Acid2 and they fail Acid3 more than any other browser. But can we declare Webkit as the best rendering engine now that they pass it?

Of course not. Since Acid3 is only one indicator of many. Webkit’s achievement is great – and there are many other really exciting things they are pioneering, like CSS transitions and transformations. And with Squirrelfish Extreme JavaScript performance looks really exciting as well.

In other regards Opera is a clear leader. It is the only browser that supports more than 90 % of the SVG test suite. It is the only browser that implements Web Forms 2.0, currently being merged into HTML 5. They supported media queries and SMIL long before Acid3 came out.

Gecko (with Spidermonkey) is no longer an underdog. Besides the fun of meeting the technical challenge it is not hard to guess that the Webkit team rushed to pass Acid3 also for marketing reasons – that they perhaps need a bit more than Mozilla. Mozilla concentrated on releasing Firefox 3 before Acid 3 received any real attention. Now that they are working on it they are impressive in another way, compared to Webkit. Looking at the discussions for bug 410460 and its related bugs, it is clear that any improvement must be rock solid. Work often continues even when a particular feature is good enough for Acid3.

In fact, there is actually one open issue still in Acid 3 that might temporarily cause Webkit to become incompliant again. http://lists.w3.org/Archives/Public/www-style/2008Sep/0218.html. I rest assured that a fix probably already is being made, though.

Perhaps one can compare this to a race where you are supposed to run a distance, with a bucket of water. One competitor crosses the finishing line first, the other, on the other hand, has not lost a single drop from his bucket. Both have done great. (By the way, internal builds of Firefox get a score of 97 now, and downloadable fonts work on Windows and Mac.)

In the end the winner is neither Webkit, Opera, Mozilla nor Microsoft, but developers who get more powerful features to work with and more consistency between browsers. And that means that in the long run they are able to focus on user experience, not browser shortcomings. This means that the true winner of Acid3 is anybody who surfs the web.

Some other test suites for your review:

Your Replies

#1 On October 3rd, 2008 4:05 am Chris Mills replied:

Very interesting post – thanks Lars.

One point that came to mind while reading this – the “smoothness” test seems bizarre for inclusion in Acid 3. Surely this is not really a test of standards compliance, and more a test of hardware performance?

At Opera, one of our main focuses has always been making our browsers as slimline and non-resource intensive as possible, so they will work on less powerful machines, but machines like this would probably not be able to render such animations smoothly, regardless of the performance of the browsers they are running.

Methinks performance should be left out of such tests in future?

#2 On October 3rd, 2008 3:55 pm James Brown replied:

Gecko (with Spidermonkey) is no longer an underdog. Besides the fun of meeting the technical challenge it is not hard to guess that the Webkit team rushed to pass Acid3 also for marketing reasons – that they perhaps need a bit more than Mozilla. Mozilla concentrated on releasing Firefox 3 before Acid 3 received any real attention. Now that they are working on it they are impressive in another way, compared to Webkit. Looking at the discussions for bug 410460 and its related bugs, it is clear that any improvement must be rock solid. Work often continues even when a particular feature is good enough for Acid3.

Wow, that’s some great WebKit-bashing right there! Provide excuses for Mozilla, question WebKit’s motivations. Just what we’d expect from the impartial Web Standards Project!

#3 On October 3rd, 2008 4:23 pm James Brown replied:

Gecko (with Spidermonkey) is no longer an underdog. Besides the fun of meeting the technical challenge it is not hard to guess that the Webkit team rushed to pass Acid3 also for marketing reasons – that they perhaps need a bit more than Mozilla. Mozilla concentrated on releasing Firefox 3 before Acid 3 received any real attention. Now that they are working on it they are impressive in another way, compared to Webkit. Looking at the discussions for bug 410460 and its related bugs, it is clear that any improvement must be rock solid. Work often continues even when a particular feature is good enough for Acid3.

Wow, that’s some great WebKit-bashing right there! Provide excuses for Mozilla, question WebKit’s motivations and then insult their engineering. Just what we’d expect from the impartial Web Standards Project!

#4 On October 3rd, 2008 4:36 pm Peter Kasting replied:

I too applaud the work all the browsers have done in improving, not just Acid3, but their performance and standards support in general.

I think the article is a bit clumsy in places. It looks like the author meant to congratulate all engines rather than heaping praise only on one, but in the process made some very questionable implications. Like the Mozilla developers are doing with Gecko, the WebKit developers have made long, continuing efforts to “do the right thing” and implement “rock solid” support for features, above and beyond what’s required for Acid3. I think the implication of some of the later paragraphs that this is a case of “fast vs. correct” is erroneous. I think Gecko’s achievements are impressive without needing to also belittle WebKit’s to do it; everyone is getting better here.

#5 On October 4th, 2008 9:45 am Lars Gunther replied:

Since I am not a core member – and only a recent member of EduTF – I do not know how much this was discussed before the test was launched. I agree though.

Speaking from a pedagogic perspective it is also very clear that our Acid tests should have very clear visual clues to failing or not. I.e. “smoothness” is too subjective. I have linked to other test suites for JavaScript performance. They do a much better job at assessing script speed. WaSP Acid tests should be boolean by nature. They are more a lobbying and teaching effort, than anything else.

BTW, there is a fourth part to test I did not mention in the article: The favicon part. I did not mention it on purpose, since it is the least important and not very well done. I.e. you can pass by such a simple measure as not supporting favicons at all.

#6 On October 7th, 2008 5:24 pm Max Design - standards based web design, development and training » Some links for light reading (8/10/08) replied:

[...] Acid3 receptions and misconceptions and do we have a winner? [...]

#7 On October 8th, 2008 12:36 pm Lars Gunther replied:

@James and Peter:

Every member of WaSP speaks for him- or herself. As I said in the article I have been following both teams efforts and both are impressive. I do however think that passing Acid3 was very much a goal in itself for the Webkit team and it clearly shows when you compare bug 17077 for Webkit and bug 216462 for Gecko. It also shows in how they initially handled bug 17086 for Webkit.

If I was bashing anyone it was those fans of Webkit who use Acid3 in a simplistic way. Some people seem to think that it is the one true indicator of a rendering engines relative merits.

For the record: I like Webkit. In fact I have filed bugs just to help them improve and most probably will do it again.

@myself:

Comment 7 for gecko bug 454226 confirms that the CSS WG will ask Ian Hickson to update the test. It may mean that Webkit will need one more update before it truly passes the test.

#8 On October 9th, 2008 11:24 am Acid3 test meanings analysis and browser compliance « BeezNest Open-Source specialists replied:

[...] Read their article [...]

#9 On October 27th, 2008 8:27 am wesele replied:

#10 On October 27th, 2008 11:45 am wesele replied:

I do however think that passing Acid3 was very much a goal in itself for the Webkit team and it clearly shows when you compare bug 17077 for Webkit and bug 216462 for Gecko. It also shows in how they initially handled bug 17086 for Webkit.

#11 On October 27th, 2008 1:02 pm isp replied:

#12 On November 1st, 2008 5:04 pm Best of Week (+/-) #16 | Insel der Engel' replied:

[...] Acid3 receptions and misconceptions and do we have a winner? [...]

#13 On November 4th, 2008 5:28 am Max Design - standards based web design, development and training » Some links for light reading (4/11/08) replied:

[...] Acid3 receptions and misconceptions and do we have a winner? [...]

#14 On November 4th, 2008 4:08 pm The Website Marketing Group » Blog Archive » Useful website Developer reading replied:

[...] Acid3 receptions and misconceptions and do we have a winner? [...]

#15 On November 6th, 2008 7:53 am Restaurant replied:

#16 On November 16th, 2008 1:51 pm منتدى replied:

thanks lars

#17 On November 16th, 2008 8:44 pm Web Design - ZachBrowne.com replied:

[...] Does this mean we should ditch Firefox, IE and all the other browsers in favour of Safari or Chrome, well no, and that’s what Lars Gunther is talking about in his article over at WaSP. [...]

#18 On November 23rd, 2008 12:35 pm Look what my RSS dragged in #7 November 2008 replied:

[...] Read: Acid3 receptions and misconceptions and do we have a winner? [...]

#19 On December 8th, 2008 5:39 am izen replied:

Opera win! Opera 4 ever! I know it!

#20 On December 19th, 2008 6:20 pm Joaquín Núñez» Blog Archive » acid3 replied:

[...] se liberó opera 10 alpha 1 no me demoré en hacer eco de que, esta nueva versión superaba el test acid3, pero que es el acid3 para quienes no lo [...]

#21 On December 22nd, 2008 2:02 am Kev replied:

Every browsers have made their own improvements but it takes time for everyone to catch up to pass the test… Thanks lars for sharing

#22 On December 28th, 2008 3:45 pm my-music-school-echo replied:

I believe all the browsers are improving their performance and standards support and not just acid3 for the good of all web users present and future. Well, for opera 4, several bugs have been introduced into Opera since v3.6 accidently or in design.

Working together for standards The Web Standards Project

Acid3 receptions and misconceptions and do we have a winner?

Your Replies

Post a Reply