The Web Standards Project » Acid3

Acid3 receptions and misconceptions and do we have a winner?

lgunther — Fri, 03 Oct 2008 03:37:42 +0000

Acid3 is probably the most visible thing that WaSP has done the last year. When Google Chrome was launched almost every review included our little test as an indicator of standards support. It is often mentioned in blogs and articles. Now the Surfin Safari blog has announced that the team behind Webkit considers that they have passed the test in every aspect. And no doubt this is a great achievement. Congratulations to the Webkit team, but even more we would like to congratulate the average web user – who in a few years thanks to our test we hope will get a better experience!

What exactly does it mean to pass the Acid3 test?

There has been some confusion about the test and its importance. Some people have been saying things like ”my browser does not pass the test and I have no problems using it”. Quite a few other people seem to think that Webkit and Gogi (Opera’s internal build) passed the test already in March – despite the fact that neither team has made this claim.

To answer these misconceptions we need to address the issue of what exactly is being tested and how. The main part of test is automated through JavaScript, a sort of test harness that runs 100 subtests. Getting a score of 100 is not the same as passing Acid3 – a common misconception, or perhaps an oversimplification.

Many subtests are high on a developer’s wish list: Full CSS 3 selectors support, media queries, SVG fonts. Admittedly a few others test edge cases and more esoteric features – but the test was supposed to be a significant challenge!

The second part is a rendering test. Some of the scripted subtests produce results that affect the rendering, but there are also rendering issues that come in addition to these. Some of them are high on many designers’ wish list: Text shadow, downloadable fonts, and display: inline-block.

The third test is the so called “smoothness” criterion. It is basically a speed test. No subtest may take too long – and especially subtest 26 is challenging. Compared to Slickspeed, Sun Spider, the V8 test suite or Dromaeo Acid3 is not so thorough. It will give some indication of a browsers speed, though.

This is exactly as planned. Acid3 was not meant to be the one and only indication of a browser’s performance. In fact many other test suites are far more important. (We provide links to some of them below.)

Testing is really important. Without tests that check how well a certain browser follows standards, i.e. applies mark up and displays the result correctly, we can never guarantee an open, fully interoperable web.

A highly visible test like Acid 3 hopefully helps to promote such interoperability. One can also hope that all the other tests will receive the attention they deserve. Writing them is not a glamorous task, but highly essential.

Apart from improving its support for CSS in its browser, Microsoft has contributed 2524 test cases to the CSS 2.1 test suite. For that they deserve credit!

We all know that Internet Explorer currently lag a bit behind the other browsers in standards compliance. Indeed they are last of the big ones to pass Acid2 and they fail Acid3 more than any other browser. But can we declare Webkit as the best rendering engine now that they pass it?

Of course not. Since Acid3 is only one indicator of many. Webkit’s achievement is great – and there are many other really exciting things they are pioneering, like CSS transitions and transformations. And with Squirrelfish Extreme JavaScript performance looks really exciting as well.

In other regards Opera is a clear leader. It is the only browser that supports more than 90 % of the SVG test suite. It is the only browser that implements Web Forms 2.0, currently being merged into HTML 5. They supported media queries and SMIL long before Acid3 came out.

Gecko (with Spidermonkey) is no longer an underdog. Besides the fun of meeting the technical challenge it is not hard to guess that the Webkit team rushed to pass Acid3 also for marketing reasons – that they perhaps need a bit more than Mozilla. Mozilla concentrated on releasing Firefox 3 before Acid 3 received any real attention. Now that they are working on it they are impressive in another way, compared to Webkit. Looking at the discussions for bug 410460 and its related bugs, it is clear that any improvement must be rock solid. Work often continues even when a particular feature is good enough for Acid3.

In fact, there is actually one open issue still in Acid 3 that might temporarily cause Webkit to become incompliant again. http://lists.w3.org/Archives/Public/www-style/2008Sep/0218.html. I rest assured that a fix probably already is being made, though.

Perhaps one can compare this to a race where you are supposed to run a distance, with a bucket of water. One competitor crosses the finishing line first, the other, on the other hand, has not lost a single drop from his bucket. Both have done great. (By the way, internal builds of Firefox get a score of 97 now, and downloadable fonts work on Windows and Mac.)

In the end the winner is neither Webkit, Opera, Mozilla nor Microsoft, but developers who get more powerful features to work with and more consistency between browsers. And that means that in the long run they are able to focus on user experience, not browser shortcomings. This means that the true winner of Acid3 is anybody who surfs the web.

Some other test suites for your review:

Acid3 Passed in 23 Days!

Kimberly Blessing — Mon, 07 Apr 2008 06:19:28 +0000

Just over a month ago, WaSP announced the Acid3 Browser Test. At the time, we knew that some of the browser development teams were aggressively working to pass the test. What we didn’t know was how long it would take for a browser to pass the test.

“When we released Acid2, the first browser passed it in about a fortnight. Acid3 is orders of magnitude more complicated. I really didn’t expect to see passing browsers this side of August, let alone within a month,” wrote Ian Hickson, the architect of the test, on his blog.

Well, now we know: It took under a month for not one but two browsers to pass the test. On March 26, the WebKit team announced that their public build produced a 100/100 result; Opera announced that they had an internal build which passed on the same day and they released the build on March 28. Congratulations to both teams!

In the past month we’ve received plenty of feedback and questions about the test. Thanks to the help of readers and browser developers, some bugs were identified and resolved. To learn more about the changes that were made as a result, read Ian’s post which lists the major changes.

Acid3: Putting Browser Makers on Notice, Again.

drewm — Mon, 03 Mar 2008 10:00:17 +0000

It’s been three years since we told browser makes that we want to see them smile, but now we wanna hold their hand.

Acid3 goes beyond the CSS tests implemented by Acid2 and tests a browser’s DOM Scripting capability, as well as continuing to probe visual rendering of CSS, SVG and webfonts. Further information can be found on the Acid3 page, in the press release, or you can just go ahead and take the test.

We know that work is already underway based on the Acid3 previews, but time will tell which browser is the first to pass all 100 tests fully, and with default settings. This is going to be interesting.

Acid3 nearing completion

Kimberly Blessing — Tue, 05 Feb 2008 14:14:04 +0000

As noted previously, Ian Hickson has been hard at work on the Acid3 browser test. We’re happy to report that his work is nearly done!

Acid3 is a complex test compared to Acid1 and Acid2. The Web is increasingly becoming a platform for application development, so Acid3 tests many of the DOM2 and ECMAScript specifications, in addition to markup languages, CSS, SVG, and others. A complete list of specifications tested will be available when the test launches, along with a guide to the test.

Due to the complexity of the test, we’re asking for a final round of feedback to ensure its fairness and accuracy. Please review the test and submit your comments here or email them to us.

What’s the best test for Acid3?

Kimberly Blessing — Wed, 16 Jan 2008 08:17:07 +0000

Perhaps you expected this announcement, given the news that IE8 passes Acid2. Or maybe it already seems like old news, if you caught the recent Slashdot post. In either case, the news is true — Acid3 will be coming soon!

Whereas Acid2 tested static HTML and CSS, Acid3 will focus on ECMAScript and DOM — the dynamic side of the Web. While it will include a few rendering tests to make things look interesting, the core of the test will be 100 scripting subtests.

And that is where you can contribute! Ian has compiled 84 of the 100 scripts but is seeking 16 additional tests. If you’re up to the challenge, read the criteria, create your test, and submit it — within the next five days!

As with Acid2, the Web Standards Project will play host to Acid3… so stay tuned for the final test, and the play-by-play on the browsers’ progress!