Update: Here's my first attempt at applying the review criteria I compiled below to an actual theme review: meet the Chipmunk Theme.
I want to make more WordPress theme reviews on this blog, but I'm still not sold on what a WordPress theme review should consist of. In this post, I set out my list of review criteria, but I know it's far from perfect. I would appreciate any feedback you might have as a fellow reviewer or as someone who might appreciate reading a review before buying/installing a WordPress theme.
- A. What makes a useful review?
- B. My WordPress theme review criteria
- Review introduction
- Rubric 1: Installation
- Rubric 2: Usability
- Rubric 3: Speed
- Rubric 4: Support
- Review conclusion
- A note on transparency
- C. Final Word
A. What makes a useful review?
When it comes to the overall purpose of a review, I think there are two general questions that need addressing: (1) which criteria are most useful to prospective users of the theme? (2) Which criteria will allow the reviewer to be as transparent and impartial as possible? To answer the first question, let's take a step back and look at how product reviews work in other contexts. We'll return to the question of transparency at the end.
The Amazon model
The most widely read user reviews of any website I know are customer product reviews on Amazon. Take, for example, the reviews of Lisa Sabin-Wilson's popular book, WordPress for Dummies. Each one consists of a star rating, a title and an open comment box. Readers can see whether the reviewer purchased the item and vote on how helpful the review content was in assisting them with their purchase.
The authority of this type of review is based on two factors: the first is whether the user has actually purchased and therefore used the product. This falls under the “usability” heading in my review criteria below.
The second is whether prospective buyers found the reviewer's opinions useful. In the case of a bias or contentious review, readers have the ability to leave comments and argue against the reviewer. This is a very nice “check and balance” system that keeps reviews relevant and (relatively) trustworthy.
In the case of WordPress theme reviews on this blog, I am also going to introduce a “was this review helpful?” thumbs up/down system, and of course I will leave comments open.
The WordPress.org model
Let's jump back to the WordPress context. In the WordPress.org theme repository, which is by far the largest collection of free WordPress themes currently available for download, there's a similar type of rating system in use.
However, unlike Amazon, the WordPress review system doesn't have the ability to verify whether the reviewer has actually used the theme under review, or whether he/she is posting an arbitrary rating. This weakens the authority and credibility of the reviews.
As designer Mel Choyce recently pointed out, this lack of specificity in the WordPress theme review system makes the ratings arbitrary and is of little use to newcomers. Choyce suggests a more “granular” approach, with a wider range of review criteria that correspond to real world usage issues. I think that's right and that's the general direction I'm heading in with reviews on this blog too.
The “expert” review model
The last type of review I want to mention, are reviews written by “experts,” often for online newspapers and magazines. One of my favourites in this field is Digital Photography Review, a long-running site that reviews digital cameras.
As you can see in the image above, expert reviews tend to be split into multiple sections and span multiple pages. In the DPReview model, they start with an overview of the camera, looking at its design and feature set, they move on to benchmark testing of image and video quality, and they end with samples.
They give each camera a bronze, silver, gold or platinum rating based on a long list of criteria. They also include a list of “pros” and “cons” of the camera and a list of its potential uses.
In this case, what gives the review its authority is on the one hand the reviewer's expert knowledge, and on the other hand the level of detail from technical information to benchmark testing and usability.
What I learnt from these models
What works well with the Amazon model is the ability of the reviwer to give personal “real world” insight into using the product and the ability of the community to hold the reviews to account in terms of accuracy and usefulness.
What works well with the expert review model is the level of technical detail, the benchmark testing and the ability to compare the product with similar items in the same field. However, I think the level of detail that we see in the DPR reviews would be overkill for a WordPress theme review. Digital cameras and WordPress themes have completely different degrees of technicality.
So, I think we need a combination of both models. WordPress theme reviews should be written from a “hands on” perspective with as much practical information about how the theme actually functions (vs. the developer's ideal view of how it functions) as possible.
The review should cover the positive and negative aspects of the theme. The negative aspects should be backed up with concrete examples and there should be scope for readers to challenge the validity of the review through a rating and comment system. The review should also give the reader a clear idea of the sorts of applications the theme could be used for and it might also reference similar themes in the field.
B. My WordPress theme review criteria
So here are my suggestions of criteria for WordPress theme reviews. I have narrowed it down to 6 steps, including an introduction and conclusion, and 4 main rubrics: installation, usability, speed, and support. Each main rubric is given a rating from 1 (lowest) to 5 (highest). Let's take a quick look at each part.
The opening section of each review will give the reader a quick impression of what the theme consists of and what it can do. It will describe the theme's distinguishing features, its looks (I was debating on whether to include this as one of the main rubrics, but since aesthetics is such a subjective affair, I have left it in a secondary position) and the developer's expectations for the theme.
Rubric 1: Installation
This rubric has two goals: first, to inform the reader on how easy or difficult the theme is to install. Is this a one-click job or does it require multiple steps? The second goal is to give the reader an idea of how easy or difficult it is to get the theme looking and working like the developer's demo site. So many themes fail on this front. The theme looks brilliant when you try the demo, but looks terrible when you install it.
|Difficult to install and/or setup without help from the developer.||Installation and setup require extensive consultation of documentation.||Installation and setup require a bit of tweaking, but nothing unusual.||Installation and setup were easy, but still require some basic||Installation and setup were a breeze.|
Rubric 2: Usability
This rubric looks at the day-to-day operation of the theme. The main concern here is the efficiency of the admin side of the theme after installation and setup are complete. So for example, does the theme have too many configuration options that makes its use overwhelming? Or does its UX merge seamlessly with the WordPress architecture (post/page composition including Gutenberg, widgets, theme options, plugin operation).
|The theme has a steep learning curve.||The theme requires some consultation of the documentation and some trial and error to operate smoothly.||The theme functions as expected. Yes, there is a bit of tweaking, but nothing unusual.||The theme is easy to operate and does what it says on the tin.||The theme is so easy to use, even my grandmother can use it.|
Rubric 3: Speed
In this rubric, I will do some benchmark testing with two aims in mind. First to assess the speed of the theme in both desktop and mobile environments. I will use Pingdom and GTMetrix for the former and Google Mobile Speed Test for the latter. The second is to assess the overall code quality of the theme with its SEO potential in mind. I will use the W3C Markup Validation check and the theme check plugin to do this.
|Very slow loading time and very poor coding quality.||Slow loading time and poor coding quality.||Average loading time and standard coding quality.||Fast load times and no code issues.||Super fast load times and no coding issues.|
Rubric 4: Support
The final rubric in the review is theme support. This is one of the most important elements in choosing a WordPress theme, particularly a paid theme. While I will not be able to assess the long-term credibility of developer support, I will use available information based on reviews and comments from current theme users to assess the support situation. I will look at how quickly the developer responds to support issues, whether there is an adequate support system in place, and whether there is adequate theme documentation and resources.
|No support or completely dysfunctional support system.||Limited support leading to an unsatisfactory end user experience.||Adequate support.||Good support, fast and efficient.||Excellent support and resources.|
The conclusion will do four things. It will give a list of the theme's pros and cons. It will assign a rating for each of the 5 main rubrics in the review resulting in a minimum score of 1 and a maximum of 10. It will list the potential uses of the theme. Finally, it will provide links to similar themes in the field.
A note on transparency
I want to add a word here about transparency and disclosure here. There are so many WordPress theme reviews and list posts which exist solely for the purpose of generating affiliate money. I have no problem with affiliate links per se, but when it comes to reviewing something I think it's crucial for the reviewer to be up front about his/her financial relationship with the product. Also, in the case of premium theme reviews, I think it's important for the reviewer to disclose whether he/she purchased the theme independently or whether the developer provided a review copy. In both cases, the arrangement should feature as an integral part of the review.
C. Final Word
So that's where I'm at so far in establishing WordPress theme review criteria. As I said at the beginning of the post, this is still a work-in-progress and I am sure there are many things I have overlooked or not fully thought through yet.
In that sense, I would be hugely grateful for any feedback you might have about any of this. So please feel free to leave critical comments and I'll be happy to revise my criteria accordingly.
The next step will be to apply these criteria to an actual review and see whether it works. Just as the development of a WordPress theme comes with updates and bug fixes, so too will these review criteria change as I streamline the review experience. Thanks so much for reading!