HOW DO WE TEST
Introduction
MusicPlayers.com takes product testing seriously. Very seriously. In fact, it’s one of the most important things we do for our readers. One of the biggest differences between MusicPlayers.com and most music magazines (whether print or online) is that we have taken a labs-based approach to standardizing the review process, while still including the essential subjective review information that is so important to our readers when making purchasing decisions about musical instruments and recording gear.
By implementing a specific system for writing product reviews, we ensure that all of our reviews are directly comparable from one product to another regardless of who authored the review. You can take two related product reviews, place them side by side, and get the same important information from each to make an informed purchasing decision.
With many magazine reviews, different freelance writers bring their own unique style and perspective to product reviews and, lacking a standardized testing methodology, the 20-year old, punk-rock loving, head banger writes a very different guitar review than the 40-something, jazz loving, classically trained player. How can a reader objectively compare two different guitars (or other instruments) reviewed in the same magazine written in completely different styles? It amazes us that more magazines haven’t standardized their approach to product reviews.
Without an objective scoring system, how do you determine which instruments, plug-ins, and recording tools are worthy of an editor’s choice or magazine award? It’s purely subjective, based on someone’s gut feeling about a product, perhaps discussed among some peers, but without factual data to back it up. Our objective ratings system has revealed that commonly, while you may subjectively think a product is excellent, it really only rates as very good, and not quite at the level of excellence needed to be worthy of an award. Without the numerical data backing up your reviewer’s specific feelings about an instrument, your mood on a given day could mean the difference between award-winning and not. We wouldn’t accept that, and serious musicians shouldn’t have to either.
Our Testing Methodology
Have you noticed that all of our reviews start out with a summary that highlights the pros and cons of the product at hand? Musicians are busy and don’t always have a lot of time to read the full-length review. Or, a few weeks after reading a particular review, a musician wants to refer back to that review for the quick summary. With this in mind, we provide you with the important points up-front. You can then choose to read on for further details or race out to the store to buy your new equipment.
What readers may not realize is that this section is actually written last, after all the results of a product review are in.
Products are reviewed for their success or failure in the following areas:
Features
We evaluate the capabilities of a product and consider whether or not the feature set is similar, better, or worse than competing products. We also consider whether or not the feature set is significant regardless of competitive products.
Usability
This section varies in scope based on the product. For an electronic product such as a synthesizer or music workstation, we consider things like ease of use, portability and quality of construction. With a piece of software, we’re looking at things like how well it integrates with other products (in addition to ease of use). For a guitar, bass, or drum product, we’re evaluating its quality of construction and playability.
Sound
For products that produce sound, this is pretty self-evident. How does an item sound both on its own and compared to similar products? For example, a modeling amp might sound good, but compared to a real tube amp it may fall short.
Documentation & Support
Documentation may not matter much with a cymbal but, for most products, instructions are often times essential for getting the most out of a given product. Sometimes we need technical support or we visit a manufacturer’s website for assistance and additional information, so we factor this into the rating, too.
Price
This rating is influenced by the competition for a particular product. $3,000 for a single-channel amp would be extremely high for a mass-produced product when typical tube combos are selling for $1,100, but reasonable for a hand-wired boutique amplifier. Likewise, $500 for a soft synth plug-in when the competition is selling them for only $199 would be a problem.
Rating Values
Products receive a rating in each category above from one to four stars, with half-stars allowed.
Excellent | |
Good | |
Below Average | |
Poor |
We determined weighted values for each of the product categories that are evaluated. A lot of thought went into determining how important each of the categories is when evaluating musical equipment, and our senior editorial staff agreed upon the following values.
Guitars, amps, keyboards,
plug-ins, signal processors, etc. |
Non-musical equipment, controllers,
racks, etc. |
Drum
Cymbals |
|
Category |
Percentage
|
Percentage
|
Percentage
|
Features |
20
|
35
|
30
|
Usability |
25
|
35
|
not applicable
|
Sound |
25
|
not applicable
|
50
|
Documentation & Support |
10
|
10
|
not applicable
|
Price |
20
|
20
|
20
|
The overall rating for an item is then calculated based on the weighted values listed in the table above to result in an overall product rating. For outstanding products, MusicPlayers.com issues its coveted WIHO Award… Wish I Had One!
3.6 stars or better | Outstanding (WIHO Award) |
3 stars or better | Worth considering |
2 stars or better | Suited to specific needs |
1 star or less | Not recommended |
If you feel that one of these categories is of more or less importance to you, you can put these values into a spreadsheet and calculate your own overall rating for a product based on your own specific needs. Always make sure your percentages total 100!
Evaluation Short-List
All of our product reviews contain an evaluation short-list identifying similar products that our writers and editors believe are comparable products. We call it a short-list because it isn’t all-inclusive. It is just a random sampling of products we think would be considered by the person interested in our reviewed product. These products are selected for the short-list based purely on our knowledge of the music equipment market – manufacturers have no influence on these listings.