Clean Label Project: Lots of Unfulfilled Dog Food Review Promises

In the comments of last week’s blog post, several readers asked us what we thought of the pet food ratings by the Clean Label Project (CLP), a “nonprofit focused on health and transparency in consumer product labeling.”

We wish we could herald the work of the group, or even join forces with it to test foods, kick ass, and take names. The Clean Label Project’s stated vision, mission, and values sound terrific.

We can’t critique any of its work other than its dog food commentary, which, in our opinion, is currently so flawed as to be without any practical use.

Also, glaringly, Clean Label Project shoots itself in the foot by offering dog food manufacturers a certification program, which is described most fully here:

“Clean Label Project’s Certification program is for the highest performing products. Not only did the individual products of these brands perform exceptionally well in our initial unannounced sampling and testing, they have voluntarily signed on to having the Clean Label Project continue to randomly sample and test a subset of their products to ensure ongoing compliance with Clean Label Project standards.”

Can we trust the Clean Label Project’s dog food ratings?

It’s not stated anywhere, but our assumption is that the program is fee-based – and as soon as a manufacturer is paying for its foods to be rated, its foods will likely receive a good rating. This sort of structure is nearly impossible to implement without at least the appearance of conflict. Only a couple of dog food manufacturers are identified as having joined the certification program, and lo and behold, they have five-star ratings and appear on the top of the first page of five-star products.

But we have a lot of other nits to pick with the Clean Label Project ratings. We don’t usually comment on other sites or individuals who rate or review dog food, but we were compelled to do so in this case out of sheer disappointment. When we first heard of the Clean Label Project, we hoped that the organization had somehow managed to fund a significant number of validated, independent lab tests of dog food, searching for contaminants and nutrient levels that were out of spec, and plainly report the results.

Instead, the site assigns “ratings” of the products tested. These ratings are described as based 60 percent on the detected presence of heavy metals (the explanation calls out arsenic, cadmium, lead, and mercury); 20 percent on the detected presence of by-product contaminants (acrylamide and mycotoxins), 20 percent on the detected presence of “process contaminants” (here, the site includes ammeline, amelide, antibiotics, BPA, melamine, pesticides, and cyanuric acid).

Unfortunately, there is also an embarrassing little math problem in the graphic that explains the Clean Label Project evaluation process, explaining that the CLP also weighs “nutritional superiority” as 20 percent of the food’s score, leading to a potential score of 120 percent. Or maybe they meant that the scores for the contaminants overlap?

Let’s put aside the “nutritional superiority” for a moment and go back to the contaminants. There isn’t any explanation of the difference between a “by-product contaminant” and a “process contaminant,” and only some explanations of how these things might come to be present in food. 

As just one example, the site names acrylamide as a “by-product contaminant” and starts its description of the substance this way: “Acrylamide is an extremely toxic chemical used in manufacturing.” We don’t know about acrylamide being “used” in manufacturing of any food; the most important thing to know is that it is created when starchy food ingredients are cooked in at high-enough temperatures. Most notoriously, it gets created when potatoes are made into potato chips and French fries; these foods have the highest levels of acrylamide in commonly consumed human foods. To be fair, the site adds that acrylamide “is used in the treatment of water, and it can sometimes wind up in soil through that route. It is also found in tobacco, and made naturally when certain plants (like potatoes) are cooked.”

But that’s burying the lede. It’s been generally acknowledged that acrylamide is produced during the cooking process of dry dog foods. But, to our knowledge, no one has attempted to quantify the amount of acrylamide that is proven to be harmful to dogs. If we knew which foods contained the most acrylamide and which foods contained the least, would we use that information when selecting a food for our dogs? We might, especially if all other factors under consideration were equal, but the Clean Label Project doesn’t give us this information “straight.”  Without reporting the actual numbers – the parts per million or parts per billion that the CLP says its has measured – we have no context. How does the level of acrylamide in a particular dog food compare to foods we eat? If there is “only” as much acrylamide in our dog’s food as there is in our breakfast cereal, maybe we shouldn’t be all that concerned about it.

The same goes for antibiotics and pesticides and even the heavy metals. Without reporting the actual test results, and offering some specific values of other products we are familiar with, we don’t know how concerned to be.

But, in our opinion, the project really wanders off into the weeds when it attempts to factor “nutritional superiority” or “ingredient quality” into its ratings.

The most in-depth discussion that we could find of how Clean Label Project defines “ingredient quality” was a single paragraph on a page that discussed how it changed its ratings from its first report on dog food to its second. Here is the paragraph:

1. We’ve factored in ingredient quality. We know you want a quality, nutritious food for your dog or cat in addition to one that is low in industrial and environmental contaminants. While it’s always best to consult your veterinarian for your pet’s specific nutritional needs, we have created a system to help. Not all pet food ingredients are created equal – some products use preservatives, artificial colors or chemicals, while other products do not. Some products are dedicated to using quality meats, vegetables, and starches, while others use loopholes to include lower quality ingredients. Our ingredient quality system captures this, rewarding products for using a smaller number of quality, transparent ingredients rather than a large number of less regulated ingredients.

We could spend a day critiquing each sentence in that paragraph, but the most glaring problem – and the one that creates the most dissonance between the CLP’s stated goals and its actual ratings – is the last sentence.

This definition of the rating for ingredient quality is comparing a bag of organic apples to a truckload of Chinese oranges; there are simply too many factors, none of them defined, in the explanation of this segment of the rating criteria.

The idea that a smaller ingredients list is a good predictor of quality is alluring – but there are so many exceptions that the number of ingredients in a food cannot be used as a reliable criterion in and of itself.

Worse, there are no indications of what the Clean Label Project might call a “transparent” ingredient, or a “less regulated ingredient.” What are they talking about? Without definitions and examples, the consumer is left without any context here, and has to look at the food ratings themselves to get a general idea of what the CLP might be measuring with these definitions.

So, you go to the ratings page and start clicking on specific foods. The ratings start off with five-star products – the highest-rated foods. When you try to select “dry dog foods,” you see there are freeze-dried foods intermingled with kibble – these are incredibly different product categories, but, okay. You click on a specific food. You already know that it has been given a five-star rating, but you want some details.

Some Clean Label Project Inconsistencies:

Unequal Category Placements of Dog Food

While each of the foods have been given an overall rating of one to five stars, some of the foods have been rated in four categories (heavy metals, process contaminants, by-products contaminants, and nutritional superiority), some have been rated in three categories (product purity, product value, and product nutrition), and some were rated in only two categories (purity and value). Huh?

No Test Results to Base Ratings

There are no test results posted that could put these ratings into context. The only explanation of the star ratings is this: “The star rating you see tells you whether the product is average (3 stars), below average (1 star), or above average (5 stars) in terms of the overall purity (lower levels of contaminants).”

The List Isn’t Ordered

While the first page of the ratings starts out with five-star products, there is no indication of how else the products have been ordered. One might imagine that they are rated from the highest to the lowest star rating, but, in fact, as we were examining the list of dry dog foods, we found a product with a three-star rating on the first page of five-star foods, so there goes that theory. Again, without any values from test results being published, and given that they are not listed alphabetically by star rating, a viewer has no way to know why they are arranged in their order of appearance. 

Does the CLP Even Look at Dog Food Ingredients?!

But any credibility that the project might have had goes completely out the window when you discover that foods like Ol’ Roy have been rated with five stars overall. Let’s compare what’s in Ol’ Roy with a one-star food from the last page of the dry dog food ratings: Earthborn Holistic Ocean Fusion variety.

The Clean Label Project doesn’t list product ingredients, but in case you weren’t familiar with what the two above-named foods generally contain, here are their ingredients lists:

• Ol Roy Complete Nutrition Dry Dog Food Ingredients: Ground Yellow Corn, Meat And Bone Meal, Soybean Meal, Poultry By-Product Meal, Animal Fat (Preserved With BHA And Citric Acid), Corn Gluten Meal, Natural Flavor, Brewers Rice, Salt, Potassium Chloride, Color Added (Titanium Dioxide, Yellow #5, Yellow #6, Red #40, Blue #2), Choline Chloride, Zinc Sulfate, Vitamin E Supplement, Ferrous Sulfate, Zinc Oxide, Niacin, Copper Sulfate, Vitamin A Supplement, Biotin, Manganous Oxide, Calcium Pantothenate, Vitamin B12 Supplement, Thiamine Mononitrate, Pyridoxine Hydrochloride, Menadione Sodium Bisulfite Complex (Source of Vitamin K), Riboflavin Supplement, Sodium Selenite, Calcium Iodate, Folic Acid, Vitamin D3 Supplement, Cobalt Carbonate.
• Earthborn Holistic Ocean Fusion Ingredients: Whitefish Meal, Sweet Potatoes, Ground Barley, Rye Flour, Potatoes, Menhaden Fish Meal, Canola Oil (preserved with Mixed Tocopherols), Ground Flaxseed, Potassium Chloride, Choline Chloride, DL-Methionine, L-Lysine, Taurine, Glucosamine Hydrochloride, Chondroitin Sulfate, Beta-Carotene, L-Carnitine, Vitamin A Supplement, Vitamin D3 Supplement, Vitamin E Supplement, Zinc Sulfate, Ferrous Sulfate, Niacin, Folic Acid, Biotin, Manganese Sulfate, Copper Sulfate, Salt, Calcium Pantothenate, Thiamine Mononitrate, Pyridoxine Hydrochloride, Riboflavin Supplement, L-Ascorbyl-2- Polyphosphate (source of Vitamin C), Zinc Proteinate, Manganese Proteinate, Copper Proteinate, Calcium Iodate, Sodium Selenite, Cobalt Carbonate, Vitamin B12 Supplement, Yucca Schidigera Extract, Rosemary Extract.

That just doesn’t make any sense! In our view, rating a food that contains artificial colors and preservatives, a cheap plant protein at the top of the ingredients list, and unnamed animal species sources of protein and fat, over a food with natural preservatives and a named animal protein source at the top of the ingredients list, is not doing dogs or consumers any favors. If they stuck to reporting the contaminant levels – not rating the products, but listing the results – we could get a little more excited. But these ratings just won’t help consumers identify good, healthy foods.

If the overall mission of this project really is “health and transparency in consumer product labeling,” more transparency is needed.

The Clean Label Project for dog food is NOT Whole Dog Journal-Approved.