We get email all of the time from readers asking why we don’t test this or that. There’s two simple reasons.
First, we do the news and we don’t have enough bandwidth to do proper test & evaluations which leads us to the second point. We don’t have the access to the right facilities and equipment to conduct proper tests.
We see folks all over the internet throwing products on their charcoal grills saying they are conducting burn tests or taking their latest gucciflauge out in the back yard to snap a couple of photos and saying they are testing camouflage or worse yet, shooting a ballistic material at a random range with some random gun. In all cases, they are wrong. All they are doing is making noise and, filling their reader’s heads full of nonsense that in some cases is libelous and in others downright dangerous.
Our editor actually conducted test & evaluation for DoD and later, after he retired from the military for commercial companies. Consequently, he knows how it is supposed to be done and, that is why you won’t see us doing it. If we aren’t going to do it right, we aren’t going to do it at all.
So, next time you see an internet “test” conducted by amateurs, ask them what qualifies them to conduct that test. Ask to see their test plan and their data collection schema. Ask them how many times they conducted the test and how they ensured that the tests were conducted properly and consistently. We could go on and on but quite frankly, question one will stump them.
Not only are there folks posting this nonsense on the internet but they are taking money from unsuspecting companies. Before you pay someone to conduct RDT&E for your company ask them what qualifies them to do this. There are tons of former end-users out there but DoD’s T&E community isn’t very big. Verify credentials.
May I suggest that if science and mathematics weren’t consistently involved, it wasn’t RDT&E.
Interestingly enough, we submitted two of our shelters to Natick under the RFP for the USMC 3-Season Sleep System.
Prior to the submission we had lab tested our product at the Cold Lab, Department of Kinesiology, Simon Fraser University and had also had them field tested by members of the Canadian Armed Forces in Canada, the US, and in Afghanistan.
We learned later from Natick that our product could not be considered because of the Buy American Clause. We were also told that our product had been tested under DoD methods and that there was no difference in results from our biomechanical shelter and all others submitted.
We asked if they had also “live” tested the product. They had not. We replied that “if they put a live body inside, they’d witness a demonstrable difference in performance. They refused. So we asked that the two units be returned.
The larger of the two shelters came back, but the smaller, higher performing unit did not. The first time we asked where it was, they said the Army Rep had it. The second time; the Army Special Forces Rep had it. The third time; they had no idea.
To date we still don’t know where it is. And we still have yet to see the results from the testing even though said results are to be provided.
So, when you see people doing their own testing and making their own claims, realize that there may just be a very good reason for it other than wishful thinking and wishful claims.
Sincerely,
Stephen Jacura
PAC RES GRU – Pacific Research Group Inc.
My site is only about reviews and I explain what it is I do with the items reviewed. I explain each of my writers backgrounds. My background consists of the army and engineering. Lab tests for things like flammability are important but things like usefulness on the ground aren’t things that can really be tested in a lab environment. I refuse to take money for advertising as it would be a possible avenue of bias.
Now I will say things like drop tests, freezer tests etc are not all that useful unless conducted in a controlled environment. I remember watching a reviewer toss sand bags off a truck onto hydration carriers. While this is interesting to see what hydro carrier is nearly bomb proof it doesn’t display a realistic test. If a soldier is blown off his feet and lands on a hydro carrier sure. However I have fallen on my back on ice with a hydro carrier several times and never had an issue. People need to take into consideration the way forces act on objects.
Buyer should always be weary when reading reviews in magazines and websites. Think of the buyers test criteria and motives. There are a lot of reviews out there that exist only as fluff peices to please manufacturers. Some manufacturers don’t take criticism well and some thrive on it as a way to improve.
I am an avid reader of Soldier Systems and find the articles tremendously helpful and informative. I take to heart especially the mention about ballistic testing of armor. Too many times, you witness on YouTube people demonstrating the ballistic capabilities of body armor by doing a “back yard” testing. For instance, I have seen people put together videos of SAPI plate testing by placing the plate into the ground and shooting it with various types of weapons and ammunition, or they put the plate into a carrier on place it onto a dummy.
Well, body armor is just that, it is designed to resist or stop the performation of certain munitions it is rated for. According to the NIJ 0101.06 Ballistic Performance of Body Armor Standard, there are 67 pages of standards and performance requirements that must be met. For example, prior to shooting the armor plate or panel, it must be conditioned for 5 days. This involves placing the armor into a chamber and subjecting it to various temperatures and moisture introduction.
Then there are very specific requirements when it comes time to shooting the armor panel or plate. This involves placing the armor onto a clay backing; Roma Plastalina#1 clay that has been calibrated and a drop test performed.
Depending on the type of armor to be tested, six (6) shots are typical of the test. Each shot must be no more that two (2) inches spaced apart from the center striking point of each previous shot, and two of the shots must be within two (2) inches from the edge of the armor panel or plate.
Moreover, the first two shots cannot have a back face signature; wich is the depression into the calibrated clay of more than 42mm when testing hard armor plates.
The ability to show viewers that the armor panel or plate does not exceed the back face signature requirement is almost all the time missing from these vedeo demonstrations. That is by far, the most important aspect of proving the armor panel or plate is worth your consideration. (For more information or to validate this information, you can download the NIJ 0101.06 standards from http://www.justnet.org.)
So Soldier Systems is absolutely correct in pointing out that issue. In order to perform the proper testing, it does require that the armor be tested at an approved laboratory and not at some uncontrolled range with who knows what type of ammo is being used.
Thank you Soldier Systems for this great article!
S.S. ….AMEN!
Guys, don’t get it twisted. This isn’t about reviews. Those are basically subjective and opinion based. They are a great way for consumers to gauge whether or not they’ll like a product or if it’s suited to their desired use.
We’re talking objective testing here. The problem is, on the Internet, everyone’s an expert. The best way to tell that someone doesn’t know what they are talking about is when they use the word “test.”
Good on ya! Appreciate the straight answers and the no nonsense approach.
Fair enough I will give you that. Most people don’t have the expertise or equipment to test protective equipment. A test doesn’t consist of one item being run through the procedures it requires many. You need to collect data on failure rates and find a meaningful way to store and decipher that data. It’s a fairly lengthy process requiring a lot of money and resources to create experiments with a repeatable test procedure that provides consistent results. Minimum experiment sizes are often twenty items. I don’t know about you but I am not buying twenty ceramic plates to test.
Consider just the simple test for accuracy of a weapon, as conducted by the Army to accept it or the ammunition. If one in a hundred civilians has actual prior service experience, I’d say of the shooting community, one in a hundred might also know it’s a TEN shot group. Not three, or five. Ten.
Test criteria would also include what distance, specify the target, how the weapon is supported – or not, what ammunition is used, and the environment. Iron sights or optic, and which. How many shots to sight in, how much time is allowed, how is the test repeated, how are the results scored and quantified.
That last one really screws with people – the group isn’t measured the way most think.
If shooters can’t even conform to milspec standards testing accuracy, what makes them any more qualified to test other gear? Just being inside a tent and thinking it does a nice job is anecdotal. Wearing a gen1 Goretex might be nice in a driving rain, but it doesn’t insulate well, or hold up in rough use over 8-10 years. Naturally hydrophobic materials are proving superior – by test.
Entirely the reason Natick does it’s empirical process – numbers don’t lie, just salesmen.
@MarkM
“Just being inside a tent and thinking it does a nice job is anecdotal.”
How about being inside a single person shelter with an Arktis rain shield beneath it, and a good old wool blanket wrapped around you.
The overnight temp averages -14C.
You’re warm, the interior of the shelter is warm, and dry.
The gear you brought in with you is dry, the wet that dripped off the gear during the night egressed out through the floor without affecting you or the blanket.
Oh, and you were able to sleep for 8 hours?
Anecdotal my butt.
You won’t do a review? Then what the hell is this?: http://youtu.be/npAmJmGlIC4
My husband works very hard protecting this country and we learned nothing about that jacket from that video. Was the jacket just tested at the local duck pond or in rain or when ‘low crawling’ (sorry I don’t really know the terminology)…
There are a lot of other sites out there for reviews and whether they have a piece of paper from the DoD saying they are qualified to test gear or not, they put together much better reviews than what we’ve seen. We just want real world tests, not high and mighty flip flopping.
Come on guys, we know you can do better. Don’t disappoint.
Feisty aren’t we?
First off, thanks for your husband’s service but I’m not sure what that has to do with the article in question or your comment. I can tell by your tone and comments that the entire point of this article was completely lost on you. You use the terms “review” and “test” interchangeably in your rant, and that is point of this article. A test is a scientific examination of an item or phenomenon. A review is a more subjective look. It gives the author’s opinion, perhaps based on experience, and perhaps not. The Anorak article was an introduction to the piece of clothing. It was by no means a test. Nowhere, does it say that it was tested or used in any conditions. The weather wasn’t where it would had to have been to give the Anorak a full wringing out so we concentrated on features.
There are very, very few actual reviews on SSD. We do the news. Reviews, if done properly, take several days (tests even longer). We publish multiple articles per day. You do the math. Many other websites use reviews to get free stuff from companies and they read our news to figure out what products to go ask for. The Combat Anorak was purchased by us as soon as they were available. I was able to take a look at one during a meeting with FirstSpear over the summer and knew right then that it was a product to use. I was happy to spend my own money on it. It is a great item.
As for the Anorak video, I am sorry that you were not able to learn anything about the garment from it. I’m sorry you didn’t learn the purpose of the Combat Anorak. I’m sorry you didn’t learn about the fit. I’m sorry you didn’t learn that it has both a front and rear opening or how to access your gear by simply opening the flap. I’m sorry that you didn’t learn that the hood is designed to fit over a helmet. And finally, I’m sorry that you didn’t learn that the Anorak fits into its own stuff sack.
But wait, the video covered all of those issues. Unfortunately, you didn’t get anything from the article or video but lots of other folks did.
I am sorry for being not being a mindless drone. Thank you for belittling me as a reader.
Wait, I mean former reader.
Something tells me by the tenor of your original post that you weren’t ever really a reader.
https://soldiersystems.net/2011/11/15/comments/
Great article. Military Moron and Milspec Monkey come to mind. Civillians who are geardo’s who try to evaluate kit.
They don’t come to my mind. Both are very up front about who and what they are.
With all the qualifications, credentials, expertise and resources – or so one would think – the US Army came up with UCP (Universal Camouflage Pattern, “ACU”). UCP is so badly flawed as “Universal” that backyard testing by amateurs is pretty sufficient to show this.
Visual camouflage is (mainly) about fooling visual perception. Testing it isn’t exactly rocket science – any grunt has what it takes to test camo in an anecdotal fashion. Systematic testing isn’t that much more difficult, it just requires more resources and – as any testing – an understanding of experiment design and basic statistics in order to get meaningful results.
Natick’s universal camo testing (2002-2004) was an amazingly modest study judging by the presentation http://www.dtic.mil/ndia/2004issc/wednesday/dugas.ppt .
Also it’s worth noting that Multicam finished 3rd out of 4 camo patterns in this Natick study. It’s mind boggling, that they came up with UCP (which, afaik, wasn’t tested against any other pattern), and then (partially) replaced it with Multicam. Surely they could do better, and it appears that that’s exactly what is being currently done.