WASHINGTON — The Army Rapid Capabilities Office, or RCO, does things differently. It has to. It’s mandated in its charter and embedded in its culture.
So when it came time for the small acquisition shop to find a way to speed up signal detection, it knew it wouldn’t seek answers using traditional methods.
Instead, the RCO studied commercial models for getting answers quickly and created a “challenge” that gave industry, academia, scientists and other agencies the opportunity to go head-to-head in a competition, with prize money awarded to the top three performers.
The challenge focused on using artificial intelligence and machine learning to speed up the rate at which electronic warfare officers, or EWOs, could sift through the congestion and noise that comes with signal detection. With an ever-increasing number of signals flooding in from satellites, radars, phones and other devices, the signal detection process is no longer efficient in understanding the vast amount of data presented to EWOs on the battlefield.
Soldiers with the Combat Electronic Warfare Intelligence Platoon, Delta Company, 54th Brigade Engineer Battalion provide signal intelligence to help the 173rd Airborne Brigade during Saber Junction 18, held in September 2018 in Germany. As more and more signals are captured by satellites, radars and other devices, the signal detection process is no longer efficient in understanding the vast amount of data presented to EWOs on the battlefield. (Photo Credit: U.S. Army photo by Spc. Josselyn Fuentes)
Within four months of setting up the Army Signal Classification Challenge, the RCO knew mathematically who had the best-performing algorithm.
The challenge also had an unexpected result. By offering an unorthodox method for garnering participation in what would have been a traditional request for information, or RFI, the RCO challenge resulted in the top three prize winners spanning the unconventional by including a federally-funded research and development center, an independent group of Australian scientists and a team from a big business.
“By structuring this as a challenge instead of an RFI, we were able to model what industry does and create something much more hands-on,” said Rob Monto, director of the RCO’s Emerging Technologies Office. “We invited anyone with a possible capability to participate and posted it on Challenge.gov and FBO.gov. This is very similar to the commercial model of posting on Kaggle.com, where data sets are sent out to communities of data scientists who want to compete against one another to determine who has the best solution.”
The RCO’s online challenge offered synthetically generated data based on what could be seen in the electromagnetic spectrum, and challenged participants to prove they had the best artificial intelligence and machine learning algorithm for performing “blind” signal classification quickly and accurately. The challenge was strictly performance-based and open to anyone. Because it was all online and completed in four months, it came with very little cost or burden placed on those participating.
“The response was overwhelming,” Monto said. “We had more than 150 participants from across traditional and nontraditional industry partners, universities, labs and government. As an incentive, we offered $150,000 in prize money.”
Team Platypus from The Aerospace Corp. won first prize in the Army Signal Classification Challenge over the summer of 2018. The team includes (front row, from left) Eugene Grayver, Alexander Utter and Andres Vila; and (back row, from left) Donna Branchevsky, Esteban Valles, Darren Semmen, Sebastian Olsen and Kyle Logue. (Photo Credit: Photo courtesy Elisa Haber, The Aerospace Corp)
The RCO announced winners on Aug. 27, 2018. First place and $100,000 went to Team Platypus from The Aerospace Corp., a national nonprofit corporation that operates a federally-funded research and development center. Second place, with an award of $30,000, went to TeamAU, made up of a small team of independent Australian data scientists. And third place, with a prize of $20,000, went to THUNDERINGPANDA of Motorola Solutions.
“Having a specific problem that can be worked on by industry, academia and private citizens is a great way to establish and build a community of innovators for this technology area,” said Dr. Andres Vila, an engineering specialist at The Aerospace Corp. and a member of Team Platypus. “This challenge, which extended for approximately three months, was the right balance of having time to formulate a unique and robust solution but also not so long that the team lost urgency to find that award-winning approach.”
The challenge proved a better way to assess industry’s capabilities, instead of using a more traditional RFI and white paper approach, Vila said, calling it “spot on.”
“The challenge arrived at a great time as we were just kicking off this research and the Army had a well-formed problem set and, most importantly, data,” Vila said. “This competition gave us the chance to take our latest innovations and prototypes and apply them to this new customer-curated, hard problem. These types of customer-sponsored competitions provide very focused challenges that give us the confidence that we are using the best technology available to meet their mission needs.”
THE PROBLEM SET
The idea for the challenge stemmed from the RCO’s partnership with the Project Manager for Electronic Warfare and Cyber, within the Program Executive Office for Enterprise Information Systems, which recently delivered new electronic warfare prototype systems in response to an operational needs statement from U.S. Army Europe. Soldiers are using the equipment to implement electronic protection for their own formations, to detect and understand enemy activity in the electromagnetic spectrum and to disrupt adversaries through electronic attack effects.
However, in enhancing the signal footprint for EWOs, the prototype systems also brought more data to an already complex electromagnetic spectrum. Through the challenge, the RCO wanted to determine if artificial intelligence and machine learning, or AI/ML, could assist them in digesting that data and sorting through what is and isn’t important.
“We knew industry was already making leaps and bounds in applying AI/ML for image recognition and video recognition, but found that there was very little work being done in this specific area of signal detection,” Monto said. “What we discovered in a very short period of time is that AI/ML could in fact be applied to a data set that could translate to being integrated into an electronic warfare system on the battlefield.”
The idea is to create this application as a layering effect, where artificial intelligence and machine learning does one subset of signal classification for the EWOs, then layers other applications that are more encompassing onto that to give the EWOs a wider range of what they can identify, said Monto.
While the EWOs would remain as the lead for identifying signals of interest and analyzing their impact, the use of artificial intelligence and machine learning could help them quickly and accurately detect patterns, identify signals of significance, filter out unwanted signal noise and paint a picture of the electromagnetic spectrum.
THE CHALLENGE
The RCO’s Army Signal Classification Challenge began April 30 and closed Aug. 13. After opening registration online, competitors were given access to the training data set, consisting of over 4.3 million instances across 24 different modulations, which included a noise class. (The noise class represents “white” noise to replicate the real-life environment that signals would be detected in, rather than a pristine lab environment.) The effort sought solutions that could perform “blind” signal classification quickly and accurately. Blind signal classification requires little to no prior knowledge about the signal being detected in that specific instance. Instead, the solution would automatically classify the modulation, or change of a radio frequency waveform, as a first step toward signal classification.
The challenge gave participants 90 days to develop their models and to work with the training data sets. That was followed by two test data sets of varying complexity that were the basis for judging submissions. The first data set was released 67 days after the challenge launch, with a solution submission window of 15 days. A second, more complex test data set was released 84 days after the challenge launch, with a shorter submission window of only seven days.
Participants’ scores were based on a combined weighted score for both test data sets. Competitors could see how well they were performing against their peers through a participant leader board that showed scores in real time.
For first-place winners Team Platypus — which participated in the Defense Advanced Research Projects Agency’s Software Defined Radio Hackfest 2017 and whose name references platypuses’ ability to detect electrical fields with their bills — the challenge lined up perfectly with its core research in artificial intelligence and advanced signal processing.
“We really enjoyed the challenge process, which included the hard problem curation, providing training data and a specific scoring algorithm,” Vila said. “To do this with the highest level of confidence, we had to use a multipronged approach. We built statistics and metrics inspired by communication principles, and we also developed deep learning classifiers that work directly on the raw data. We ended up using several state-of-the-art AI techniques to achieve the winning submission.”
Their technology includes an algorithm trained to identify what kind of signal is present in the midst of a congested radio frequency environment, much like Soldiers would find in an urban core or battlefield where both friendly and enemy radio communications are being detected.
WHAT’S NEXT?
By structuring this effort as a challenge and not going through the traditional RFI process, the RCO proved it could take an industry model and move fast. For its efforts, it is substantially closer to identifying a potential solution that could be applied to battlefield electronic warfare capabilities in the very near future. It also showed the RCO could harness the promise of artificial intelligence and machine learning by applying it to a specific problem. The amount of interest and quality of performance, including from nontraditional organizations, was remarkable.
Now the RCO is quickly moving forward to the next step, with two possible options. First, the RCO could initiate a second, more intense challenge and open it up to only the top performers in the first challenge. Or, the RCO could begin to immediately move the algorithms into the hands of Soldiers through software enhancements to their existing electronic warfare equipment. This would enable the Soldiers to give immediate feedback and enable the Army to incrementally build capability.
Over the next several months, the RCO will begin to advance what was learned from the challenge, potentially prototyping the leading artificial intelligence and machine learning algorithms into Army electronic warfare systems.
For more information on the Army RCO, go to rapidcapabilitiesoffice.army.mil.
By Nancy Jones-Bonbrest, Army Rapid Capabilities
NANCY JONES-BONBREST is a public communications specialist for the Army RCO and has written extensively about Army modernization and acquisition for several years, including multiple training and testing events. She holds a B.S. in journalism from the University of Maryland, College Park.
This article will be published in the January – March 2019 issue of Army AL&T.