<pre><pre>Meet the American espionage system of the future - it's Sentient

During the final session of the Colorado Springs Space Symposium 2019, attendees staggered into a gigantic ballroom to listen to an Air Force official and a National Geospatial Intelligence Agency (NGA) official, as the panel title said, “Enterprise Disruption. & # 39; The presentation remained as vague as the title until a direct question from the audience seemed to wind the panel members.


How good, the person wondered, had the algorithms from the military and intelligence communities received to interpret data and take action based on that analysis? They pointed out that the commercial satellite industry has software that counts containers on cargo ships and cars in parking lots shortly after their photos are taken in space. "When will the Ministry of Defense have a real-time, automated, global battle order?" They asked.

"That's a great question," said Chirag Parikh, director of the NGA & # 39; s Office of Sciences and Methodologies. "And there are many really good classified answers."

He paused and sat down in his chair. "What is the next question?" He asked smiling. But he kept talking and described how "geospatial intelligence" no longer just means photos of satellites. It means everything with a time stamp and a location stamp and the attempt to integrate all those different data.

Parikh then answered this question: when would that translate into an almost immediate understanding and strategy development?

"If not now," he said, "very soon."

Parkih has not mentioned specific programs & # 39; s that can help make this kind of autonomous, real-time interpretation possible. But an initiative called Sentient has relevant possibilities. A product of the National Reconnaissance Office (NRO), Sentient is (or at least wants to be) one all-eating analysis tool, able to devour all kinds of data, understand the past and the present, look ahead to the future and point satellites to what they think will be the most suitable interesting parts of that future. Ideally, this makes things downstream easier for human analysts at other organizations, such as the NGA, with which the satellite-oriented NRO partners work.


Until now, Sentient has been treated as a government secret, except for vague allusions in some speeches and presentations. But recently released documents – much formerly secret or top secret – reveal new details about the goals, progress and reach of the program.

Investigations regarding Sentient have been ongoing since at least October 2010, when the agency published this A request for Sentient Enterprise white papers. A presentation says the program reached its first R&D milestone in 2013, but details about what that milestone actually was are retained. (Deputy Director of NRO & # 39; s Public Office Karen Furgerson declined to comment on this timing in an email to The edge.) A 2016 House Armed Services Committee hearing The National Security Area included a quick summary of this data-driven brain, but public meetings have not mentioned it since. A presentation in 2018 posted online claimed that Sentient would go live that year, although Furgerson said The edge it was currently under development.

"The NRO didn't say much about Sentient in public because it's a secret program," Furgerson says in an email, "and NRO rarely appears before Congress in open hearings."

The agency has been developing this artificial brain for years, but details that are available to the public remain scarce. "It takes large amounts of data and processes it," says Furgerson. "Sentient catalogs normal patterns, detects anomalies and helps to predict and model the potential actions of opponents." The NRO did not give any examples of patterns or anomalies, but it could be imagined that things like "not moving a rocket" versus "moving a rocket may be on the list. With those predictions in hand, Sentient can control the sensors put satellites in the right place at the right time to capture evil will (or whatever it wants to see) in action. "Sentient is a thinking system," says Furgerson.

It is not all dystopian: the documents issued by the NRO also imply that Sentient can make satellites more efficient and productive. That could also be free up people to focus on in-depth analysis rather than difficult needle search. But it can also contain undisputed prejudices, come to dubious conclusions and raise concerns about civil liberties. Due to the secret nature we do not know much about those potential problems.

"The standard practice of the NRO & # 39; s and the Intelligence Community is NOT to disclose sensitive sources and methods, because such disclosure involves a high risk that opponents will counteract it," Furgerson says. “Such a loss hurts our nation and its allies; it reduces the American information advantage and national security. For these reasons, details about Sentient remain secret and what we can say about it is limited. "

Satellite programs & # 39; s were generally one of the most silent intelligence initiatives. The first program to take photos from space, Corona, began in 1958 and the satellite successfully shot its first bucket of film back into the atmosphere in August 1960. A few days later, Edwin Land, CEO of Polaroid, flushed the film over the floor of the Oval Office while Dwight Eisenhower watched. Before him lay similarities between airports and military installations in the Soviet Union.


"Here are your photos, Mr President," said Land.

That great unveiling according to one official history of the NRO, directly led to the creation of a new agency responsible for "design, acquisition and operation of reconnaissance satellites".

The NRO was formally established the following year.

In the 70s the office started with the launch of & # 39;Keyhole"-Satellites with specifications similar to those of the Hubble Space Telescope, but they were aimed at Earth instead of other galaxies. We also know that The orbital collection of NRO includes data collected during the test or operation of aircraft, missiles or other systems; intercepted speech, text or image communication; and radar, among other sources. Of the more than 150 (known) American military satellites, the NRO works around 50.

Satellite image of the USSR taken by CORONA.
Image: NRO

Now, almost six decades after the founding of the NRO, the sky is full of other downward-looking satellites, some in the hands of private intelligence companies. One of these, BlackSky, uses those satellites to feed a system that is essentially Sentient's unclassified Doppelgänger.


When two oil tankers were attacked in the Strait of Hormuz on June 13, the BlackSky program sent its satellites into action and took photos of the incident while smoke from the explosions still moved to the sky. The floating beacons of the ships and local news reports indicated that something was going on, causing BlackSky analysts to focus their attention on the busy shipping route near Iran.

Insiders call that process & # 39; tipping and cueing & # 39 ;: using a data point from a source to cueen a satellite to look at a particular place, or using information from a satellite to view the collection of another instrument stimulate. In the ideal version of that process, an automated system extracts all types of data, synthesizes it into something sensible, displays satellite symphony, incorporates satellite data into the analysis cycle, comes to a smarter conclusion, points the satellites or other sensors again and repeat the entire process. Do it well enough, and a company (or intelligence agency) could build a tower of knowledge about the past, understand current events faster than their competitors, and – perhaps someday – predict the future.

Satellite image of oil tanker struck in the Gulf of Oman, June 2019.
Image: BlackSky

While the commercial industry has built up its network of orbit observers for the earth, the intelligence community has taken note of this. In 2016, the NGA, which analyzes data collected by the NRO, and the NRO jointly announced the commercial GEOINT program to better buy this data. In 2017, the NRO taken over purchasing responsibility, and has since signed at least three new contracts. One was with a company called Maxar, which owns some of the most powerful satellites with a high resolution in the private sector and for a long time virtually the only company sale of satellite images to the NRO. This time, however, the agency also signed another deal with Planet, which operates a constellation of small satellites that map the entire country of the Earth every day. The third contractor is BlackSky.

Here Sentient comes into the picture again: all images of the NRO, the army and these commercial satellite companies, combined with other geospatial intelligence – all with a time label and a location tag – create a huge amount of information that much more then a literal army of people could comb through. To keep up with the fire hose of information, the NRO partially focuses on AI. "Sentient wants to help analysts" connect the dots "in a large amount of data," says Furgerson.

How can Sentient connect the dots? We don't know exactly. Documents issued do not explicitly indicate the types of data that Sentient can transfer, but it is clear that the program is interested in all kinds of information. "It could include electronic interception of international communication; it may contain earlier images; it could contain human resources, ”says Steven Aftergood, a researcher at the Federation of American Scientists and director of the Project on Government Secrecy. "People say," Hey, there's something wrong with that hill. "


Retired CIA analyst Allen Thomson goes further. "As I understand it, the intended – and ambitious – answer is" everything, "he says. In addition to images, including financial data, weather information, shipping statistics, information from Google searches, records of pharmaceutical purchases and more, he says.

Consider what is happening in the private sector: BlackSky takes data from 25 satellites, more than 40,000 news sources, 100 million mobile devices, 70,000 ships and planes, eight social networks, 5,000 environmental sensors and thousands of Internet-of-Things devices. In the future, it plans to have up to 60 of its own Earth observation satellites. All that information goes to different processing pipelines based on the type. BlackSky can extract people, places, organizations and keywords from a news story. From an image it can map which buildings appear damaged after an earthquake. All that processed, but still disparate data goes to what makes BlackSky CTO Scott Herman a & # 39; giant analytical fusion engine & # 39; mentions, who tries to make more of it than the sum of its parts, tells satellites what to do about it, and alerts human analysts when events meet certain predetermined criteria.

In the real world, BlackSky can use that to track the positions of Russian jets. The company has images of places where the Russian army is parking its planes, and it has the rough shape of different types of flyers.

The company also has shape recognition algorithms, which can select pixels that together map a certain pattern. It can tune that algorithm to choose the contours of Russian jets, such as the MiG Fulcrum and Foxhound planes. After you have placed the satellite photos in that algorithm, you can learn how many of those planes are on runways. Understanding the meaning of that count – what "45 Fulcrums with Aleysk but none with Krymsk" resources – takes even more data. The system should know the history of jet demography, which it could have established from previous observations. It might collect data about how often they fly and where, or even watch news to find out if there is agitation or action around Aleysk: now the system knows exactly where to aim their real-time satellites to collect the information that their customer needs.

BlackSky has only just begun and has recently launched its own satellites. The ultimate success and usefulness of its system has yet to be proven. And based on the available information, it is unclear how far the comparable system of Sentient is. Thomson suspects that his more grandiose goals are still simple: goals. "How far Sentient is or will come is not clear," he says.


"If it were successfully implemented on a large scale, it would certainly be an important step forward," says Thomson, "but I saw no indication that it had actually happened."

Even if Sentient is still stuck at an early stage, its existence sends many red flags for privacy experts. Do the algorithms really work? In what ways are they biased? How many false positives do they generate? "How often does Sentient go hunting our billions of satellites for a wild goose?", Aftergood asks. "We must take into account the impact of Sentient that leads us astray."

The NRO notes that Sentient does not completely keep people out of the process, and offers some sort of control over its state. "Keeping people informed of intelligence data and information is an important way to monitor the performance of the algorithm," says Furgerson. "Sentient is machine-to-machine learning supported by people."

From the work of his company, Herman notes that when you ask silicon to make sense, you have to train it on the right kind of data. If you want algorithms to learn what radio towers look like, but you only show them towers in full sunlight, they will think that "black shadow next to tower" is actually an integral part of the tower's tower. If they then view a snapshot of such a structure on a cloudy day, they will not recognize it at all.

That is a banal example, but you can imagine more sinister, such as software that has learned that the word & # 39; bomb & # 39; is associated with terrorist plans, but never learns that & # 39; that's the bomb & # 39; a cool sentence. In other areas of AI, it has actually not been omniscient or objective, usually in ways that hurt non-white men. For example, at Amazon, face recognition software has consistently failed to identify the gender of women and people with dark skin. Analytics company Palantir created a predictive law enforcement program for the New Orleans police, but entered data that according to some reports targeted minorities unfair.

What prejudices can lurk in Sentient? What did the training data look like? Who validates his conclusions and how? At the moment, the answers are opaque, but secrecy watchdogs like Aftergood ask for. "Those are the questions you don't want the sponsors or financiers of Sentient to ask themselves and answer," says Aftergood.

Questions about who and what Sentient monitors are equally persistent and almost impossible to answer, but there are some indications as to where the program is peeping or not. Spy satellites, as used by the NRO, are primarily intended to focus on the world beyond the borders of the United States. And in contrast to his fellow intelligence services – including the NSA and C.I.A – the NRO has not really got caught up in major domestic espionage scandals. The biggest recent upset was probably about the mission patch for the launch of its NROL-39 satellite: it depicts a giant yellow octopus being sucked to earth – actually to North America. Tentacles surround the planet. The words "NOTHING IS OUT OF OUR REACH" smile in an arch among the cephalopods.

Despite the sentiment of the patch, there are some places where the NRO and Sentient are generally not expected to reach. In heaven and on earth, laws protect American citizens against unreasonable search and seizure by their government. "Under the existing legal regime, Sentient-driven exploration should not take place in the US," says Aftergood. “If that were the case, it would of course immediately raise concerns about privacy and civil liberties, and a whole series of related questions about how that collected information was used and stored, and so on. But it should not be collected in the first place. "

In response to questions about possible domestic surveillance, the spokesperson noted that the NRO obeys the most important guideline of the intelligence community: Executive Order 12333, as well as other applicable laws. This specific order of execution outlines when "collecting, storing and distributing" information about people in the US is permitted – provided that the collectors follow the correct procedure. One of the exceptions to the policy on & # 39; leave the US alone & # 39; However, intelligence is & # 39; obtained through reconnaissance over the head and not aimed at specific American individuals & # 39 ;.

But Executive Order 12333 outlines principles for intelligence services and does not apply in the same way to private earth observation companies that have spread in recent years, including BlackSky. Companies can aim their telescopes almost everywhere they like. Although the government reserves the right to perform "shutter control", which prohibits photography of a certain area, it has never done so (sometimes the government buys exclusive access to an area, a practice known as "checkbook shutter control") "). There are limits to resolutions whereby private companies can sell images to the public and to other countries.


For the most part, companies like Maxar, Planet and BlackSky are taking photos that anyone with a thick enough checkbook can buy – including you, and including organizations such as the NRO. That raises some interesting legal questions that researchers like Aftergood are still trying to figure out: if NRO was interested in researching the US and could not intentionally use its satellites to focus on your home, it would just be photo & # 39 can buy from your house from a private company instead?

The NRO did not provide a specific answer to the role of commercial data in Sentient. But limitations, says Aftergood, must still exist for paid data. "What they do with it must be mission-oriented in a certain way," says Aftergood. "They can't sniff because of sniffing."

Sentient's snooping remains a secret for now. The thinking system only speaks to those with a security clearance, although, as Thomson says, it can listen to just about anything.