Conjunction Streaming Service (Demo)

CSS is a horizontally scalable streaming service that ingests position data and emits conjunction reports, within a configurable horizon, for objects on orbit or crossing orbits. Depending on resources, alerts are emitted throughout the horizon in seconds after position data arrives. The system is designed to scale to 1,000,000 tracked objects, and the system supports probability annotations and multiple data sources.

This demo uses TLEs for input, and the service propagates these TLEs using SGP4. In addition, the service is designed to support high resolution ephemerides and pluggable interpolators. Note that this service emits "reports" rather than "alerts" to suggest that the former be used in subsequent analysis to generate the latter when warranted.

The core of the service is a custom distributed spatial-temporal streaming index, which runs in main memory across multiple machines. For the current version, a large-scale deployment tracking 100,000 objects with a three-day horizon would use approximately 5TB of RAM.

// env.horizonReports is an object that represents the reports in the
// current view.  The object has report ids as keys and reports as
// values.  Let's iterate over the reports and compute the mean and
// min Dist.
var acc = 0, n = 0;
var min = null;
for (var id in env.horizonReports) {
    var r = env.horizonReports[id];
    acc += r.Dist,
    n++;
    min = (min === null || r.Dist < min) ? r.Dist : min;
}
var mean = acc/n;

// SQLite is available.  Let's have a table that contains recent
// reports.
env.db.run("CREATE TABLE IF NOT EXISTS reports (n INTEGER PRIMARY KEY  AUTOINCREMENT, r);");
env.db.run("CREATE INDEX IF NOT EXISTS reports_t ON reports (JSON_EXTRACT(r, '$.At'));");

// Delete old records (so we don't consume too much memory hopefully).
// We'll delete reports that are approximately an hour old.
var then = new Date();
then.setSeconds(then.getSeconds() + -60*60);
function pad10(n) {
    if (n < 10) {
        return '0' + n;
    }
    return '' + n;
};

var ts = '' + then.getUTCFullYear() + '-' +
    pad10(then.getUTCMonth() + 1) + '-' +
    pad10(then.getUTCDate()) + 'T' +
    pad10(then.getUTCHours()) + ':';

env.db.run("DELETE FROM reports WHERE JSON_EXTRACT(r, '$.At') < $ts", {'$ts':ts});

// env.reports contains new reports since this code was last executed.
// Add them to the table.
for (var i = 0; i < env.reports.length; i++) {
    var r = env.reports.shift();
    env.db.run("INSERT INTO reports (r) VALUES ($js)", { $js: JSON.stringify(r) });
}

// Query the mean Dist for the 10 most recent reports.
var query = [
    "SELECT AVG(dist) AS mean, COUNT(*) AS n ",
    " FROM (SELECT JSON_EXTRACT(r, '$.Dist') AS dist FROM reports ORDER BY n DESC LIMIT 10)"
    // ToDo: Don't consider canceled reports.
    ].join("\n");

var recs = env.db.exec(query);

// Display what we've learned.
var output = {
    dist: {
	win: {
	    min: min.toFixed(3),
	    mean: mean.toFixed(3)
	},
	recent: {
	    mean: recs[0].values[0][0].toFixed(3),
	    n: recs[0].values[0][1]
	}
    }
};

d3.select("#appOutput").html("<code>" + JSON.stringify(output) + "</code>");

In view:
TLEs:
Goroutines:
Heap:
Lag: