Microsoft open sources Trill, an internal tool for processing “a trillion events per day”

Update: We mentioned that Trill will be available in the coming months, but it appears that it is already available on GitHub and developers can grab it here. 

Another day, another open source project gets a blessing from Microsoft. Today, December 17, 2018, Microsoft is open sourcing an internal project it’s dubbed Trill, so that more developers have access to a new temporal query language as well as “a trillion events per day” worth of aggregated data.

For those unfamiliar with Trill, it started as a Microsoft Research project back in 2012 and has since been featured in published research papers. Moving from the research phase to actual implementation, Microsoft has used Trill internally for Azure Data products and other mission-critical streaming initiatives throughout the company including Bing Ads and Halo.

Microsoft details Trill as,

Trill was the first streaming engine to incorporate techniques and algorithms that process events in small batches of data based on the latency tolerated by the user. It was also the first engine to organize those batches in columnar format, enabling queries to execute much more efficiently than before. To users, working with Trill is the same as working with any .NET library, so there is no need to leave the .NET environment. Users can embed Trill within a variety of distributed processing infrastructures such as Orleans and a streaming version of Microsoft’s SCOPE data processing infrastructure.

As of today, open source developers get access to the following Trill enabled data and toolsets:

• As a single-node engine library, any .NET application, service, or platform can easily use Trill and start processing queries.
• A temporal query language allows users to express complex queries over real-time and/or offline data sets.
• Trill’s high performance across its intended usage scenarios means users get results with incredible speed and low latency. For example, filters operate at memory bandwidth speeds up to several billions of events per second, while grouped aggregates operate at 10 to 100 million events per second.

As far as the benefit to Microsoft from open sourcing Trill, the company seems to want Trill as the foundation layer for temporal processing for all businesses and developers. Microsoft also sees community involvement as the future of IStreamable abstraction for customers as well as IEnumerable and IObservable being made available to more people.