Last week the STAC Benchmark Council unveiled the results of its latest STAC-A3 benchmark test, consisting of Dell R-Series servers powered by Intel Xeon processors, Dell Networking switches, Cloudera Enterprise Hadoop software, and implementation services from Dell and Cloudera. The report is available here.
The STAC Backtesting Special Interest Group (SIG) is comprised of end-user organizations and vendors focused on the engineering challenges associated with the backtesting of trading strategies, using historical data to forecast the profitability of new trading algorithms. This is crucial to the success of financial firms today, as any glitch or error can cost billions. The requirements for STAC-A3 was spurred by the need of several financial firms in order to expedite software and hardware comparisons.
The Hadoop Streaming implementation tested in this project was authored by Think Big Analytics and determined Hadoop Streaming can scale with respect to both simulations and symbols per simulation. Think Big also wrote the reference implementations for STAC-A3. This is a good gauge to sizing up whether your workload can scale to a larger dataset, or requires faster response time and if additional capacity is needed. This is key to determining the capabilities and performance of a financial firm’s copious amounts of data and trade volume, so that IT can spend its time in creating formulas for success, rather than worrying about overloaded infrastructure. For more detail regarding this backtesting, please see a recent article I wrote for CIO.
A Spark-based implementation of STAC-A3 is also under development by Intel, with the report expected soon.