Financial services
Case Study:
Citi builds Microsoft-based compliance system
30 April 2008
Citi’s Lava Technology team needed to create a compliance solution for the US Securities and Exchange Commission’s Regulation NMS. After consideration, the team selected the Microsoft Application Platform to meet the challenge.
When the Citi Lava Technology team started to consider a new compliance solution, it thought it would need a Cray supercomputer or other specialised solution, because of the terabytes of data needed to recreate market conditions at any point in time.
After doing competitive testing with other solutions, the company, which provides a suite of products to support real-time feeds across multiple equity markets, determined that the Microsoft Application Platform, including Microsoft SQL Server 2005, provided the best performance and lowest total cost of ownership. The solution supports peak loads that exceed 200,000 messages per second.
Citi’s Suite of Lava products, has earned a reputation for market-leading performance and innovation in one of the most demanding computational environments of the world — providing real-time feeds across multiple stock markets (also known as equity pools) to help traders achieve the best execution prices for their customers.
The suite of Lava solutions provide real-time aggregated market data across multiple liquidity pools including the New York Stock Exchange (NYSE), NYSE Arca, NASDAQ, and other major US exchanges.
The Lava ColorBook solution integrates the feeds from multiple liquidity pools for display on a single-screen montage to give clients an integrated direct market access (DMA) solution. Prior to the Lava DMA montage solution, traders accessing multiple markets would have separate, non-integrated systems sitting on their desk. The Lava ColorMaker solution builds market-making capabilities on top of Lava ColorBook by integrating price discovery, direct access, and quote management into a single application.
Several years ago, sell-side firms would have separate trading workstations for each stock exchange they worked with. Integration consisted of traders whirling around on their swivel chairs to enter their orders on separate dedicated terminals.
The Lava DMA solution was built using the Microsoft Application Platform, including Microsoft SQL Server 2005 database software. A pioneer in creating DMA solutions, the Lava technology team prides itself in having been the first to provide access to all the US market centres.
When the Lava development team created its first solutions its technical architects had a wealth of experience building UNIX based solutions for the financial services industry. Mindful of expense, they determined that redundant PCs could provide the same or better reliability as other platforms at a fraction of the cost. The decision was made to build Lava solutions using the Microsoft Application Platform.
Recently they again considered a range of technology choices as the team prepared to create a new solution to help it comply with regulations from the US Securities and Exchange Commission (SEC). Regulation NMS (Reg NMS), a set of rules designed to modernise and strengthen the national market system for equity securities, was published in the Federal Register in June 2005, with a final phase-in required by October 2007.
At the heart of Reg NMS is the Order Protection Rule (Rule 611), which requires trading firms to execute a trade against the best bid or offer in the national market system. Reg NMS also requires that trading firms be able to demonstrate to investors that the best price was given.
Reg NMS compliance requires capture and storage of vast data sets. This is because if an investor comes back to an equity trader 20 minutes — or even several days or weeks — after a trade is executed and says, “I don’t think you gave me the best available price,” the trader must essentially be able to recreate pricing for that given stock across all equity pools at the exact time the trade was executed.
“I thought we would have to deploy a Cray supercomputer, mainframe, or something similar to create our compliance solution because it requires being able to traverse more than 2 billion records to recreate the montage for any point in time,” says Jeff Hays, vice president for product management at Citi’s Lava Technology operation. “I didn’t think this could be done with SQL Server.”
Hays wasn’t alone. “To be quite honest, when this project started we really didn't even consider Microsoft because providing Reg NMS compliance is so demanding,” says Rocco Mangino, Senior Vice President Product Development, Lava Technology, Citi. “We were looking at specialised solutions that had been created specifically for tick storage and retrieval, but they were very expensive, and would have required our obtaining new skill sets to deploy and manage.”
Among those who believed the Microsoft Application Platform could support Reg NMS compliance was Ann Neidenbach, Managing Director, Citi, who had spent 20 years at NASDAQ before joining Citi’s Lava Technology group.
“The requirements were extraordinary — processing what has now grown to more than 200,000 updates per second across more than 100 million rows while supporting query responses of less than two seconds — that the development team simply didn’t think SQL Server could do it,” says Neidenbach. “But I’d seen how SQL Server had worked for NASDAQ and believed that it could.” A comparison of possible solutions was evaluated to determine which technology platform best met the requirements for creating a Reg NMS solution that could keep up with peak demands that exceed 200,000 messages per second, and that could scale into the future to keep up with the rapid growth in equities trading.
Citi’s Lava Technology group developed its Reg NMS solution, called the Tick Store, using SQL Server 2005 Enterprise Edition (64-bit) running on the Windows Server 2003 Enterprise Edition for 64-bit Systems operating system. The Tick Store solution includes some 70 terabytes of data stored in zipped flat files. Each file holds one day’s data from one trading system. The solution grows by about 30 gigabytes (GB) per day, adding a terabyte of historical data every 33 days or so. The Tick Store is hosted on an HP ProLiant DL585 server computer with eight 64-bit AMD Opteron processors and 32 GB of RAM. “We have all the ticks we’ve received since November 2001, as we lost data in 9/11,” says Mangino.
When fielding a Reg NMS query about pricing, the relevant data is imported from the flat files into the SQL Server relational environment of the Tick Store’s database. The Tick Store includes a key that associates the order routing with the market data.
The solution, which supports more than 16,000 symbols, or equities, was created using Microsoft Visual Studio 2005 development system, with C++ used for generating the most performance-critical code. The Tick Store was created within three months, by a team of three contributing time equating to about two full-time equivalents.
Building the Tick Store using the Microsoft Application Platform has enhanced performance so well that the Lava technology team values it as a competitive advantage. With Tick Store support for pumping historical data, as needed, into the Tick Database relational environment, compliance query response times have gone from hours to less than a second. The solution also provides the agility to meet internal and customer needs, scalability to match trading growth, and the lowest total cost of ownership of any solution it looked at.
Neidenbach notes that the high performance SQL Server provides is especially important because of the burstiness of market activity. “A year ago we were seeing peaks of about 60,000 messages per second,” says Neidenbach. “Now we are seeing peaks of 200,000 messages per second, and we expect that to go even higher because of the growth in algorithmic trading. Supporting heavy loads like this requires a sophisticated and fast database, because even when a trading algorithm behaves erratically and generates what appears to be questionable market data, every tick must be captured.”
“Until recently we were seeing 800 million updates in a day,” says Hays. “We’re now sometimes seeing nearly two billion messages a day. We’re gaining a competitive advantage because we have the technological fondation that gives us the performance to keep up with the growing loads.”
Prior to deployment of its Tick Store, Reg NMS queries could take four or more hours to complete, as historic data was assembled into an order book to recreate the montage for the order being asked about. “By pumping the archive data into SQL Server we’ve reduced montage creation time for historic data from four or more hours to less than a second,” says Mangino. The company reports processing speeds of 1.6 billion rows per second during montage creation.
Citi values the agility that enables it to easily change its Lava Tick Store solution to meet internal and customer needs. Updating the application is easy for the company because its internal developers created the solution themselves using the Microsoft Application Platform.
“Other products we looked at would have required us to go through a product change request process, in which we were asking a third party to change their product to better meet our needs,” says Neidenbach. “We don’t have time for that. We need the ability to fine-tune our own solution whenever we like, and that is what we are able to do with SQL Server 2005.”
The team knows that the need for change isn’t just theoretical. “The industry changes constantly,” says Neidenbach. “We get notices every day of changes in the data we’re receiving. They may be adding new fields, or new order types, and we need to be able to modify and accommodate those changes as they happen. Because this is our own product we can make those changes as they are needed.”
The agility of SQL Server and the rest of the Microsoft Application Platform makes it easy for the Lava Technology team to use the information it handles to gain new value, including using SQL Server 2005 Analysis Services to do analytics.
“If we were nailed down to a fixed third-party product we would be limited to the existing functionality,” says Hays. “Instead we are in a position where we can do whatever we like. For example, although we haven’t yet made use of Analysis Services, we could use it to open up new possibilities in what we do with our vast amount of data.”
Customers are intrigued by running analytics against terabytes of historical data to see what value might emerge. Hays notes, “Our customers are very interested in what they might learn from this data about how their algorithmic trading programs are working or anything else that yields insight into the market.”
“Without getting into names specifically, one of the reasons I kept pushing for SQL Server is the fact that so many of the alternatives had high licensing costs as well as difficult to use proprietary interfaces and cryptic codes that would have required our training or hiring application specialists,” says Neidenbach. “Some solutions were so specialised they would have been more like integrating an outsourced project, and we didn’t want to lose our ability to do our own fine-tuning.”
Mangino estimates that the cost of using an alternative solution would have been higher by approximately US$1 million on the initial purchase cost and higher by an additional cost of several hundred thousand dollars a year for ongoing licensing and administrative services.