Integrated Data and Analytics Management

Xenomorph was one of the sponsors on the “Integrated Data Management” webcast last week, hosted by Inside Reference Data (audio recording available here). There were a number of interesting questions that arose from the Webinar.

One fundamental although somewhat academic question was "What is Integrated Data Management?". Certainly everyone seemed convinced that there would be less "Enterprise Data Management" (EDM) projects in future, given the expense, scope and scale of such projects. The concensus was that whilst the need for data management was better under stood across all financial institutions, data management projects would be bitten off in more manageable chunks by asset type, business function or division (so are silos back in fashion I ask myself?!). Coming back to the original question, I guess my slant on Integrated Data Management is that we are seeing more and more data management projects that have an integrated reference data and market data elements to them, primarily driven by the need to sort out data quality/completeness/depth for use within risk management (in light of the financial crisis).

Related to risk management, a topic I pushed was that given the origins of data management for STP/back office, and given the interest in low latency tick data management/analyis in the front office, there seems to be a market gap (particularly in the US?) on how to manage data such as IR/credit curves, volatility surfaces and other derived data sets. These data sets seem to fall into the gap between what is thought of as market data (primarily just prices) and what is reference data (IDs and terms & conditions). This is another area where a more integrated approach to data management would be beneficial, particularly in making all these datasets available for risk management.

Coming back to a "hobby-horse" of mine, then I also raised the issue that whilst it is fine to be doing great data management (high quality, complete datasets etc) what is the point if all of your data is ignored by the front office and Excel is used to download the data traders and risk managers need from Open Bloomberg. I think the management of unstructured data (spreadsheets, word docs etc) needs to be elevated as an issue since this (unfortunately?) is where most data resides currently, despite what we data management professionals like to think.

I also think that the principles of good data management (centralisation, quality and transparency) could apply to other things and not just raw "data", but what about centralised pricing and valuation, centralised curves and centralised scenarios for risk? Again what is the point of doing good data management if the ultimate "information" (e.g. a valuation) is done using poor quality data, with a complete lack of transparency over the data and model used.

A good question was asked about models, which was that given pricing models and their weaknesses have formed some part of the recent crisis, do we need more complex models. On having a few conversations about this and thought about it some more, then some would say it is complexity that got us into the crisis so this is the last thing we need. My view is that we do not necessarily need more complex pricing models and valuation techniques, but we certainly need more robust ones which does not necessarily imply more complexity. Coming back to a point raised by David Rowe previously, then I think all quants and risk managers should think about a "second means of valuation" for all the theoretical models they use, and that hedgeability (see recent post on pricing model validation) seems to be the common theme in producing more robust pricing models.

High Performance Spreadsheets

Another article about the operational risk generated by the usage of spreadsheets within the financial markets (see earlier posts), appeared in the April issue of Waters Magazine.
 
The articles highlights how spreadsheets are largely used within financial institutions and suggests that the current regulation requirements for more transparency and ad-hoc risk management might push the proliferation of spreadsheets even further. The articles also refers to the progress and improvements made by Microsoft in recent versions of Excel to increase the security of spreadsheets.
 
Xenomorph has worked closely with Microsoft on hosting its time series database within SQL Server 2008. The case study we have written together describes how SQL Server 2008 offers integration within Office Excel 2007 so that whilst the spreadsheet is still the end-user viewing tool, operational risk is reduced by engaging Excel 2007 as an analytics and reporting tool and not as a mean of storing data.
 
Our TimeScape solution offers more than 700 easy to use add-in functions to Office Excel 2007 and we are currently working on the use of Excel Services, part of Microsoft Office Share Point Server 2007, to further enhance the centralized approach to spreadsheet.
 
If you are interested in how Xenomorph solves the problem of spreadsheet management, then take a look at our (newly updated) website. Here we explain how to solve the problem and how Xenomorph Spreadsheet Inside technology can bring unstructured spreadsheet data and complex calculation within a centralized data management system, increasing transparency and reducing operational risk.

Data management, derivative analytics and the spreadsheet

Interesting article out doing the rounds on the newswires announcing a forthcoming report called "The Enterprise Spreadsheet: Pushing towards Transparency" by the analyst firm the Tabb Group. It is great to see an analyst firm acknowledging the importance of spreadsheets within the markets, particularly in the area of combining data and analytics together in OTC derivatives management (see earlier post).

Adam Sussman of the Tabb Group reckons that despite its shortcomings, Excel is a valuable tool: “Spreadsheets, either alone or in conjunction with other components, can meet the same requirements as a business application.” In this he seems to be agreeing with the UK Regulator the FSA, who have been recently advocating that spreadsheets and spreadsheet data needs actively managing as an institutional resource. The findings of the Tabb Group on management also seem to echo a recent report called "Buy-Side Data Management in a Changing Landscape" done by Lepus for Asset Control (registered link to report here).

Spreadsheets are a great tool and fulfil a real need in the market to pull together pricing models and data quickly, easily and with a timeframe that is meaningful to the business (see earlier post for some work by Xenomorph in this area). Spreadsheets are a big problem to manage, but they are also the symptom of failings in core systems that are not able to rapidly support new instrument types and pricing models. An institution that ignores analytics, spreadsheets and spreadsheet data within any EDM transparency initiative has already failed before it begins, and so to paraphrase the author Aldous Huxley:

"Spreadsheets do not cease to contain data because they are ignored."

Happy Birthday Spreadsheet!

Article on PCMag saying that the spreadsheet is 30 years old. Whilst wishing it a happy birthday, the author, John C. Dvorak, has a good rant about how spreadsheets have been the major weapon in the rise to power of the accountant in business.

Good job he did not spend too much time looking at their usage in financial markets or else his rant would have been much longer, given past issues with spreadsheets in financial markets. The spreadsheet (which means Excel at the moment) is a great tool that is:

• a calculator;
• a report writer;
• a database

In my view it is the latter usage of desktop spreadsheets to store data where the problems mainly reside, not its usage as an analysis tool. Faced with inflexible trading and risk management systems that do not allow instrument and trade data to be represented quickly or correctly, it is unsurprising that traders, portfolio managers and risk managers resort to spreadsheets as the "pressure relief valve" for their business activities.

Delivering systems that can support both complex and non-standard instruments and trades in a transparent manner should be a focus in a world where that the lack of transparency over credit derivative pricing has been such an issue. The inappropriate usage of spreadsheets is a very small part of the current problems we are experiencing in the markets, but addressing this would be a positive step in creating a data management foundation that encompasses all data used by a financial institution, not just that data that is easy for software vendors to represent in their systems.

Anyway, enough of my spreadsheet hobby-horse, for some light relief and to celebrate 30 years of summing rows and columns, take a look at the Eusprig web site for a list of the most notable spreadsheet failures.

Unstructured Data Management Anyone?

Good example on Finextra this week of another spreadsheet debacle, this time involving Barclays and Lehmans (see article).

Not sure when the financial markets will get serious about managing unstructured data and spreadsheet usage. We can all apply Enterprise Data Management until the cows come home, but if the "real" business is being managed in spreadsheets then really why bother with grand aims like EDM?

Even the regulators are not totally against the usage of spreadsheets (see presentation), then simply want unstructured data to be managed properly as part of an institution's formal processes and not ignored until the inevitable problems arise...

Spreadsheet meets database at Cornel's Arxiv...

The paper I presented at a "spreadsheet-risk" conference a while back has now been approved and entered at Cornel's Arxiv academic archive, see:

http://arxiv.org/abs/0802.2932

The paper details some of the work that Xenomorph's product development team have done on combining spreadsheet ease of use with database consistency and centralisation. We call this tech "SpreadSheet Inside" and a more detailed whitepaper on it can be found at http://www.xenomorph.com/downloads/whitepapers/spreadsheet-inside/. Clients are using it to centralise and control access to spreadsheet calculations and also using it within (extremely!) complex reports for risk management and trading.

Many thanks to Grenville Croll at Eusprig (www.eusprig.org) for assisting on this!