Rambling Thoughts

2005-07-12T03:26:00.000-07:00

Rambling ThoughtsThis post was an article that I wrote in December 2004 whilst working at a management consultancy.

This was extensively reviewed by one of my colleagues at the time who I take my hat off to for helping.

I've checked that they haven't done anything with the original article first.

Enabling the Agile Enterprise

2005-07-12T03:09:00.000-07:00

Every day news ticker tapes overflow with headlines: armed conflicts, scientific discoveries, mergers and acquisitions, new products, new regulations, new partnerships, interest rate decisions, capital market movements - the list goes on and on. And those are only the observed and reported events. Both reported and unreported events impact the lives of individuals and organizations around the world. More often than not they demand action and response.

The world is becoming increasingly dynamic and uncertain. In all of this uncertainty one certain thing remains: the pace and scale of change will only grow. Business agility will become a competitive advantage by itself. This article discusses the business implications of agility and how emerging IT techniques can be used to help organizations respond to change.

Facets of Agility and Organizations’ Response to Change
Merriam-Webster Dictionary defines “agility” as a state of being
1 : marked by ready ability to move with quick easy grace
2 : having a quick resourceful and adaptable character

Modern organizations have to respond to a number of changes in their operating environment. These factors are both internal and external to the organization. Economic shocks, social trends, world politics, competition, technology, financial failings, and mergers can all have an impact. Examples of events that have influenced businesses recently include the collapse of Enron, and the introduction of various associated regulatory measures such as the Sarbanes Oxley Act (SarbOx).

Organizations respond to shifts in their operating environment in broadly three ways:
• Strategic change – changing products, services or target markets
• Operational change – changing operational practices and procedures
• IT change – changing computer systems to support strategy or operations

An organization’s response to change is analogous to the human reaction cycle of perceive, think, react, and result. This is demonstrated when someone brakes in a car. They perceive some signal, such as seeing the brake light on the car in-front. The information travels up the optic nerve from the eye (perceive), to the brain which calculates a response (think), causing the driver to press the brake pedal (react) with appropriate pressure that slows the car (result).

Businesses’ response to change is often not quick enough. Changes in their operating environment are rarely perceived or understood for some time. When a company does alter its strategy or operating model to respond to some event the company often goes through a transition period, the reaction to the event. During this time new systems and processes to support the strategy are developed, the organization may be re-structured and so on. An organization’s response to change often causes a noticeable drop in performance as it goes through the transition period. A drop in performance may contribute to a drop in revenue, reduced product and service quality, or increased operational costs.

An organization’s performance in response to change is shown in the diagram below.

Compared to the organization’s ideal performance, the “event” and subsequent response represent a decrease in performance and value lost. Improved business agility reduces this. It allows the benefits of strategic change to impact the organization’s bottom line quicker and reduces the cost of change.

The Agile Response
Through improved, more accurate, and timely reporting mechanisms an agile enterprise will be able to perceive the potential impact of events quicker. Alternate strategies can be rapidly assessed as relevant data will be more easily accessed. Agile enterprises can then react by implementing the required operational processes and IT systems with equal speed and ease. This will allow the results to be realized sooner and with less reduction in enterprise performance.

Agile IT
Information Technology (IT) is often seen as a business enabler and a cornerstone of business agility. Technology can be used to improve an organization’s performance in each of the phases outlined above. IT can be used to:
• Help to spot that the environment is changing, to provide early indications that the enterprise needs to respond to some event
• Help determine the best enterprise response, to provide information to support decision making
• Help implement solutions quicker, to allow changes to systems and processes to be implemented quicker and with higher quality
• Help assess the result, adjust, and iterate over the response as necessary

To help enable business agility new IT systems must first overcome some of the challenges that legacy systems pose.

Legacy systems are among the most valuable assets in a business. They may represent massive investment over years or even decades. In addition they are likely to contain data about some of the core elements in the business such as customers and products. When companies merge they find that each half of the merger has their own systems for representing these. Replacing these legacy systems after so much investment, both in time and money, is often prohibitively expensive and operationally undesirable.

There have been a number of recent developments in technologies and architectural styles that have helped companies overcome their legacy issues and become more agile. These include:
• Service Oriented Architectures
• Enterprise Information Integration software
• Message Oriented Middleware
• Business Process Management software

These can be used in isolation or together to improve an organization’s agility in different ways. Each technology and its contribution to an agile enterprise is discussed below.

Service Oriented Architectures (SOA) are not a new concept within IT. However partly due to increased vendor marketing they are gaining popularity. SOAs encourage organizations to expose their existing systems as a series of services. For the purposes of this discussion a service is a collection of well defined operations. These services can be used by other projects or services through standard mechanisms. New software projects can also use the concept of services to provide the ability to re-use components from day one.

Vendors are encouraging the use of standards and technologies known as “Web-Services” though this is not a pre-requisite. What is important is that the technology chosen to implement the services is widely used within the company and is standards based. A number of different tools are available to make the development of these services as simple as possible.

Large organizations often have a lot of legacy applications. These may need to be re-used by newer applications to offer new services and products via various delivery channels. One large UK based Telco found themselves in this situation. They had several important legacy systems that needed to be re-used. Like many other organizations the Telco found the option of rewriting the legacy system unpalatable and extending the system virtually impossible.

In order to reduce development time for new applications a web-service based service layer was written to expose the legacy systems’ functionality. The web-services were then used by new applications. The complexities of the legacy systems were completely hidden from the new applications. By hiding the complexity new applications could be delivered quicker. Quicker application development improves the ability of the IT department to react to changes.

Enterprise Information Integration (EII) tools provide mechanisms for unifying the data contained in legacy systems. They can be thought of as a “database of databases”. They allow information from various databases to be collated in a central location. Some solutions actively retrieve the information and store it locally; others provide the logic to retrieve the information from disparate data sources. The more advanced EII applications can query data contained in a number of different data repositories such as databases, legacy applications exposed as services, or files.

EII tools are often used in conjunction with an agreed “canonical data model”. A canonical data model is an enterprise wide data-structure. This type of data model is also known as a reference data-model. Data contained within disparate applications then maps to this data-structure.

As part of an initiative to improve customer service and efficiency a large UK insurance firm wanted to provide a unified representation of its customers. By doing this they could centralize their customer support functions across the lines of business and off-shore part of the operation. Customer information was held in a variety of systems, one for each line of business. They investigated the use of EII tools to extract information from Cobol based systems that had been exposed as services. EII tools proved capable of meeting their requirements. In addition EII tools could do so at a fraction of the development cost and time required for a traditionally developed system.

The use of an EII tool can reduce development time, part of the reaction phase, thus increasing IT agility. Its use in scenarios similar to that described above can help to improve operational effectiveness and agility too. EII tools can also be used to provide data for executive dashboards. Unifying data from different systems in real-time can be a powerful tool in management decision making, improving strategic agility and the organization’s ability to perceive changes in the market, understand their own performance and then adjust accordingly.

Business Process Management (BPM) tools, which can include workflow functionality, can be used to develop process driven applications. BPM tools often structure application logic using a workflow based language similar to flowcharts. In contrast to pure workflow tools, that simply model existing business processes, BPM tools provide an execution environment for these workflows. Process driven applications can be based around existing business processes that may have been modeled using a workflow tool. Tasks within a process are defined in such a way that changing the process involves simply moving the task to a different place in the workflow. The closer system interaction mimics existing business processes the more easily changes to the process can be implemented.

Knowledge of business processes also plays a large part in regulatory compliance. Vendors have realized this and some even provide specific templates for their BPM tools to help support SarbOx compliance. Vendor supplied templates provide a good starting point for further customization by the enterprise.

The investment banking arm of a large multinational bank has used BPM tools to provide trade exception management facilities. When the validation of a trade fails it is routed to the BPM based application. The application goes through a number of steps to try and correct the trade data before routing the data to an operator for manual correction. The steps that have been automated would have traditionally been performed by the operator. By using a BPM based tool changes to the validation process can be easily reflected in the system improving IT’s ability to react quicker.

BPM based applications can also be used effectively in enterprise application integration (EAI) scenarios and business-to-business (B2B) e-commerce applications. Using a process to implement a B2B interface allows the contract between two parties to be better understood and adhered to. BPM tools are often used in RosettaNet, a B2B protocol, applications.

Message oriented middleware (MOM) provides applications with an asynchronous communications mechanism. MOM applications generally have two types of clients: those who consume messages and those that produce messages. MOM applications either send messages to a single listener (peer-to-peer communication) or to a group of listeners (publish/subscribe communication). Vendors have started to provide value added features, such as routing and transformation, on-top of basic MOM. These often go under the name of message brokers, and more recently Enterprise Service Buses (ESBs).

Consider the situation where the number of IT systems in a business has grown through M&A activity such as Vodafone’s did through the late 1990’s. In this case there are likely to be systems performing duplicate or similar roles that will eventually need decommissioning to reduce operational costs. The act of optimization and decommissioning may change which IT systems participate in a given business process. It would be desirable if systems could be removed or added almost at will with minimal impact to other systems or the business processes.

Software engineering has been solving this problem in computer programs for years through the technique of “de-coupling”. The theory being that the more tightly coupled two components, or systems, are the harder it is to change either one without affecting the other. Tight coupling reduces the system’s and hence the organization’s agility. An asynchronous communications mechanism, provided by a MOM application, allows systems to be completely decoupled from each other. This improves an organization’s ability to react quicker to changes in the number or types of systems being used. MOM tools can be a central part of an organizations’ architecture and so need to be suitably fault-tolerant.

Analysts talks about the “real-time enterprise”, providing real-time access to corporate data as it is created or received. By having instantaneous access to data organizations can produce more accurate reports and armed with this information make quicker and more informed decisions. The delivery of real-time data frequently involves the use of MOM tools.

Other Parts of the Puzzle
These new technologies can only do so much. Developers still have to assemble these components into meaningful applications. Various communities have sprung up focused on how to improve the efficiency of the development process in an effort to improve IT and developer agility. The Agile Alliance is one such community complete with their own manifesto that includes statements such as:
• “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”
• “Working software is the primary measure of progress.”

After years of ever increasing time-scales, escalating costs and development teams procrastinating over the delivery of software these ideals will come as welcome news. By delivering working solutions to the users more frequently it is possible to start gaining benefit from the solution sooner. In addition early feedback from the users allows improved software quality.

The solutions that have been discussed in this article only address long-term agility issues. Another dimension that should be considered is how to respond to short-term fluctuations in demand, whether from an operational or IT perspective. The manufacturing industry has had to respond to short term fluctuations in demand for many years. Fluctuating demands in IT systems, especially from a capacity point-of-view (computation power, network and storage capacity etc.) are the focus of many emerging products but will not be discussed further in this article.

Unfortunately a bevy of acronyms, vendor solutions, and developer communities will only provide part of the solution. Organizations must have effective project management, architecture, investment and leadership capabilities to implement these IT solutions successfully. Project management is required to manage the dependencies between projects, mitigate the risks and issues, and work with a project sponsor to communicate the benefits of a given project to the wider organization. They should also work with the architecture team to understand the system dependencies and determine the most appropriate project phasing. The architecture team ensures that there is a wider enterprise view and that new solutions are adopted by other development teams.

Flexibility and agility are not free. To become more agile organizations need to make significant initial and on-going investments. Investment in agile solutions will increasingly be required to enable organizational survival. There may be some early return on this investment but the real benefit will be when future change is required. Leadership will be required in order to justify this investment to stakeholders, to help the organization see the benefits of business agility, and to help enable it.

Putting IT All Together
So the organization has selected a set of technologies that it will use to reduce the cost of change. It now faces the daunting task of implementing the solution - on time and on budget.

The organization may have undertaken large scale development projects before and may have used project management teams and methodologies. How effective were these teams? If external consultants were used did they represent value for money?

Many organizations simply do not have experience in managing projects of the scale typically associated with enabling business agility. In order to ensure the success of their projects they often turn to external consultancies. Unfortunately the use of external consultancies often causes organizations to feel disengaged from the project. In addition anti-consultant sentiment can run high amongst the organization’s employees. This is especially true in large scale developments. There are a number of points that should be considered when investigating the use of external or internal project management functions. These include:
· What experience does the team (internal or external) have in delivering this type of project?
· Is the team experienced in handling cross project, often multi-site dependencies and risks?
· Is the team experienced in communicating project successes and failures?
· Is the team experienced in handling complex project roll outs?

Large organizations have an even more complex problem. They often have to manage the interaction of several, competing systems integrators (SIs). Trying to successfully manage these projects can be a challenge for the most experienced project management teams.

Architecture teams are often perceived to sit in an ivory tower, dictating what the development teams do. Effective architecture teams should not be viewed in this way. Any indication of an ivory tower architecture team is often a sign that their communication ability needs to be improved. It is imperative that the architecture team’s vision is communicated successfully with all relevant members of the organization. Only by doing so can the architecture be executed effectively.

Development teams often complain that the architecture team is out of touch – that they do not properly understand the technologies or solutions that they are proposing. One view is that architects need to have had experience in a development team prior to becoming an architect. Having some development experience, whilst undeniably beneficial, is not a pre-requisite for effective architecture teams. Having the ability to liaise with other teams in the project in a meaningful, jargon-free way is often far more important.

Companies sometimes seek to try and demystify the view of architecture through a governance policy. Without appropriate care and attention governance can actually do the opposite and increase the view that the architecture team is at odds with the development team and company goals.

In many cases strong leadership can makeup for deficiencies in project management or architecture teams. Strong leaders are able to uncover the route cause of problems quickly and resolve them.

Unfortunately problems are a daily part of project life. Some are more serious than others. Projects face problems that include opposition or financial competition from other projects, fluctuating scope, and shrinking deadlines. Strong leaders are required to gain support for their project and convince budgetary approval bodies of the financial justification of the project. For projects which may not have significant short term benefits this is a very difficult process. The ability to communicate effectively is even more important for the leaders than it is for project management and architecture teams.

Summary
Business agility is increasingly becoming a pre-request for sustained success. As new markets, new opportunities, new competitive threats, and new regulations emerge this need will only become more apparent. Businesses will need to become more agile and responsive to change or loose out to their competitors.

In order to become more agile businesses will have to examine their leadership and also their IT systems and departments. Some IT departments will find it easier to make the shift to agile thinking than others. However tools and techniques such as Service Orientated Architectures (SOA), Enterprise Information Integration (EII), Business Process Management (BPM) software, and Message Oriented Middleware (MOM) will all help to make this transition smoother.

If the concepts of agile business seem alien now there’s a multi-billion dollar IT industry betting that they will not be for very long.

Why Not Web Services?

2005-07-12T03:03:00.000-07:00

Motivation
I’ve been reading the Economist for a while now. It offers me a good digest of the week in politics, business, the arts, and occasionally things I’m really interested in - technology. The edition I was reading recently actually went as far as to include a sizable section on Web Services. The current “Geek Chic” of technology had reached the Economist!

But wait a moment, isn’t the Economist read by some of the most influential people in business (not including myself at this point)? Doesn’t that respected journal have profound impacts on these people’s opinions? Did it cover some of the most critical aspects (in my opinion) of Web Services? No, it didn’t.

The trade press, the CXO publications etc., have all talked about Web Services. They talk about the benefits this “miraculous” technology brings to an enterprise: the reduction in IT cost, the simplification of the data-centre, additional agility… the list of superlatives, claims, and products just keeps growing. But what about what Web Services won’t do? How many articles are there about that? In order to satisfy my cloying doubt I turned to Google, who incidentally have a Web Service available, and had a look. Disappointingly only 12 documents matched my search criteria.

The problem is that with so much positive press about their benefits it is difficult to find a balanced opinion. And in order to make and executive decision you need a balanced opinion. By examining certain business case studies this paper aims to address not what Web Services will do, you can try any of the countless documents in cyberspace for that, but what they will not do.

Introduction
Web Services started becoming the talk of the IT industry back in 2000. I remember reading several articles discussing a raft of acronyms (UDDI, SOAP, XML, WSDL) and scratching my head. Hadn’t the industry just dealt with Y2K, wasn’t Java (J2EE) the way forward. Here I was being presented with yet again another technology option to solve my business issues.

At a very high-level Web Services expose a series of operations to a client. To access the Web Service the client needs to be aware of very little. Issues such as system A being Unix and system B being Windows doesn’t matter – the use of common standards has seen to that. All the client needs to do is submit a request too a known location and they receive a response. It’s as simple, and as complicated as that.

It took another 18 months or so before vendors started seriously promoting Web Service technologies. Amid the rash of early adopters there were many early mistakes. I’ve presented some of the most common using either real or fictional case studies as examples.

Case 1 – Quick-Fire!
Overview
In this case study a large financial institution is dissuaded from using Internet Web Services as part of their risk management framework as a result of the performance implications of their usage pattern.

Context
One of my clients was a large-scale derivatives institution. They were considering the use of an Application Service Provider (ASP) that could be used to out-source their risk management function. My client’s internal risk-management function used aging technology which would be expensive to replace. An out-sourced solution would mean the ASP had to worry about how up-to-date the risk algorithms and systems were. However was an ASP and Web Service approach correct?

The Problem
Risk management is one of those “hard” problems. It involves complicated maths and is seen as a black art performed by a rare breed of people called “quants” or quantative mathematicians. Banks tend to be quite secretive about the precise algorithms that they use since this can erode their competitive advantage. Finding an ASP vendor who would offer this service was a problem in itself but not insurmountable. The bigger problem was how my client currently used this information, or rather the rate at which they used it.

Derivatives are priced based on the spot price of the underlying stock. So a Vodafone option would be priced on the spot price of the Vodafone share traded on a given stock exchange. As the underlying price of this option changes the value of the option needs to be re-calculated. In addition each underlying stock rarely equates to a single derivative instrument. There can be tens if not hundreds of options and futures for each underlying equity.

There were two main factors why this client couldn’t have used Internet based Web Services. Firstly the amount of data that would have needed to be transferred would have required considerable network bandwidth. This would have been compounded given the use of verbose communication protocols such as XML (eXtended Mark-up Language) and its Web Service sibling, SOAP (Simple Object Access Protocol). Secondly the rate at which this information would need to be provided, simply expressed as transactions per second, would likely be outside the capacity of many, if not all, Internet based Web Service applications.

The Solution
The solution was quite simple: the client couldn’t implement Web Services where they wanted to. They were then left with the situation where they could either do nothing for a while longer, or tackle a major systems improvement programme. Their existing approach still worked, and they could still afford to keep it so for the time being at least, their existing risk management application is still running.

One thing to note at this point is that they may have had less performance issues if they’d considered the use of internal Web Services. This would have removed much of the network overhead of the internet.

Case 2 – Fort Knox
Overview
This case study shows a situation where the security levels required prevented the use of Web Services.

Context
Whilst working for a government programme I was asked to review their technology choice. They wanted to distribute highly sensitive information between different governmental bodies throughout the country. This information had to be distributed using not only a secure channel, also known as Link Level Encryption (LLE), but each message had to be encrypted and digitally signed. If these security requirements weren’t enough the system also had to be standards-based.

The Problem
Web Services have been offering link-level encryption for a while now. Nine times out of ten this is the same type of encryption you find when you access your bank details over the internet. However the use of a standards based message level encryption and digital signatures prevented further consideration of Web Services at this time.

The various Web Services bodies are currently colluding to develop a series of specifications known collectively as WS-I. In fact there’s a whole website dedicated to it (www.ws-i.org). Amongst these specifications is WS-Security which addresses many of the issues faced by my client. However this specification is still in its infancy and at the time was unsuitable for my client’s application.

The Solution
The Dot-Com era had seen the development of many protocols used for B2B communication including the ebXML protocol (www.ebxml.org). This has been in use since 1999 is reasonably supported by the industry. Some people believe that ebXML and Web Services standards have a great deal of synergy. The Web Services bodies are borrowing several of the more successful ideas from this and other standards. At the time ebXML was seen as the most established and supported protocol for their application.

Case 3 – My deposit account
Overview
This fictional example of an on-line backing service shows where issues with transactions would prevent the use of Web Services.

Context
The Internet banking phenomenon has revolutionised my life. I can be at work or home and move money from one account to another. I can pay my bills, view my balances – I even have a mini-statement sent to my mobile every week. I’m now more in control of what’s going on in my finances than I ever was before.

Most Internet banks have a standard web-based user interface which is accessed over a secure connection. This provides me with all the features I need to carry out my financial transactions on line. I’m pretty sure that if I confirm a transfer from my account to the electricity company’s the money will be transferred on the day, and for the amount I expect. If it didn’t I’d be pretty angry and I’m just a regular consumer. Imagine banks transferring millions of dollars or pounds from one major account to another. The prospect of this information going “missing” would be likely to result in legal action!

Could I use Web Services as a mechanism of accessing the information in these various institutions?

The Problem
The reason that I’m reasonably sure that the money will get transferred as I requested is that my request will be carried out as a “transaction”. In this context I’m talking about a transaction in which all work is carried out all at once, or not all. There is no half way house. The details of transaction processing are quite convoluted so I’ll avoid a discussion of this for the moment.

Most of the information systems in financial institutions are transactional. This allows them to participate in the interactions I outlined above. Unfortunately Web Services aren’t transactional. There is a lot of complexity in transaction semantics. The use of the Internet as a communications medium and the fact that Web Services may be hosted by another organisation just add even more complexity.

The Solution
Once more there are several lessons that can be learned from the B2B community. One common use-case for B2B applications in supply chain management. Orders placed using a B2B application are sometimes worth in excess of a million dollars, especially in the automotive industry. Here the failure to place an order can result in delays to the production line and massive costs.

The way in which these applications overcome this issue is to have well defined mechanisms for message receipt and acknowledgement. For example if I used one of these applications to place my order for some widgets I would expect to get a receipt that the order has been accepted. Through a defined relationship with my supplier and agreed service levels I would know that if I hadn’t received a receipt within a given time I should try again. This process is known as reliable delivery and again a WS-I (WS-Reliability) standard is addressing this.

Conclusion
Web Services are undoubtedly having a great impact on IT within the enterprise. The chances are if you haven’t at least one project that uses them you soon will have. Web Services have a huge amount to offer enterprises. However just be careful that you don’t get caught out by any of the situations outlined above: performance, security, and transactions.

Certain vendors provide pseudo standards-based or proprietary capabilities to develop high-performance, secure, transactional Web Services and various techniques can be used in addition to re-enforce this. However this sophistication will come at a cost which a project may have to bare and senior IT management need to be aware of. Web Services are revolutionary but not a panacea – at least not yet.

The number of emerging standards and focus by the majority of the IT industry are undoubtedly a good sign. This leads me to believe that a majority of the current issues with Web Services will be resolved over the coming months or years. However in the meantime at least there’s one more paper dedicated to a discussion of “Why not Web Services”.