What Every Executive Needs to Know
About Information Technology (IT) Projects
Dr. Paul Dorsey
Dulcian, Inc.
Most of the Information Technology (IT) industry is populated by hard-working professionals who start out as developers. Some become team leaders or system architects; however, few of these people ever seem to find themselves in charge of very large projects. The very large (hundreds of millions of dollars) IT projects always seem to be run by professional managers or administrators, few of whom have any recent experience "in the trenches." Often, these politically savvy managers rise quickly through the ranks of large organizations. Although they may have very effective political and people skills, they have little understanding of the factors that cause system projects to succeed or fail.
Those individuals who are technically focused tend to rise through the ranks within the IT group itself, but rarely make it to top management. They are frequently able to participate in large projects, but rarely have enough power or influence to steer the direction of the project as a whole.
The result is that significant technical architectural decisions are made by individuals who have neither a technical nor a system architecture background. Meanwhile, the senior technical personnel sit on the project sidelines watching the next multi-million dollar system failure, unable to make their voices heard or get someone in authority to listen to them.
This article represents input from a number of senior information architects
responding to the statement: "If I had a chance to tell the people in
charge what they need to know about large software projects, this is what I
would say."
One of the great lies perpetrated in the software industry is that our projects are not like other projects. They require totally unique thinking. Wisdom gained from other types of projects does not apply in IT.
When I co-authored my first book dealing with systems development (Oracle Designer 2000 Handbook, Oracle Press, 1996), I gave it to my brother to read. He had many years of experience in the electronics industry, having been an executive at a few large companies. His response to the book was curious: "This is all good stuff, but is it really news to IT professionals? Most of what you are describing is good project methodology, but it does not seem particularly unique to software development.” Unfortunately, most project management concepts are news to IT professionals. The main focus of most IT professionals tends to be placed on the next new technology and often ignores project management. At IT conferences, the most widely attended presentations are those about features of the newest technologies while presentations about system architecture and project management are largely ignored.
If you do not have an IT background or training as a system developer or DBA, you can still manage an IT project. You need to lay out a plan that spans the project launch all the way through to implementation and system maintenance. You must periodically check to make sure that the project stays on track. Dealing with organizational politics, interpersonal conflicts, and managing expectations is the same for any project.
When there are unrealistic deadlines, impossible specifications, incompetent decisions, project failure will follow. Software projects that follow a clear project plan can be built on time and within budget. However, if the requirements change, you can expect project delays.
A good, non-technical project manager can bring value to an IT project; however, the lack of domain-specific knowledge will require more interaction with the technical project lead in order to determine the deliverables and milestones.
Ultimately, the responsibility for the failure of any
project cannot be shifted to the contractor, or to the Commercial-Off-The-Shelf
(COTS) package. For example, the blame for the recent failure of the
Every book and article ever written about systems development indicates that user involvement is critical to success. Yet it is very common for the involvement of users in systems development projects to be superficial. Users are often "too busy" to give up time to provide feedback or be involved during system development. They are reluctant to take ownership of a systems project and often wait to be presented with a "completed" system that often does not meet their requirements before they take a meaningful interest in the project.
Systems belong to the users they support, and not to the members of the IT department who developed them. Users should be in control of every non-technical aspect of the system and play an integral role within the development team. If some users who will ultimately be working with the system are not assigned to work with the developers throughout the entire life of the project, there is very little chance of project success.
There are several competing theories driving the process of software development. Some approaches such as the Rational Unified Process (RUP) have tools to support them, while others lack any significant tool support. Most traditional database developers adhere to some type of iterative waterfall method driven by careful analysis and a structured approach. The current trend among object-oriented (OO) developers is to use some flavor of Agile technology, which is characterized by rapid-fire delivery of iterative system versions. Adherents to each approach are frequently highly critical of other approaches.
Several years ago, I was contacted by a graduate student who was working on a thesis comparing system development methodologies. The student wanted to create a contingency model that would indicate which style of development was appropriate for different types of projects. I was asked to give my opinion about when each development methodology should be employed. I thought it was an excellent question; however, I don’t think the answer I gave satisfied the student very much. I replied that the software development method employed had very little effect on project success. I have seen successes and failures using every method employed. If the development team used no formal process at all, failure was guaranteed, but it did not seem to matter which development method was used.
The analogy I often use is that project management is like leading a bunch of children on a wilderness hike. Periodically, you should climb to the top of a tree or hill and see if you are still going in the right direction, find out if there are obstacles in the way, and make sure that the path you are on will get you to the final destination. Climb back down from the tree, gather all of the children around (most have wandered off into the woods), and point them all in the right direction prior to moving on.
As simple as this description of project management sounds, it is shocking how often managers do not follow it. Project managers may go for months (if not years) without taking stock of the real status of a project. As a result, the architecture of the system may be unsound, but this critical information will not be discovered in a timely manner. Sometimes an unsound system foundation is not discovered until there is an attempt to put the system into production. Of the many shortcuts that are taken in the name of expediency, there is none more dangerous than failing to make a periodic assessment of the overall project status.
Periodic assessments can help prevent several common problems:
· when determining project scope
· when the data model is thought to be relatively stable
· after the basic user interface design is complete
· after each major architectural decision.
Such assessments are best made with the assistance of an outside auditor who is not otherwise connected to the project. The act of defending a decision to a third party often helps to focus the thinking of the team.
Over the years, many organizations have often been lured into the “next new thing,” from too early adoption of relational databases in the early 1980's, the horrors of Computer Aided Software Engineering (CASE), and the failures of early Object-Oriented (OO) projects. The latest industry trend is that Service Oriented Architecture (SOA), Agile programming, and abandoning databases will solve all of our problems. For the last 25 years, we have been told by the IT industry, that the next new thing will solve all of our problems. Yet projects still consistently fail.
New knowledge enters into the field of science though a rigorous process of peer reviewed journals using proven scientific methods. In IT, new trends and technologies are adopted through a market-driven mechanism largely controlled by a few thought leaders.
Despite the perception of a large, diverse IT community, industry conventional wisdom is actually controlled by a very small group of individuals, many of whom know each other personally and who usually agree. This group tends to be reluctant to listen to any discordant voices from outside of their circle. Therefore, instead of a free flow and exchange of ideas, the industry movers and shakers consist of a relatively small number of "clubs" of homogeneously-thinking individuals who are resistant to change.
Moving from one "club" to another is virtually impossible. In my own experience, I have been a very active member of the Oracle database community for many years. I am familiar with how the "best practices" in the database community have evolved and have been able to participate in their evolution. As the industry shifted towards web development and Java, I began to participate in some of the Java community events. However, I have been totally unsuccessful in securing opportunities to speak at major OO conferences. Being a “database guy” makes my ideas suspect. I have tried to reach out to the OO community and get OO speakers to speak at Oracle conferences, but have been mostly rebuffed by leading OO thinkers. As long as this type of community exclusion persists, the database people will remain ignorant of many of the advances in the OO world, and the OO people remain ignorant of the value of the database.
The thought leaders in the respective groups have evolved very different sets of best practices. OO people have embraced Agile methodologies and tend to minimize reliance on the database. Many database professionals have moved towards a heavier reliance on placing information into the database("thick-database approach") and minimizing reliance on non-server side code.
Most organizations include a thought leader who embraces a particular conventional wisdom. Most recently, more technical project leads favor the OO approach, so reliance on the database tends to be minimized. Often the database professionals within organizations that are led by OO-slanted managers rebel against this approach. This creates a ongoing battle between the database and OO people within an organization, which usually results in project failure.
A disturbing trend within both groups is the lack of respect that members of each group have for members of the other. The database community views the OO community as architecturally sloppy and focused only on pretty user interfaces and do-it-yourself programming. Conversely, many OO developers view the database as virtually irrelevant and merely a persistent place to store copies of their object classes. This lack of respect on both sides often keeps the groups from working together.
The irony is that both groups are ignoring and therefore missing the wisdom gleaned by the other group about how to successfully build and maintain information systems. Almost all newly developed systems are object-oriented and web-based but also rely on databases as the persistent data store. Therefore, both groups clearly need each other.
The current trend in developing systems is to buy a Commercial-Off-The-Shelf (COTS) software package to avoid custom application development. Buying COTS packages is thought by many to greatly reduce the cost, risk, and time to delivery for a system. Many large government and private organizations have set strategic objectives to decrease reliance on custom-built applications.
Those of us who have worked on many projects over a number of years know that COTS development is neither less risky nor less costly than custom development. In fact, modifying a COTS package is often more time consuming, more expensive, and more difficult than a custom application development effort to satisfy the same system requirements.
The experience of a COTS implementation can usually be described as follows:
· After a brief analysis phase, a decision is made to purchase a COTS package and an integrator is brought in to handle the installation and migration.
· A budget of $2-$4 million is typically established and the project is initiated with the expectation that in 18 months-2 years, the new system will be ready and in production.
· After two years, the project is declared to be 15% complete with possibly 1-2 small modules in production. The estimated completion time is an additional two years with another $2-$4 million budget.
These types of projects are usually cancelled after 3-4 years, at which point, the project is started over again with a different COTS package touted to be better than the previous one. At a dinner (where significant quantities of alcohol were consumed), executives from one of the largest IT COTS vendors admitted that 50% of their large implementations fail on the first try. (Keep in mind that this is the statistic to which they were willing to admit.)
Most of the large COTS packages are created by functionally focused designers and developers rather than people who are focused on the overall system design. They are typically originally built in conjunction with a specific consulting contract for one client. The products are often not flexible or easily modified and are based on data models that include thousands (or tens of thousands) of tables with millions (or hundreds of millions) of lines of code written by many different programmers.
COTS architectures are frequently unsound. Since COTS packages are most often purchased based on "look-and-feel" and feature-laden demos, therefore the companies producing them focus more on the surface user interface features rather than the underlying architecture.
COTS packages can be implemented successfully if organizations are willing to reengineer their business to fit the functionality and structure of the software. However, this rarely happens. At some point, organizations inevitably need to extend or customize the COTS software packages. When this happens, the lid to Pandora's Box has been opened and you are effectively creating a custom-based system; but by starting with a COTS package, you end up in an even worse scenario, since you are now forcing a system designed to perform a specific set of operations to perform a different set of operations. This usually results in dire consequences. A number of years ago, a large insurance company spent $600,000 to purchase a payroll system and then spent over $6 million to modify it to meet their requirements.
One organizations' insistence on relying on COTS packages resulted in a completely nonsensical decision where a package designed to support Human Resources Scheduling was purchased to support IT project management. COTS vendors always claim that their product can be extended to support other requirements but, in the end, modifying COTS packages is the same (or usually worse) than ground-up custom development efforts.
Once modifications of a COTS package are made, if the vendor does not include those modifications as part of the core product going forward, it is difficult or impossible for the organization to take advantage of upgrades to the original product. Each time a new release is announced, all of the extensions to the old product must be rolled forward into the new version. Unless the modifications made were flexible enough, many millions of additional dollars will need to be expended to support newer versions. Frequently, older COTS product versions are unable to run on newer operating systems, database servers, etc. There are some organizations still stuck running Oracle Version 7 because of their inability to migrate a COTS package to a newer version of the database.
Custom application development frequently masquerades as COTS modifications. Organizations using COTS solutions will frequently be forced into modifying the software beyond recognition in order to support a specific set of requirements. Starting with a COTS package and making significant modifications is equivalent to custom application development. The resulting system often costs more and is less well designed than if it had been built from scratch. Transforming a car into a boat is more expensive than building a boat from scratch.
So why does this pattern of turning to COTS software as the "obvious solution" persist in the industry?
Not all COTS projects fail. There are a number of strategies that can be employed to increase the likelihood of making a COTS implementation successful:
To summarize, use a COTS implementation only if that software can successfully meet a very large percentage your organization's requirements, or you are willing to change your business practices to fit the COTS package. If your organization has unique requirements or is not willing to change business practices, custom application development may be the safer alternative.
The “wisdom” behind insisting that everything be built with open source tools is baffling. A 100% open source philosophy effectively asserts that no organization can ever build any intellectual value into a product. Clearly, there are costs to using non-open source technology. But those costs are not infinite, and so should be weighed against the value of the technology.
Open source projects benefit from a community getting behind a technology and having everyone participate in the effort. Some open source projects, such as Linux have been wonderfully successful. However, just as there are failures of proprietary products, there are open source initiatives that never take off, or are quickly superseded by the next new thing.
Organizations working on open source projects tend to have less of an economic stake than is the case for proprietary technologies. If an organization’s future is dependent upon a technology, there may be more willingness to make sure that it is sound (even at a very high cost).
Some technologies are very expensive to develop and require a massive, coordinated development effort to succeed. Data migration (often referred to as Extract-Transform-Load abbreviated as ETL) tools are a perfect example of this kind of technology. This sector of the IT industry is totally dominated by a few large vendors who can charge very large licensing fees for their products. The barrier to entry to this market is huge due to the cost of developing a good ETL tool. Ad hoc query and reporting tools are another area where high quality user interfaces and high development costs will make an open source solution unlikely to succeed.
Oracle’s Application Development Framework (ADF) technology is a perfect example of a proprietary technology stack that should not be dismissed without careful evaluation. Oracle has spent millions of dollars on this technology and will be using it for all of their internal application development. It is more architecturally rich than any open source alternative. Any organization using Oracle applications should carefully review this stack. Organizations building web applications to interface with Oracle databases should also strongly consider this framework.
Ironically, domain-specific open source package solutions have had very little success. Consulting companies take their successful projects and attempt to leverage that success by productizing the project and marketing it to additional organizations. Somehow it is acceptable to be locked into using domain-specific vendors but not technology vendors. This makes little sense.
The larger a system becomes, the harder it is to develop successfully. There are a number of reasons why it is true that the larger the project, the less likely it is to succeed:
The solution to the issue of avoiding mega-projects is not to attempt the large project in the first place. Rethink what needs to be accomplished and break it into smaller, manageable projects at the functional and not the implementation level. This is where a Service Oriented Architecture (SOA) orientation at the enterprise level is the critical success factor. If large projects can be rethought and divided into several smaller projects that bolt together cleanly, there is a much better chance of success.
Massive IT projects frequently imply totally reengineering an organization’s business processes. This kind of change is very difficult to undertake in a single step. An incremental, selective approach that doesn't attempt to accomplish too much too quickly will usually succeed. Even if the smaller project fails, it will fail faster and at a much smaller cost. Many lessons will be learned and the next increment will be more likely to succeed.
Do not reject expenditures on a project based solely on their price. Make buying decisions based on the economic value of the asset. There are several places where non-economic decisions are usually made that often contribute to project failure:
1. Not buying adequate hardware to support the development staff. I find it almost comical that an organization will pay a thousand or more dollars a day for a consulting resource and then make him/her use a 15-inch monitor with an underpowered computer attached to an inadequate network. At one large insurance company, I encountered developers working on an application that took several minutes to compile. Developers would avoid compiling the code (which increased their development time) and then take a coffee, cigarette, or Internet surfing break every time they tried to compile their code. When my company came to assist them, we brought our own hardware. When we showed the management team that, on our hardware, compilation only took a few seconds, they relented and upgraded the hardware.
2. Not buying adequate hardware to support the users. It costs much more to build a system to run on a 15-inch monitor than on a 19-inch one. It is harder to lay out the screen real estate when there is little space to work. Users will then be stuck with less productive, more unfriendly software, ultimately costing the organization much more than upgrading. Similar mistakes are often made in forcing software to run on an inadequate infrastructure, thus slowing down performance. Wasting a few minutes of each users' time every day costs much more than most hardware/network upgrades in the long run.
3. Not buying good software because of the license costs. If a software utility is going to make your developers more productive, it is probably worth buying. If the best platform for your project is more expensive than a poorer alternative, then you are probably better off with the more expensive alternative. The cost of software licenses rarely constitutes a significant percentage of the total system cost.
This is one of the most difficult questions for a project manager to answer since he/she often needs to find a team with expertise that they themselves may not have. In the field of medicine, there are industry certifications that guarantee a minimum level of competence. Certifications in the IT industry are all but useless. I once had a conversation with one of the software development certification marketing people who admitted that she had taken and passed the certification test, but was not able to build anything but the simplest applications. She had never even worked as a developer. Certification tests measure whether or not someone has memorized a number of details, but do not demonstrate whether or not someone can actually build a system. Meaningful certification requires demonstrating the ability to write working code. Oracle attempted to market this kind of extensive hands-on certification through their Oracle Certified Master program but few IT professionals were ever willing to sign up for the test.
Experience is not always a reliable predictor of ability. Many IT professionals with ten years of experience in reality have "one year of experience ten times over" because they have never gone beyond the basics of system development or worked on a very complex project.
Education cannot be counted on completely either to ensure valid expertise. Some of the most talented system architects and developers I have encountered and ultimately hired did not distinguish themselves academically in college. Others who possessed advanced degrees and/or impressive resumes were sometimes quickly fired.
There are two criteria to look for when selecting a systems development expert:
1) They must be connected to the larger community of other experts. No one person knows how to do everything. A true expert can call upon a host of other expert resources to answer questions and troubleshoot problems. Evidence of this connectedness is attendance and presentations at conferences, published articles, books, blogs, and technical postings on forums.
2) Successful experts have actually managed large projects. Be sure to personally check references and do not rely solely on the information contained in a resume.
Large projects are typically run by large consulting companies. But often, these companies may be the worst choice when it comes to project success. This is not to say that large consulting companies cannot provide value. They often have highly skilled people available and bring a vast pool of experience as well as solid infrastructures and great process manuals for their employees to follow for every contingency. With such wonderful resources and so much experience, why do large consulting companies have such poor track records?
The first problem is the hierarchical structure these organizations. Most of the truly talented people have been promoted to management or sales. Skilled resources move up or out quickly. The most valuable person in a consulting company is the one who brings in the large contracts. Therefore, the best talent that an organization may interact with on a project is the team that sells them on using the consulting firm. After the contract is signed, that level of talent may never be seen again. Large consulting companies are mainly strategic consulting companies. When they take on implementation projects, they frequently sub-contract the technical implementation to a different firm but still charge a premium for that talent.
Large consulting companies typically staff projects with "kids" just out of college. To be fair, they probably graduated from a first rate school and got good grades, but software projects require more than smart kids to succeed. Large consulting companies also charge a premium for that inexperienced talent, typically $150/hr or more. For that same price, you can often hire a senior independent consultant with 20 years of experience.
Large consulting firms are motivated to sell you the most expensive hardware and software they can. Most large firms have contracts with software and hardware vendors that grant them fees when they recommend the vendor’s products. Rarely do consultants recommend open source products.
What big consulting companies used to bring to the table was depth. If someone on the team could not do something, there was someone else in the organization who could. Now, experts in the industry are in constant contact with other experts through the Internet. There is no longer much advantage to using a big consulting company as the system integrator. Networking ability and being able to tap the right resources are more important skills than having a large development team. One of the impacts of Sarbanes Oxley was forcing large auditing firms to spin off their consulting organizations. This now means that the auditors, who deeply understand the financial parts of your system, can no longer work for the same company as the consulting firm.
Most large government contracts are awarded to large consulting companies. So how can you work with a large consulting company and still succeed?
Projects do not succeed because of what company name is on the business cards of the development team. They succeed because the project has a project manager and technical lead who really understand the technology and what it takes to successfully build a system. If a project doesn’t have the right technical lead and project manager, there is little chance that the project will succeed. When hiring a consulting firm, you should meet the technical lead and project manager who will be assigned to the project. Those two people matter more than everyone else on the team combined. The technical lead makes the technical decisions and the project manager manages the schedule and the politics.
Most projects will need other technical talents on the team who can be tasked with specific deliverables. In this case, a first-rate resource will outperform an average resource by a factor of ten to one, so it is worth spending whatever it takes for the required specialized expertise. A second rate resource will do more harm than good.
On one large project, I was given a team of 40 developers to manage and was told to get the project moving as soon as possible. I was told “We will give you whatever resources you need. Just ask for it, and it is yours.” My response was to get rid of 30 of the developers. Since this was a government contract and the large consulting company had all of these people billing, this was not a viable option. In this situation, I used the top ten developers to design the system and had the other 30 of them work on documentation, screen design, and preparing documents to assist with system certification. One week before the initial delivery, we were behind schedule and there was no chance of meeting the deadline with all 40 people working. I further reduced the team to the top two people, used the rest of the team to “help” and we delivered on schedule.
This paper has mainly focused on all of the things that can go wrong on a project. But there are many successful projects as well. In this context, "successful" means:
· The system went into production and met the users' expectations.
· The system did not cost significantly more or take much longer to build than was expected.
· The completed system is maintainable and continues to operate successfully as requirements change.
The following sections describe some criteria for making projects successful.
It is critical to set up incentives for project success. Typically, IT contracts employ the wrong incentive. Because they are billing for the hours expended, vendors are encouraged to work as slowly as possible and there are no ramifications for failure to deliver. The underlying message that a time and materials contract with no additional controls sends to a vendor that success is measured by how many people can bill on a project for as long a period as possible. This guarantees that large numbers of people will be billing at all times.
If managers are not held accountable for the failure of a project, they will not be motivated to ensure its success. For example, on a large government contract, bids were requested to work on a task that was already behind schedule and absolutely had to be delivered in three months. The company that was awarded the project contract was the one who presented the lowest estimated cost on a time and materials contract. To date, they have yet to deliver even though the “mandatory” deadline passed some 8 months ago. However, the same company is still working on the project, continuing to bill. If a consulting firm is hired to deliver a project and they do not deliver, why do they continue to be rewarded?
Effective monitoring has an instantaneous effect on project quality. Back in the 1970’s Atari purchased a significant percentage of the world’s computer chips. The quality of those chips was atrocious. Failure rates were up to 30% from some vendors and Atari was not monitoring the quality of the chips. An astute manager started measuring the quality of the delivered chips and holding the manufacturers accountable. Quality instantaneously improved.
Each branch of the
Monitoring progress on a project is very difficult. How does a manager know that the reports being received about progress on the project are accurate? Optimistic reports about how a project is on schedule and within budget are the norm. Up until the time that the project is nearly slated to go into production, things are usually reported as going according to plan. The higher one goes in an organization, the more sanitized any ongoing feedback about the progress of a project becomes. Everyone is reluctant to admit failure.
Project demos of evolving software are almost always meaningless. Anyone can fake progress in an hour-long demo. Until a system has been placed into production and users are actively using it and deriving value, the project has done nothing but spend money.
Set meaningful milestones that demonstrate actual progress. Consulting organizations are very good at putting on demonstrations to give the illusion of progress. On a project for a small insurance company, a huge amount of resources were devoted to making software that appeared to work for a critical design review demonstration to top management. The underlying software was completely unsound but the team was able to create partially functioning screens that were sufficient to convince management that the project was near completion. The project team threw themselves a dinner party to celebrate the successful “putting one over on management.”
Having someone working in the trenches of a project who can provide direct reports of project progress to the overall Project Manager is critical for success.
For 25 years, the database community evolved a knowledge base about how to best design and use a database. A well designed database can greatly reduce both the initial and ongoing maintenance costs of a system. Code that is data-intensive (meaning that it requires lots of interaction with data in the database) is much easier to write and will perform more quickly if it is written using the database’s native execution language and then executed directly by the database server.
Conventional wisdom is to pull code out of the database and instead to write it in the middle tier in Visual Basic or Java. The justifications given for doing this are as follows:
Both of these justifications are unsupportable.
The general argument for
placing all of the logic into the middle tier is to make the application
database-independent thus enabling easy movement from one database platform to
another. The reality is that very few applications need to be
database-independent. Companies change their underlying database platform about
as often as they move their corporate headquarters.
It is very rare that an organization chooses to shift their database platform. Learning how to support a DBMS is a quite complex task. Larger, more complex organizations have millions of dollars of intellectual capital invested in knowing how to support a DBMS. Shifting from one platform to another is a very expensive proposition.
Web architecture is
inherently more volatile than the database platform. Even if you are committed
to the Java EE platform, it continues to evolve. In the last few years, Oracle
has moved its preferred platform from JavaServer Pages (JSPs) to JSP/Struts, to
JavaServer Faces. Similarly, changes are being made to the .Net platform. Organizations may switch between the two
major platforms (Java EE or .Net) or to any of the variants (like Swing). There are also other platforms that
developers may use such as Cold Fusion or 4GL-like platforms such as Oracle’s
Portal or Application Express. Recently
SOA architecture has been added to the mix to further complicate the picture.
Every year, the choices
for web development change. There has been a constant evolution in the web
environment that forces a complete generational shift every 12-18 months, and
there seems to be no end in sight to this trend. The latest evolution is
towards “rich internet applications” where it is possible to achieve even
greater sophistication than in client/server applications a few years ago.
The defense against the chaos of a rapidly evolving standard is
to not build web applications with very much logic placed in these volatile web
technologies. The larger the percentage of the application in the database, the
less rewriting will be required with the next shift in web architecture.
Using the database to store much of the system logic and validation (a "thick database" approach) has many benefits. It minimizes development risk and helps build working applications that scale well.
One of the real strengths
of the thick database approach is that the two portions of an application can
be built independently. Once the
interface between these two parts of the application is defined, the teams can
work in isolation until substantive portions are working.
In practice, this approach works very well. Usually the first version of the user interface can be built within a few days so it can be used as the first version testing environment for the database team and feedback can be received from users.
Other advantages of a database-centric or thick database approach
include the following:
It is very easy to make an otherwise excellently built application perform poorly because of poorly written code. In some percentage of systems, the database is being "overheated" and there is the perception that performance could be improved by moving processing out of the database. These situations are very rare. Overheated databases are almost always the result of poorly written code that is passing much more information to the database than would be necessary if the code were written efficiently.
Database developers can
usually write highly efficient database code that can outperform the code
written by their highly skilled Java counterparts. The reason for this is that there are so many
options available to skilled database programmers that are not in the skilled
Java programmer's "bag of tricks." Therefore, database-intensive code
will always be more efficiently written in the database.
The thick database
approach provides a clear roadmap for application development, which simplifies
the decisions to be made with respect to the application architecture. Furthermore, the development tasks can be
neatly divided between the database team and the user interface team. This partitioning of the application
development effort effectively means building two smaller applications rather
than one large one, which is usually easier.
Because the user
interface can be built more quickly, it can be shown to users right away. This provides faster feedback to the
development team and helps to identify design errors much earlier in the
process.
Because the application
being built is divided into two parts, each has less code to maintain. In
addition, the application is clearly partitioned so that when a business rule
changes, it is only necessary to look through half of the code to find it. The
usual rule of thumb is that as the number of lines of code in an application
doubles, the complexity increases by a factor of four.
Using a thick database
approach allows organizations with existing database talent to use that talent
more effectively. With minimal
additional training, skilled database developers can help build web
applications with no web skills whatsoever.
If an organization has limited user interface development skills,
then thick database usage allows minimally skilled user interface developers to
build applications with little new training needed. If sophisticated developers are available,
they can focus on delivering very high quality user interfaces.
There are a number of additional factors that must be taken into account when building successful systems.
Achieving true system security is an increasingly difficult problem in the IT industry. Security measures must be designed into a system prior to its development. Usually the security of a system is evaluated after development is complete. At that point, either the security holes are not detected, or when they are detected, they are too expensive to fix.
Most security evaluations are quite superficial. Often they do not even detect simple cross scripting or code insertion vulnerabilities. Security experts rarely have much difficulty in finding vulnerabilities in certified systems. Audits that take place after a system is built and do not have the cooperation of the developers are meaningless and only provide the illusion of security.
One of the easiest ways to waste money on IT projects is the reluctance to recognize that there is a better way to architect the system and that the existing work should be thrown away and the project started over. Better system architecture can reduce the development cost of a project by up to a factor of 10. A poor architecture may become a money pit and still never deliver a working system.
Designing systems is a very expensive and time consuming task. The actual building of the system is comparatively inexpensive. Ask yourself the following question about any system: “If all of the source code were lost, what would it cost to rebuild the entire system from scratch?” If the original team is still around, the cost of rebuilding a system is usually less than 10% of the original cost of the system. The rebuilt system will usually require far less code and be architected much more soundly. Most developers, when looking at a system they have built previously, will exclaim, “I can’t believe I wrote that! I wish I could do it over.”
As development progresses, the level of understanding about the system deepens. This greater understanding frequently means that the original technical architecture is sub-optimal to support the evolved requirements. At this point, the original architecture can continue to be patched or you can start over with a new architecture. Frequently it is less difficult to start over.
One of the best indicators of a mature development team is its willingness to throw away work and do it again. However, it is usually very difficult to get management to agree to any rework. Among architects who understand the importance of rework, it will be hidden behind various euphemisms such as “upgrade,” “modernization,” or “taking advantage of new features”. Management will sometimes refuse to pay for rework by saying “Since you are doing rework, you must have made a mistake building it wrong in the first place. Why should I pay for your mistakes?” Such attitudes are counter-productive.
The reluctance to rebuild can greatly increase costs. Slowly evolving a bad architecture to a good one is much more difficult that starting over. We should all recognize that frequently the best way to deliver a working system is to start over.
Actually implementing a system can be more expensive than designing and building it. Getting everyone trained on the new system, retiring the old way of doing things, moving the data from the old system to the new, are all potentially huge and time consuming tasks.
Data migration is very often overlooked in system project planning. Everyone thinks that their data is clean, and it never is. Migrating data can cost as much as developing a new system. Usually the new data structure is quite different from the old one. The old data may have been collected with different details and it frequently violates the business rules of the new system. You must decide if you want to “fix” the data, throw it away or migrate all problematic data into unsearchable text fields. Fixing the data sounds like the best solution, but can be very expensive and may be impossible.
Not every system is a failure. The following sections include descriptions of two very successful systems.
The New York State Office of Alcohol and Substance Abuse
Services manages 1500 programs from 700 medical providers who deliver services
to the people of
The application designed to support this organization is complicated, requiring a sophisticated user interface, complex process flows for budgets, claims, and the ability to make periodic modifications to them. One particularly complex part of the system involves the automatic adjustment of claims to support rules governing the ability to automatically reallocate money from one program to another.
Oracle Database
Oracle Application Server
Oracle Forms front-end (converting to Java EE for the web)
Despite the complexity of the system, the system successfully meets all requirements. Designing and building the system took somewhat longer than expected, but this was mainly due to the lack of understanding of the complexity of the system early on. Some of the requirements took quite a long time to gather. The rules governing automatic reallocation of funds within a claim took several months to describe.
Early in the project, the contractors were not well supervised, leading to more schedule slippage than might have been necessary. Also, there was extreme budgetary pressure (due to contract delays) on the project causing it to be understaffed for long periods of time. Currently, there is adequate funding for the project.
Despite a changing process, both the contractors and the state agency are satisfied and the project continues to evolve and additional functionality is being added to the system.
Over time, the amount of supervision given to the project has significantly increased. There are now specific task orders comprising several weeks of work that are approved individually for each mini-increment of the project. Every work item is pre-approved and added to a task order prior to starting work.
The following numbers show the budget allocation for this system over a three-year period as well as a projected budget for the current year.
2004 $200,000
2005 $500,000
2006 $350,000
2007 $600,000 (projected)
The US Air Force Reserve Recruiting Command supports the enlistment of 10,000 people every year. This is accomplished by 400 recruiters stationed on military bases and in offices around the world.
The system developed supports recruiters processing applications from the time of their initial contact through to enlisting them in the service. It involves a very complex process flow with thousands of business rules. The system is an automated version of the recruiters traditional, manual, paper-form based system. The recruiters fill out forms on screen as their main user interface mechanism. There are also several external interfaces to support automatic acceptance of leads from various sources as well as submitting customer data for security clearance applications. The system supports a complex user interface and an extensive reporting system.
Oracle Database
Oracle Application server
Client-side Java front end (converting to Java EE for the web)
When this project was taken over by the current contractor it had been declared to be “fully operational” by the previous contractor. Nothing could have been further from the truth. The system was far from functional. Basic operation of the system by a single user caused the system to crash after a short period of time. However, the project was so far behind schedule that the entire system was mandated to be in production in 6-8 weeks. The first year was spent getting the system to provide basic functionality by meeting several milestones. Each milestone was possible to achieve, but the team was stretched to the limit to create each deliverable quickly.
Over time, as the system evolved, it moved closer and closer to a well architected system. The need to remain in production while significant functionality was added helped to improve the quality of the architecture.
The system is now fully functional. The current direction is to migrate the system from its client/server roots to a web-based system. Making the system 100% rules-based will ease that transition.
The following numbers show the budget allocation for this system over a three-year period as well as a projected budget for the current year.
2005 $600,000
2006 $1,200,000
2007 $1,200,000 (projected)
Software projects continue to fail at an alarming rate. As we have moved our applications to the web and made them more service oriented, our architectural environment has become much more complex. The web development environment continues to evolve faster than developers can master the new technologies. After over 30 years of building large systems, as an industry, we have failed to decrease our project failure rate. Relational databases, CASE, OO, business rules, Agile development, COTS and open source have not significantly reduced the system failure rate.
Whatever technology is employed, good technical management and common sense have never gone out of style. Those are the only silver bullets that will ever make a project successful.
Projects fail because we make bad decisions. No one is exempt from making bad decisions, so every project is destined to fail... unless we can notice those bad decisions and correct them in a timely manner. The key to project success is intelligent self-monitoring.
|
Term/Acronym |
Definition |
|
Agile |
An approach to software projects where developers produce multiple, iterative releases of the application very quickly. |
|
Business rules |
The rules describing the operations and goals of an organization. Business Rules products employ formal grammars and/or repositories to articulate and describe the business rules which then drive or generate some part or parts of the system. |
|
CASE |
Computer-Aided Software Engineering uses computer software tools to develop, modify and/or enhance new software. CASE tools in the 1980s promised (and largly did not deliver) to make the software development process much more efficient. |
|
COTS |
Commercial-Off-The-Shelf packaged software |
|
Data warehouse |
Electronic storage repository for an organization's data which usually contains historical data as well as information to be accessed for reporting purposes. |
|
DBA |
Database Administrator is a person who is responsible for the environmental aspects of a database including creating database backups, performance analysis, database tuning, and disaster recovery. |
|
ETL |
Extract-Transfer-Load process involves extracting data from outside sources, performing some
transformation of the data and finally loading data, usually into a data
warehouse. |
|
Java EE |
Java Enterprise Edition is the latest name for the Java-based development standards. Previously called J2EE. |