Time to Kill the Relational Model

It’s time to kill the relational database model. Well not the relational model per se, but rather the practice of building systems by developing relational schemas first.

This is quite the revelation for me. When first I discovered relational database design over three decades ago I thought it was religion. The simplicity of tables and relationships represented freedom from the rigid hierarchical structures of the day. Since then, relational design and its’ normal form remained a constant foundation for the systems I constructed.

But during a recent project I tired of the burden that relational design placed on our development effort. In particular, a tight adherence to rules of normalization and relational philosophy stifled our opportunity for nimble software engineering. What on paper looked like a method of enforcing integrity became complex morass of surrogate keys and multiple joins.

Fundamentally my complaint is over the complexity that normalization brings; an odd turn considering that relational databases were popularized on simplicity of tables. Frankly, though, there is nothing intuitive about joining tables; and the more joins the more complexity.

While relational databases are not (and should not) go away, there are flaws with some relational theory that makes writing software difficult. For example…

• Normalization produces a lot of tables. A lot of tables translate into a lot of joins. A lot of joins is a red-flag for complexity. When queries frequently have three or more joins, the schema is probably overly complex.
• Relational theorists discourage use of null foreign keys. The only way to accomplish this goal is to introduce a link table for the one to many relationships in the model. This introduces two problems, first, the extra table adds an additional (and unnecessary join). And second, it is harder to determine if a row in one table is related to a row in the other.
• Using data to determine state (or status) introduces complexity. While not necessarily a tenant of relational models, some designers prefer to compute the state of an item on-the-fly based on data in the tables. This is easy enough when all the data needed to determine the state are stored on the same table row. Unfortunately it is frequently necessary to scan additional tables on multiple rows to calculate the status.

A better method of system design starts with a design of objects. The database schema, then, is modeled to keep these objects intact. In this way, the database becomes nothing more than the repository for data at rest. The validity of the data is managed by methods within the objects and not by referential integrity or database triggers. Think of this as object first, data last design; there OFDL, I just coined a new acronym.

Why Digital Talent Doesn’t Want To Work At Your Company | Fast Company

I like this blog post so much that I am re-posting it in it's entirety.


Why Digital Talent Doesn’t Want To Work At Your Company | Fast Company:

Why doesn't digital talent want to work at your company? It’s not because you’re a consumer packaged goods company, rather than Google. It’s not because you’re in Ohio instead of Silicon Valley. It’s not because your salaries are too low, or because you don’t offer free food and laundry services.

It’s because you’re not providing them the right opportunity. The talent you want would be happy to work in an un-air-conditioned garage in New Mexico if it meant the chance to change the world.

This, the opportunity to do great things, to make a real difference, is what drives most digital talent--whether they’re developers, designers, producers, marketers or business folks. 

Most companies don’t offer this, so they skip your company and work somewhere that’s more innovative and exciting. End of story. But the good news is that you can offer them something exciting and great. The promise of changing a giant, behind-the-times organization into an Internet-savvy business is an incredibly exciting challenge and a big way for ambitious people to make an impact.  

But it takes more than lip service to make the sale. Job candidates and new hires with digital chops must truly believe in the company’s dedication to digital transformation and they must see that they are empowered to make this change. Trouble is, many big businesses aren’t structured to deliver on this type of opportunity. The attributes of a soul-crushing, Sisyphean, anti-digital workplace run deep.  

Digital talent won’t want to work at your company if:  

• Every element of their work will be pored over by multiple layers of bureaucracy. Even if that’s how the rest of the company operates, it can’t spill into the digital department. In a technology environment, new products and businesses spring up daily and a new endeavor can go from conception to launch in a matter of months. Reining in the momentum will be read as inaction and a clear signal the company isn’t willing to grasp the new way of the world.  
• Mediocre is good enough. While clocking out at 5 p.m. is attractive to some, it will discourage digital talent. They want to be expected to do something great. They want to be pushed. They care about their work. Their leadership, and those they rely on to get things done, must match their appetite for success.   
• Trial and error is condemned. The freedom to try out new ideas allows employees to take initiative, make decisions, and learn from their mistakes. It also demonstrates an attractive and inspiring entrepreneurial spirit.
• Your company is structured so it takes a lifetime to get to the top, and as such there are no digital experts in company-wide leadership positions.Digital talent--often in their 20s and 30s--need to see a clear path for uninhibited career development that’s based on merit, not years spent, and that’s beyond the confines of the digital department. If they don’t, they won’t see a reason to stay with the company in the long term.  
• Your offices are cold, impersonal and downright stodgy. It may sound like it conflicts with the “you don’t need to be in Silicon Valley point,” but appreciate the nuance. A traditional office layout is designed to communicate power among certain individuals and barriers between departments. This does not support the collaborative ethos which is intrinsic to the web. Companies should do everything possible to provide the digital team friendlier, open office space. A location in a hip, young neighborhood (which surely exists in every mid- to large-sized city) is also a big plus.  

When all of these digital-talent deterring points are addressed, company leadership has effectively and proactively demonstrated the company’s dedication to a digital transformation. It is at this time that their words, a broadly communicated firm stance on the significance of the company’s digital goals, will make the most impact. Without this conspicuous top-down support, politics in the organization or simply one influential disbeliever can hinder the effort, limit the extent of digital integration possible, and discourage valuable employees.  

You need them more than they need you. Demand for their services is so high, they can afford to be finicky. If they don’t like where they’re working, another firm with a more attractive culture and more grand opportunity will quickly swipe them up. That could be your company. But it could just as easily be someone else.

Adapted from Users Not Customers: Who Really Determines the Success of Your Business(Portfolio), by Aaron Shapiro, CEO of HUGE, a digital agency that helps companies including PespiCo, Comcast, Target, HBO, and Unilever reimagine how they interact with their customers and manage their business in the online economy. Visit aaronshapiro.com.

'via Blog this'

Don’t think of BI as an all-in-one solution

I recently had a short conversation on Business Intelligence with one of my peers. I tried to explain the premise that a Business Intelligence application in our industry (Health Care) should not be a one-size-fits all solution. Instead the technology should be tailored to the types of questions that it will need to answer most frequently.

When he claimed "of course it has to be able to answer any question, otherwise we could just write queries," I realized I failed to make my point. He is not the first person I've met to have this opinion. In fact, the opinion is pretty pervasive among my peers; and it is wrong.

Our Business Intelligence solution is a textbook case highlighting this point. Its' saga is a story for another day, but in our attempt to make it very flexible we failed to make it strong. That is, there's no limit to the reports you can create, but it's not great at answering any particular question.

I liken this to the difference between a hammer and a Swiss Army Knife. A hammer is great at driving nails, better than any other tool for this task. It also happens to be pretty good at removing nails too. A Swiss Army knife can do a lot of things from clipping nails to opening cans. But it's not particularly good at any of them.

The real beauty of a hammer, though, is the other things that it can do pretty well. In fact, if put to the test, it isn't hard to come up with at least as many tasks as can be done with a hammer as with the Swiss Army Knife. It can be a door-stop, a paper-weight, a meat tenderizer, a garden shovel, and more. Sure, it's lousy at tightening screws, but it can really drive nails.

Don't get me wrong. There's a place for firms that build all-in-one software. In fact, my former employer Information Builders is one such firm. My current BI solution is built on a MicroStrategy platform, another Swiss Army Knife vendor. Vertical solutions, like our health care application, need to be targeted; they need to be really good at particular questions.

Unfortunately, many designers of Business Intelligence solutions try to make Swiss Army Knives when they really need hammers. And given a good hammer and an innovative user, there will soon be many other tasks suitable for the tool.

Engaging the entrepreneurial power of employees with a venture capital model

The entrepreneur in us sees opportunities everywhere we look, but many people see only problems everywhere they look. The entrepreneur in us is more concerned with discriminating between opportunities than he or she is with failing to see the opportunities - Michael Gerber, author, entrepreneur

This year I participated, in an innovative project as we prepared for review by an internal leadership team. As of this writing, the project is not approved, but the financials are very strong and the technology is very low risk. By the time the project is reviewed by the board later this year, my firm will have spent nearly 18 months, hundreds of man-hours, and tens of thousands of dollars. Yet the work product produced is merely a document and an idea. There is no code and no proof of concept.

The procedure is at best a difficult process; at its’ worst an impossible process. There should be little doubt that the net effect of the process is to inhibit growth through innovation. It is, in fact, a barrier to entry for teams with go-to-market ideas.

Industry is full of companies with high profile examples of innovation. Firms like Google, Pixar, GE, and Apple show that a culture of innovation can bring success. This paper describes how any firm can evolve into a venture driven company that empowers employees with an entrepreneurial spirit.

The goal is plain and simple: achieve explosive growth through a culture of innovation.

To foster this culture, I suggest a model based on capital markets for startup companies. There are three aspects to this proposal:

• The corporation acts as a Hedge Fund by allocating funds for investment and measuring  aggregate results;
• Senior executives function as Venture Capitalists by soliciting ideas and managing a portfolio of investments; and
• Employees become the Entrepreneurs championing their ideas and finding resources to execute a project.

 

The critical ingredient is getting off your butt and doing something. It's as simple as that. A lot of people have ideas, but there are few who decide to do something about them now. Not tomorrow. Not next week. But today. The true entrepreneur is a doer, not a dreamer - Nolan Bushnell, founder of Atari and Chuck E. Cheese's

The Hedge Fund

A hedge fund is a private investment fund whose managers can make speculative investment and participate in profits from money invested. The company can function much like a hedge fund by structuring projects as a portfolio of investments. The managers of the “hedge fund” are the executives who normally vet within for the company.

The corporation, then, decides how much to invest in the fund. Each year these funds are set aside for innovation and it becomes the responsibility of the hedge fund managers to assure the funds are invested, and that the portfolio achieves an appropriate return.

As the Hedge Fund, the company would:

• Invest in the fund by allocating monies for innovation. An investment to the innovation fund each year;
• Appoint  executives to function as the Venture Capitalists;
• Set an ROI threshold for the fund

Taking Risk

I have not failed. I've just found 10,000 ways that won't work - Thomas Edison, inventor and scientist

A great percentage of start-ups fail. Hedge funds that specialize in innovation spread their investments over a wide range of projects with the understanding that a small number of winners will earn a return for the entire portfolio. Risky projects will be an integral part of the firm’s portfolio.

Like true VC hedge funds, the firm can mitigate risk by funding projects that already have some track record of success. This track record could be a successful competitor, a custom built technology, or a joint venture with a client.

Exit Strategies

Unlike the holdings in venture capital hedge funds, the company’s investments generally do not end with a liquidity event (sale of the company or IPO).Instead successful investments will result in product lines or business that generate revenue and profit. Less successful projects are absorbed into existing businesses or closed outright.

It should be noted that there is the possibility of a liquidity event. Although these may be rare, some projects could be spun off as independent companies. In such cases the firm would act as a private equity fund, holding a stake in the company until an appropriate liquidity event occurs.

Venture Capitalists

Since one fails often, address markets that make it worthwhile when one does succeed. Vinod Khosla, co-founder of Sun Microsystems

Executives that normally vet for capital investment will serve as the venture capitalists in this model. As VC, each executive is expected to oversee a portfolio of projects and to assure that their funds are invested. Each VC would have a specialty where they would focus their investments. It would be expected, though, that they would invest outside their area on occasion.

The VC has four primary roles:

• Invest a share of the fund in projects of their choosing;
• Provide oversight of the projects in their portfolio;
• Mentor the entrepreneurs assigned to the projects in their portfolio; and
• Arrange exit strategies for their projects.

Key to the success of the hedge fund is the oversight and mentoring role of the VC. Unlike an “approve-and-forget” capital investment model, VCs of the hedge fund share responsibility for the success of their portfolio. This oversight keeps entrepreneurs accountable to accomplish milestones and meet their financial goals.

VCs solicit, review, and select projects based on the business plan from the entrepreneur. The VCs is empowered and required to invest his share of the fund.  VCs will be expected to accomplish the appropriate due diligence, but being empowered will make the decision process move quickly. With empowerment comes accountability, and the VC will be expected to make rational decisions based on standard venture criteria such as, experience of management, amount of existing business, size of the market, first-in status, and the competitive landscape.

Like true VCs, the managers in the firm’s hedge fund will collaborate on some projects. Some of the VCs will assume an “angle investor” role by making small investments in a large set of projects in their early stages. As angle projects progress, the VC may decide to invest additional rounds of financing. Also, several VCs may invest together in a project, thereby sharing the risk, or implementing a large scale innovation.

VCs shall participate is the financial success of their portfolio. A bonus would be paid based on the success of the projects in the portfolio.

I never perfected an invention that I did not think about in terms of the service it might give others... I find out what the world needs, then I proceed to invent -Thomas Edison, inventor and scientist

Intrapreneurs

Some people dream of great accomplishments, while others stay awake and do them – Anonymous

Entrepreneur : one who organizes, manages, and assumes the risks of a business or enterprise.

Intrapreneur : one who organizes, manages, and assumes the risks of an idea within a business or enterprise.

The entrepreneurs are the employees that champion an idea (intrapreneur is a term made up for this white paper). Entrepreneurs are passionate about an idea, and work to see it through to its completion. This includes building prototypes, writing business plans, seeking funding from VCs, and managing the project day-to-day.

Much of the motivation and reward of these employees is intrinsic and non-financial. This includes the excitement of working on a new and innovative idea. In addition, once funding is approved, the employee will have effectively transferred into a new position with the company. Like the VC, however, the entrepreneur will be rewarded based on the financial success of the project or its exit strategy.

The entrepreneur has the following roles:

• Develop a business plan;
• Seek and obtain funding from the company VCs;
• Build prototypes and proof-of-concepts for the project;
• Obtain resources needed to complete the project;
• Manage the project, day-to-day; and
• Report progress and financial results to the VC

 

It should be expected that projects are present by a team of employees. In these cases, one member of the team shall be designated as the entrepreneur. It is likely that the entrepreneurs are employees that already hold management or leadership positions (but that are not a requirement).

Mitigating Personal Risk

Entrepreneurs assume a great deal of personal risk with regard to the security of their career. Although positions cannot be held open for the entrepreneur pending completion of their project, the corporation should guarantee that a position will be made available regardless of the success or failure of the project. Truly motivated entrepreneurs, however, do not consider failure, and will pursue the idea under the assumption of success.

The Business Plan

Entrepreneurs must submit a business plan to obtain funding for their projects. Business plans should follow common standards for start-up plans. At a minimum, a business plan shall include:

• Description of the Idea;
• Marketing Plan;
• List of existing customers
• List of target customers
• Size of the market
• Pricing
• Competitive Analysis
• List of competitors
• Advantages and Disadvantages
• Operational Plan
• Management Team
• Personnel
• Technology
• Financial Plan
• Start-up Expenses
• 3 Year Pro-Forma

 

Advantages of the Hedge Fund Model

This approach to funding invention, and building a culture of innovation has distinct advantages over my company’s current model.

• The current procedure in of itself is an expensive and labor intense process. It requires multiple stages of presentation and approval. The current process is self-regulating, scaring away employees that are intimidated by the process regardless of the quality of their idea.
• An innovation provides a method of easily submitting an idea, but except for quick rejections, it routes ideas through the same burdensome approval process.
• The hedge fund model is simple; write a business plan and present it to a VC.
• Our current procedure is slow.  It will be very difficult to have first-in status if seeking funding through the current process. Real-time Fraud and Abuse, for example, has been in-process for months.
• The hedge fund model is fast.
• Where our current procedure is focused on preparation and planning during the approval process, hedge fund is focused on execution of the idea.
• Hedge fund rewards entrepreneurs and VCs for taking risk and accomplishing goals.
• Hedge fund provides a tangible portfolio of innovation and R&D.
• Hedge fund is based on tried and true practices of bring an idea to market.

 

Twenty years from now you will be more disappointed by the things that you didn't do than by the ones you did do. So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. Explore. Dream. Discover -Mark Twain, author

eWeek looks at Computing on the Cloud

My paper copy of eWeek is generally fodder for the circular file. In fact, the magazine rarely makes it as far as my office. Instead it stays in our reception area on a coffee table along with other unread magazines. This week, though, I picked up a couple of editions to browse while taking lunch. In the process I discovered some interesting cloud computing technologies.

The first is old news for leading edge developers, but new to me. The June 30 edition ran an analysis on the Google App Engine. The Google App Engine competes with Amazon Web Services in scope and intent. Although Google currently only supports the Python language, betting an application on Google infrastructure seems pretty safe.

In a similar vein, GigaSpaces offers an application server that sits on Amazon's EC2 cloud computing solution. The GigaSpaces application server provides a middleware layer between Java or .Net applications and the Amazon Web Services backend.

A third product, Jungle Disk, has a different goal, but uses computing on the cloud none-the-less. Jungle Disk is a backup and storage solution that works with Amazon's S3 Web Service. I have not tried Jungle Disk, but eWeek offers a fairly thorough analysis that puts the product on my to-do list.

Of course the interesting common thread with all these applications is computing on the cloud. GigaSpaces and Jungle Disk take advantage of Amazon's early entry into the utility computing space. Google, however, does everything well, and should have little trouble catching up. Old timers like myself remember the days of time-sharing on mainframes. Cloud computing proves what's old is new again with one important difference; now utility computing is affordable.

Cash is King

There is a common notion that most start-ups fail for one of two reasons; either they are under capitalized or do not sell aggressively. In both these cases, the firm fails to raise the appropriate amount of cash needed to sustain operations. Of course all failed businesses run out of money, but some never give themselves a chance.

A firm that never gave itself was chance is Elite Technology Partners. The company was born from the technical brain trust of Blackwood Trading, and quickly conceived a product and business model based on their experience from the other firm. Their product was greatly inspired by Blackwood, but targeted a specific strategy and different distribution channel.

Elite's founders had learned from their experience at Blackwood. Certainly, they put to use the intellectual capital gained at Blackwood. They also learned from Blackwood's failed effort to market themselves successfully. In addition, the founders recognized that Blackwood's operational model was very expensive.

However, Elite underestimated the effort required to develop a product and bring it to market. Very quickly the firm fell into a chicken and egg situation. In Elite's case, they couldn't sell product because they did not have the capital to complete it. And they could not raise cash through sales because their product was not finished. Add to their situation the economic environment during 2002, where investment capital had effectively dried up. Venture capitalists were only looking at firms generating cash through sales.

In Elite's case, the lack of cash led to a series of strategies changes that sealed the fate of the company. First, a shortage of cash caused many of their best engineers to secure positions at other employers. Elite sought an investment arrangement through key prospects that would enable them to bring their product to market. But partnerships with a large investment come with restrictions. Demands of exclusive use and ownership of intellectual property made partnership deals impractical.

To raise cash, Elite sold its' talent in consulting arrangements. The consulting business was profitable and brought in enough cash to keep the operating. Unfortunately the opportunity cost of consulting was a halt to further development of Elite's product. In the end, Elite failed to complete its' product; Elite failed to complete its' product because it failed to raise the capital necessary to build the necessary technology.

I am working with an entrepreneur who is in a similar position. He is following a conscious strategy of delaying raising despite having an established relationship with investment bankers. The entrepreneur is betting that signed contracts will make the firm more attractive to investors. This may be true, but having no cash in his firm, he will have a difficult time meeting any operational commitments.

My observations of Elite Technology Partners leads me to believe that my friend has a risky strategy. Because he refuses to seek capital, he will not have a fully functional team in place when he signs his first contract. When the contract is signed he will need to complete his technology, hire staff, and seek capital within a very short timeframe. In the three to six months needed to receive private equity cash his venture could fail.

Elite Technology Partners failed because they did not raise cash for their operations. Successful entrepreneurs beg, borrow, or steal (Ok, maybe not steal) enough to give their companies inertia. A firm with inertia will attract further investment, or generate cash organically.

Data Warehousing Dilemma

We are in the midst of a long running Business Intelligence project. The original thought behind the iniative was a solution allowing analysts to explore their data. Through a combination of historical data and predictive modelling, the solution would provide key metrics to manage their business. After several mis-starts, we discovered the wisdom of Ralph Kimball.


Kimball's Data Warehouse Toolkit was an ephifany and inspiration. Suddenly before us was the solution to performance woes. His solution? A star schema, where with the warehouse measures are retained in a single table linked with related descriptive data. All the descriptive data (dimensions) were related through the single fact table containing the measures.

Our project started simple enough. Using sample report templates from our Product Management team, we determined a grain for the warehouse and computed the appropriate measures. The grain, by the way, is the lowest level of detail needed to answer the questions asked of the data.

It soon became apparent that there was a flaw in our design. Not the design of the warehouse, per se, but the design of the system. The reports designed by our Product Management team were at the level of the grain. Running them would produce thousands of pages of detail. Nowhere were we taking advantage of the warehouse's dimensions to drill into these reports. Seeing the flaw, we tasked our Product Managers with spec'ing the entry points to their reports.

What was returned to us was a disaster. Instead of taking advantage of the warehouse or even the capabilities of the Business Intelligence technology, the PMs designed more reports. These new reports were summaries with drill paths to the detail provided earlier. They also contained an entirely new set of metrics, all calculated at a different grain.

The problem of the different grain was exocerbated by many of the new metrics. These metrics were computed with division of aggregated counts. The counts, however, were not on the fact table, instead they were distinct counts of dimension values. There is the dilemma, we needed figures that could not be pre-computed into our cubes. The metrics were computed "on-the-fly" and resulted in tremendous performance problems.

I believe the solution is simple and obvious. We need additional fact tables a different grains. The purists among my team didn't see it so clearly (they will by time we're finished). The problem is Kimble's treatse on data warehouses discourages multiple fact tables in the database schema.

Kimble oversimplifyies warehouses with the star schema. Any complex set of data will have measures that can not be summarized into a single fact table. In truth, though, multiple fact tables will be an integral part of any practical solution based on a data warehouse.

How much detail in requirement?

I mentioned that over the course of twenty five plus years, I've tried nearly every development process you can name. In my experience, I find that projects tend to break down at the start. Yep. Failure is built in into the project during the requirements stage.

Requirements are written in varying detail. During my tenure with DoubleClick in 2000, we had initiated a project for the advertiser side of our industry. In my ten months with the firm, the document was never finished. The last time I saw it, it exceeded 300 pages. We appointed a steering committee to oversee the project and they would debate the document endlessly. And the result of the debate? A bigger document.

In short, the entire exercise was useless. In parallel to the requirements fiasco, an engineering team was already well into the project, virtually ignoring the written requirements. I have seen this practice of over doing requirements at several firms, including DoubleClick, Information Builders, and Ameritech. In most cases the resulting document is either too large to be useful, or is ignored by the developers.

Of course the reverse is often true too. In fact, the reverse is probably much more frequent. For example, my team recently received the following requirement for a component of a new product:

"Management reports – using multiple selection criteria, produce reports that track turnaround time and other metrics."

Thing about the number of questions that are opened by this simple one sentence requirement. What reports? How many? What columns, rows, sorting, totals? What selection criteria? What is turnaround time? What other metrics? In fact, this requirement tells a developer nothing. There is absolutely nothing the system architect can do except return to the business analyst and ask questions. To make matters worse, the business analyst believe he has provided a useful document.

Then there are requirements that mask the business need in technical details. Some might read something like:

"Add a column to the x table to hold a flag. Display the flag on the detail page. When transactions are received for records with the flag set to 0, ignore the transaction."

This might look like the appropriate middle ground, between the extreme examples illustrated earlier. But it's not. The business analyst has interpreted the business requirement and supplied a solution. Although many business analysts fancy themselves as system architects, I have yet to meet one who is better than the engineer. What is the requirements above? A method of disabling master records? A method to manually override batch operations?

The plain fact is it hard to write useful requirements. And a project with poor requirements is destined for delays and cost overruns. If you wish to keep your requirements meaningful, stick to the following rules of thumb:

• Describe the business need (not the technical solution)
• Keep it brief (it a single capability takes more than a couple pages then your too verbose)
• Describe everything (if there are four reports, describe each in with their own requirement)

When you avoid the pitfalls from poor requirements, your project will be constructed on a solid foundation. The chances of success are will be far greater.

SDLC

Agile. Waterfall. CMM. Spiral. Test Driven. Blah, blah, blah. I believe I have researched a dozen software development methodology and have tried many of them in practice. Unfortunately I have yet to find one that works. One that combines rapid development with high quality and delivers projects within their targeted timeframes.

I suppose if it was easy, then there wouldn't be entire shelves at Barnes and Noble devoted to it. It it was easy, then everyone would deliver with great success. I'm becoming convinced that no single methodology is portable through development shops. And that successful SDLC (System Development Life Cycle) is a painful trial-and-error process that adopts aspects of several disciplines.

I've tried the standard waterfall. It generally has two problems. First, it is nearly impossible to completely define all the functions of a system prior to providing any code. And second it is very difficult to prevent scope creep.

In the first case, a business analyst has to be imaginative enough to define all aspects of the system. Then she must be able to accurately write these into requirements that are understandable by developers. I've never seen this done well. When overdone, the volumes of requirements become impossible of shift through, when underdone entire aspects of logic are left undocumented.

Waterfall project tend to be very long, running months or even years. This causes the inevitable panic when business analysts realize a pet feature is not included. The panic often results in scope creep, which in turn causes the project to run longer.

We've tried incremental methodologies, such as Agile too. Ok, you say, Agile isn't a methodology, but a class of methodologies. But does anyone really implement strict Extreme or Scrum or EVO?

Regardless, Agile methods have their built-in weakness too. For instance, how do you know when you are done? Or, as in the case with my current projects, resources get diverted into new critical projects, leaving others unfinished.

When it's all said and done, some hybrid method seems to work best. Concrete, but overlapping, phases are necessary for project management. Within each phase, iterative cycles with feedback have great benefit. Who knows, maybe I'll develop my own methodology, and then write a book (that no one will read).

Implementing Agile Development

Sometime late last year I was introduced to the concept of Agile Development. Actually, I had been formulating iterative techniques over the past 10 years. Craig Larman's book simply reinforced my thinking. That is, software is best developed in manageable increments.

The bigger problem is moving a team steeped in serial waterfall methods to interative methods. Some people simply don't get it; they don't get collaboration; they don't get accepting change; and they don't get emphasizing software over documentation. All of which is surprising because most development teams never receive adequate documentation, constantly deal with change, and usually brain storm to solve problems.

For us de-emphasizing documentation shouldn't be so bad, after all, we don't receive good requirements anyway. But strangely enough, there are developers who still believe that one big master documents is necessary for successful projects. These people are wrong. It is wasteful and expensive to attempt to write a complete specification prior to engineering the software.

I saw this first hand a DoubleClick, Inc several years back. The firm hired a team (larger than my current department) dedicated to writing specs. The documents produced by this group were huge, numbering hundreds of pages. And the details were debated ad-nausium, leading to stagnation. What was produced was documents, what wasn't produced was working software.

A greater problem for us is managing multiple projects. Our technologies are not constructed on a common code base. Therefore each project becomes its' own set of increments. Currently we have five projects under development. That's a lot of work for a staff of seven plus four consultants. With all these concurrent projects, managing increments becomes difficult. Afterall, it isn't practical to deliver an increment every week. Or is it?

We seem to do alright with collaboration, but there is room for improvement. It helped to implement daily stand-up, or scrum, meeting. The meetings are short and focus on the goals for the day. We have not attempted to implement strict pair programming, althought there is very frequent teaming on tasks.

I remain a strong proponent of agile and iterative development. Over the next couple of months we will be able to take an objective look at the results of these methods.

Strategies for Performance

Like many growing technology companies, we frequently wrestle with performance and scalability of our offerings. In our case, we're tied pretty tightly to Microsoft; a hold over from a time when the products were built using Access. We've since moved on to Visual Basic, c#, .Net, and SQL Server, but we are not platform agnostic.

We're also taking steps to move off a pure client-server architecture, and to an n-tier architecture delivered through a browser. Typically our applications have a small number of users who submit long running queries. There are two primary stress points for performance, loading the data repository, and running queries (reports).

We are attacking the problem across several fronts. First, we're throwing hardware at it. Second, we are upgrading the OS and database platform. And finally, we are optimizing the applications. Note that we are not considering using a server farm for the application servers. We believe the low number of hits to the web apps make scaling the application server a lower priority.

Throwing hardware at the problem is the easiest and quickest way to scale. In our case, that means moving the application server to a separate box, and purchasing more power. More memory, more speed, and more processors. We all know, however, that this type of solution simply covers up bottlenecks in the application.

We are also stepping up to SQL Server 2005. In the standard edition, which most of our customers deploy, SQL Server 2005 will use 4 CPUs and as much memory as the OS can give it. Some of our customers are CPU and memory bound when using SQL Server 2000. Stepping up to 2005 is a significant boost. SQL Server 2005 also runs on Windows 2003 64 bit.

The 64 bit OS appears, in our sample testing, to give a huge performance boost. Unfortunately, it also gives us problems with some of our applications. Most significantly, we have not successfully deployed .Net 1.1 on the OS. Therefore, all our web applications must be migrated to Visual Studio 2005. We found that moving our web applications from VS 2003 to VS 2005 required some work. We are still trying to work through problems with deployment projects for these applications. Our Visual Basic legacy clients flat out do not run in the Terminal Services environment.

Finally, we are confronting the code in the applications themselves. The products have evolved from a client-server architecture. The software requires an active user that performs key functions synchronously. We will move file operations and reporting to asynchronous classes. This frees up the UI and gives the user a responsive experience. But asynchronous does not make queries run faster. Improving query performance requires reviews of the execution plan, indexes, and index views.

It is tedious work, but it will payoff in greater revenue.

Open Source Software

As a builder and seller of software, you may think I would not be a proponent of Open Source. And you would be wrong. I love Open Source; not for idealogical reasons, but because I can get great applications at little or no cost. Our firm is fairly small, and it's important that my modest development staff remain focused on engineering products. For internal requirements, we often turn to Open Source.

One such application is the Blogging software blojsom. And no, this blog is not written with blojsom, I use regular ole Blogger here. Blojsom was easy to set up and meets our needs for corporate blogging; which is actually a journal style knowledge base. Blojsom isn't perfect, administering a blog is clumsy, as is adding and editing entries. But the application is stable and attractive.

We also use Moodle. Moodle is software for running web based education. It is very mature and feature rich. We have only scratched the surface of the application's capabilities, but I am impressed. Moodle has the advantage of an active community.

Of course underneath these applications is Open Source infrastructure. In our case this includes MySQL, Apache Tomcat, and PHP. I don't believe any of these are up to the task of their commercial counterparts, but for the internal needs of a small business they are perfectly fine. With the exception of PHP, these were very easy to set up on a Windows 2003 Server. PHP was a bit more difficult as we configured it with Microsoft IIS instead of the recommended Apache web server.

Although not open source, we also use the free application Actitime from Actimind. Actitime performs timetracking. Interestingly, it uses a combination of ASP.NET and MySQL. I highly recommend Actitime and have considered using their developers as consultants on some of our development projects.

These are real apps being used by a real company to do business on a daily basis. Our usage of these applications clearly demonstrates the traction made by open source providers. I look forward to expanding our list; anyone know a good open source accounting package? Or how about CRM? ERP?