Introduction

neatComponents is a comprehensive website development platform and management system. neatComponents is all about hiding the technical aspects of web-based systems away from the user, and indeed, away from the system designer, in order to make the process efficient and profitable. However, whilst it's not strictly necessary, it's sometimes interesting to know what's going on under the hood.

In this document we'll take a quick overview of the underlying architecture, and some of the reasoning behind the architectural principles. Hopefully this will also help you understand the neatComponents 'mindset'.

As we explore the system we'll see that there's a distinct evolution – a timeline – in the development of the system. It was definitely not all designed in a single moment, and subsequently coded up. Certainly the basic structure was designed first, and has stood the test of time, but most of the detail that you see today was designed in countless subsequent iterations layering on top of the original structure. The features in neatComponents tend to be the ones you'd hope would be there, as our development priorities have been directed by our clients, not by our whims.

Some guiding principles

When we started designing the system we laid down some guiding principles to help us evaluate the merits of subsequent choices.

The most obvious ones were that we wanted to make website design easier (ie less technical) and quicker (to be more profitable). We defined this to mean that:

• No coding should be needed
• No FTPing should be needed
• No direct access to the server should be needed

Let's focus on those for a moment. 'No coding'. That means that the system needs to provide, through some sort of graphical interface, options and settings that allow all the usual business logic found in websites to be configured, without the use of Java or VB or any other coding or scripting language by the designer. 'No FTPing' and 'No direct access' go together; as they serve to both save the designer from technical aspects, and equally importantly, protect the server from inadvertent damage. As an example of how this affects the functionality of the system, it meant that we had to build in much of the admin interface found in the Web server (IIS in this case), and present this in a controlled fashion to the site designers.

Another guiding principle that underlines the depth of the system is that the self-imposed limitations that the above principles impose shouldn't impede the designer in their quest to create unique and distinctive sites. So the system can't simply present a few 'templates' designed outside of the system, and force the designer to choose from them; rather it must provide a rich graphical layer that can be styled and controlled.

All that is focussed on the designer creating individual sites, but we wanted to widen the scope somewhat to leverage some extra cost savings for the designer (or their employer). So we included a principle of 'zero-administration', meaning that any task on the business administration side that could be automated should be included. In practise this includes two areas: The first is client billing – where the system manages a credit balance along with low balance warning emails, and interfaces with bank credit-card systems to handle payments. The second is client sign-up – not appropriate in all situations, but useful where a provider is offering templated sites that the designer has created, and you need the client to be able to sign up, maybe get a 30 day free trial, and then be moved on to the client billing system.

It's taken as read in the above that the system is server-based, rather than sitting on individual designers' machines, but that of course was a considered decision. Whilst it's true that some popular design tools are based on designers' machines, and then upload finished pages to a web server, they have difficulty when it comes to anything but the most trivial of content management. That approach also seems to create a harsh distinction between site designers and site users, whereas in reality a large system with fine-grained delegation may have thousands of users with permission to make changes. There's no way you'd want the expense of individual developer licenses for each of them, let alone the administrative burden of keeping them all up to date. In fact, when you start opening data manipulation up to clients, either in an extranet scenario, or even in an e-commerce system where you invite feedback on products, the distinction between a designer and a visitor becomes blurred, and its best to equip everyone with the tools to make any change. And if the only tool that's needed is a web browser, it starts to become cost-effective and scalable.

The net result is a system which, to be fair, is not going to be at all popular with quite a few people! Hard-core programmers for one, as they won't be needed for this, nor accounts assistants working on the sales ledger. However, neatComponents isn't aimed at them, but at two different audiences: graphic web designers, who want to create sites that don't just look good, but also include technical functionality; and directors of web design companies who want to grow without taking on extra staff, and thus become (more) profitable.

Platforms, languages and databases

Before we wrote any code some really fundamental decisions were needed. We knew our core principles, but now we needed to consider implementation.

The first choice was whether to go down the open-source route, or follow the established 'closed-source' route. Open-source has some advantages, but we were extremely nervous about the stability of a solution created from a melange of other systems, and whilst open source does mean, in theory, that the underlying code of all the parts is accessible and editable, in practise modifying someone else's code is often harder than just writing your own.

So, with a couple of exceptions, we decided to code everything ourselves. The exceptions being the database layer, for which we initially used MS Access (but not for long) before migrating to an architecture supporting MySQL and PostgreSQL. The other exception is the Text Editor.

On the subject of platforms, we decided to initially support the Microsoft IIS web server, running on a Windows Server, however we have been careful to keep our options open so that we can port to Linux or NetBSD relatively straightforwardly.

Of course, not all code is at the server: modern browsers enable AJAX principles to be used to deliver a smooth and interactive design environment, and we have embraced these technologies wherever possible, using XML, background HTTP transfers and plenty of DOM manipulation.

Web pages – and how we make them

Whilst neatComponents does many things besides, the basic fabric is a content management system (CMS), by which we mean a system where the content that appears on pages is stored in a database, can be modified by suitably authorised people, and is then displayed to users.

The architectural choice we faced here was whether to get the CMS to write HTML files to disk in public facing web-space, and then allow visitors to browse that: updating the pages as needs be, or whether we should create the pages on-the-fly each and every time the user visits the page.

From a performance standpoint it seems a no-contest. Pages are viewed far more often than they are changed, and it takes longer to construct the pages from the database than to simply deliver them from pre-constructed files on disk. So why did we do the opposite, and create a system where the pages are constructed each and every time? The answer is 'users'.

The static file approach is fine if you have just two sets of users: admins, who can change things, and visitors, who see them. However, if you have another user-group, say 'clients', then you need two sets of the pages, as the same page may look different depending on who you are logged in as. In fact, since an individual can be a member of more than one user-group, if you have two groups, they could be in one of three states (one group, the other, or both). And if you have more groups, the problem gets worse. Exponentially. Before long, the permutations mean the hard disk space required gets seriously out of hand. With just ten user groups that means 3,628,800 (10 x 9 x 8 x ...) different copies of the site would be needed to be stored - and updated whenever a change is made.

Of course building a page from the database each time a copy of it is requested still leaves us with the performance issue to solve, so, using a lot of profiling we determined which database queries and page construction functions were taking the most time, and implemented a multi-level caching system. Thus the first time a page is requested after a data change it might take a bit longer than normal, but subsequent requests will be swifter as intermediate results can be obtained 'for free'. As responsiveness is always important, the system monitors how long it takes to construct pages, and logs any particularly slow ones for further analysis and optimisation.

Web pages – and how we name them

What do you call a web page? It sounds like the start of a bad joke, but it's actually a serious issue which pits the desire for human-readable words (www.domain.com/january-sale.html) against the practical need to keep everything linking together.

We chose to use a numerical naming scheme (www.domain.com/27). The number is system-generated for each page, and can't be changed. The fact you can't change it turns out to be really important, as it means that you never get broken links from external sites outside of your control. Of course, if you want a user to be able to type in www.domain.com/january-sale.html and get taken to page 27, that's still possible, using DirectURLs, a mechanism which allows a list of aliases to be stored against each page, and then if a user attempts to visit them they get redirected to the correct page. In fact, DirectURLs go even further than that, as they can be aliased not just against pages, but against individual table records (say a particular product in an e-commerce scenario), so asking to visit that DirectURL will bring up the catalogue page right at that product.

The biggest objection we found to using the numerical addressing schema was that search engines would not take kindly to indexing them, but in practise we've found that that seems not to be an issue, so long as the pages are titled correctly, use meta tags, and follow all the other standard best practice.

Storing data – the interesting bit

Earlier we described how the basic fabric of neatComponents is a content management system, where individual pages are edited online, and the changes are stored in a database. Five years ago that might have sounded radical, but today that's pretty much standard fare. What sets neatComponents apart is that that's not our goal; it's just our starting point. Where it gets interesting is working with structured data. A product catalogue, or a list of employee profiles. Scenarios where you would traditionally reach for a programmer familiar with a relational database (Oracle, etc), and integrate their back-end systems with a web front-end.

Looking back at our guiding principles: no code, no access to the machine, that's clearly not going to be easy. However, we created the ability to add special pages that encapsulate the essence of various relational database concepts. (Since these pages are going to do far more that just display text, we'll call them 'Components' instead of pages).

So where a database is made up of Tables, we provide a component (called a Table) that allows you to define a record structure, and fill in forms to populate those records.

Where a database has Queries that allow you to retrieve grids of records matching certain criteria, so do we (and this time we call them Queries, too!). These queries can then be dropped on to pages to display structured information in context, formatted appropriately.

There's also a series of components that provide specialised views of the data, and ways to query it: query-by-example forms, search engines, category views, etc, and numerous tricks to make building websites that knit all this together into a swift, precise and even fun experience.

And in case you were wondering, it was this collection of neat components that lent the system its name – neatComponents.

Behind the scenes, the system maps all the data manipulation on to the underlying database, creating tables and fields on the fly, and protecting the user and the site designer from any of the technical complexity that normally happens whenever databases are mentioned.

Wrapping it up

To make the system easy to get started with, we've wrapped the entire system, database install and all, into a single-file installer, to make the process of converting a vanilla Windows Server into a functioning neatComponents server that's capable of handling hundreds of sites, into a simple five minute task.

Further information

This document wasn't intended to be a complete description of what neatComponents does:  there's plenty of information on the neatcomponents.com website that details the individual functions, and of course, much like a programming language, there are plenty of applications that it can be turned to that we haven't even thought of. However, hopefully you'll have gained some insight into the rationale behind the system, and can see how it may be of use in your organisation.