Search:  

Previous pageComponents Reference Next page
Data Import 

This component has been replaced with the more powerful, multi-tabbed Data Import Component.


The External Lookup Data Feed component provides a powerful interface to pull data into a site from XML sources.
The ELD-Feed is used to connect to external database sources,

For detailed documentation on the External Lookup Data Feed see the Data Integration section.

It can handle a wide range of usage scenarios, including:

  • One-off bulk import
  • Regular scheduled imports
  • Ad-hoc imports based on user input or event triggers

It can understand a wide range of XML-based input:

  • XML
  • RSS feeds
  • Atom feeds

The component works by mapping fields in the input to fields in a Table. The input can contain one or more records.

When a record is imported, a record is added to the Table, or, if the cache control fields indicate the record already exists in the Table, the the existing record is updated. The Created and Updated dates on records are refreshed as appropriate when the import affects a Table's records.

The Table can contain extra fields that are not mapped by the import component. These fields will take on their default values, or for existing records, their existing values will be preserved.

Event triggering

When a record in the Table is added or updated, the Events associated with the Record Update will be triggered (allowing emails and tweets to be sent, and further imports to be started)

Datatypes

The component attempts to intelligently work will all datatypes. The XML input is inherently a text format, but within that it can contain text, rich text and date fields.

Images

Images are normally handled by the input providing an URL to an image file.

If an image is mapped to a text field in the Table, the the URL will be imported, however if the image is mapped to an image field, then the image itself will be imported. (If you want both, you can map the image twice)

Note that all images within a Table record share the same namespace (they are stored in the same underlying folder) so must have different names. However each record has a separate namespace, so different records can have images with the same name.

Importing the image into an image field (rather than just taking the URL) will naturally increase storage requirements on your server, however it will remove future reliance on the external URLs remaining available and responsive, and will in any case be necessary if the images are secured in a manner rendering them inaccessible to your users' browsers.

Image links in rich text content are not currently imported as images, but are linked as URLs to the external resources.

Feed URLs

The input is provided by giving the component the URL of the input file. (This is far more powerful and flexible than uploading the file to the server).

Secure feeds

You can optionally specify a username and password to be used if the feed requires the request to use Basic Authentication.

The feed URL can be hosted on either an http or an https URL.

Alternate / fallback URLs

Some providers of feeds may give alternate URLs to be used if there is a server outage or problem with the first URL. To handle this the component allows you to specify a list of URLs to try in sequence until it gets a response. It will only use the response from one of these URLs - as soon as it is has received a response from one URL (even if it is an 'empty' response) it will not query further URLs in the list.

Request Parameters

Simple feeds may contain all the information they need to know what to respond with in the base URL, whilst others will respond with different contents depending on the values of one or more parameters.

These parameters can themselves be categorised as 'fixed' or 'variable', from the perspective of the import component.

  • 'Fixed Request parameters' are parameters which will be the same for each lookup. For example some feeds use fixed parameters to identify you.
  • 'Cached Table Field Mappings' are variable parameters, and can change with each lookup, often determined by user input.

When to lookup

Processing a feed request can, depending on scale and server loading at either end, take time, so for efficiency reasons you may not want to perform a lookup on every request. This is especially important if the feed provider charges you for each lookup, in which case you will want to avoid making lookups if you already have the information to hand in a Table.

To control this you can set the import to either lookup 'always', or to lookup only when there are 'no matching records in the table'. To determine this you can specify one or more fields in the Table to be used for 'cache control'. When a user search is made triggering a lookup to be performed, it first inspects the Table to see if there are any records where all the fields marked for cache control are a match. If there is one or more such records, the lookup is bypassed, and results are returned from the Table. If there wasn't a match, then the lookup is performed, which if successful will add records to the Table, and results are then returned from the Table.

Cache control applies only to variable parameters, and thus does not apply to one-off or scheduled lookups, which only use fixed parameters.

Input validation

Some feed services require parameters to be presented in a particular way, for example only within a certain string length range. For each variable parameter you can specify the minimum and maximum length. If the lookup would violate these limits, the lookup stage is bypassed, and results are returned from the Table (which would probably mean no results are returned to the user).

Cascading multiple lookups

In complex scenarios the feed provider may require you to make multiple feed requests to obtain all the information you need. You can chain lookups together by setting a Table's Event tree to contain a Lookup action. That way, when the first lookup adds or updates a record in its Table, that triggers one or more further lookups. These lookups can either use the default request URLs as specified in their configurations, or can take a request URL specified by a field in the Table , which means that one feed can specify the URLs for subsequent lookups.