Migrating historical data between historians

11 December 2012

Organisations that wish to change the data Historians use are often faced with the challenge of what to do with the data contained in the existing historian. Simply powering the old historian server off is usually not a viable option, and maintaining two parallel systems introduces additional complexity and cost. Migrating the data from the existing system to the new system is the preferred route, but when large quantities of data are involved over extended periods, this is not as straightforward as it may appear.

Historians often include tools such as an OLEDB client that can be used to import data from other OLEDB-compliant sources. These tools are usually fairly limited, single query-based affairs that return and insert data in large lumps and tend to be slow given the quantity of data queried.

Also, the data is inserted blindly into the new historian, without providing any way of validating the inserted data, bar manually eye-balling trends of the inserted data on the target historian and comparing those with the source. This cannot possibly be considered a thorough and reliable migration process for valuable historical data!

Because we have encountered the need for a verifiable data migration process between historians on more than one occasion, when we developed the IDX Historian, our cost-effective enterprise-grade historian, we decided to incorporate a Data Migration tool that addresses this need directly into the Historian’s tool set.

The tool does not require any additional license to use and allows for the migration of data between essentially any historian with an OLEDB interface or software API and the IDX Historian. However, the tool also allows data to be migrated directly between 3rd party historians; for example, it is possible to migrate data from a Wonderware (InSQL) Historian directly to an OSIsoft PI Historian.

The migration tool uses the following methods and items to effectively migrate historical data:

  • Assistance from IDX 8 Tag Manager:
    The first step in migrating data often involves ensuring the tags for the data to be migrated already exist or are set up in the new historian. IDX Tag Manager allows the tag configurations to be automatically synchronised between systems before the migration takes place thus averting having to perform this process manually.
  • Source/Target Plugins:
    It is possible to migrate from any supported source historian to any supported target historian, which currently includes the Wonderware (InSQL) Historian, OSIsoft Historian and IDX Historian. Plugins can be developed to meet other requirements on request.
  • Jobs and Batches:
    The migration tool carves up the data under migration into various levels of repeatable work items. The first step is to define a job which runs as a Windows service that encompasses a configured set of tags over a specified period. Once a job is configured, the migration tool automatically creates a set number of work batches to be executed that equally divides the job data to be migrated. The two principle advantages of using the job/batch approach are:
    • Parallelisation: today’s computer hardware benefits from parallel operations. This speeds up the migration process considerably.

    • Recoverability: because the migration process is divided into batches, any batch can be resumed, re-executed or analysed as required.
    • Data Analysis and Reporting: Post migration, the migration tool provides the option of analysing and comparing the data from the source and target historians at the batch level, allowing for greatly simplified side-by-side data comparison. It can also be configured to automatically highlight anomalies in the migrated batches that require further investigation. The provided analysis reporting can be used to provide an overview of the result of the complete migration.

Migration job summary page illustrating the pre-creation of job batches and batch status.

Using these features, the migration tool, as illustrated by this case study, can perform data migration where vendor-provided tools are slow, or simply fail, and provides a fully analysed and verified migration process in a reasonable period.

Because we understand historian users may not wish to delve into the world of historical data migration themselves or purchase IDX for a once-off migration project, we also perform data migration using the IDX Data Migration tool as a service, should this be more suitable.

<a href="https://blog.idx.co.za/author/alex/" target="_self">Alex Thalmann</a>

Alex Thalmann

Development Manager

Alex is passionate about researching and solving problems, and designing and architecting technical solutions, especially those that involve both hardware and software.

Show your love!

Recent Posts

Driving Productivity with LineView Training

In light of our partnership with LineView Solutions, our team has taken up the opportunity to equip themselves with insights into the LineView system and its impact on the industry. “LineView is a production line monitoring and recording system that automatically...

Exciting New Launch: Onsite ProfiDay

IDX is excited to announce the launch of our onsite ProfiDay offering. This is available to all industries and companies that utilise PROFIBUS and/or PROFINET within their industrial communications systems. What is ProfiDAY? An exclusive onsite program offering...

Introducing our Partnership with LineView Solutions

We are excited to announce our partnership with LineView Solutions, a leading UK-based company specialising in smart factory solutions for optimising production processes. LineView developed a cutting-edge software suite designed to automatically gather data from the...

Optimise PROFIBUS and PROFINET Network Performance

As a field technician handling PROFIBUS and PROFINET systems, it is crucial to recognise the common errors that can result in costly downtime. From improper network configuration to neglecting regular maintenance, these mistakes can greatly affect system performance...

Delighted Customer Shares Experience

We recently assisted a client in rectifying damage to the Anybus ABC4090 Communicator caused by incorrect power usage onsite. We arranged for the device to be sent to HMS in Sweden for repairs, and they restored it before promptly returning it to us. Here are some...

Price Decrease on PROFIBUS Connectors

PROFIBUS connectors play a crucial role in industrial automation and data exchange systems by facilitating smooth communication between devices and networks. We are pleased to announce a significant price decrease on all our PROFIBUS connectors. Thanks to recent...

PROFIBUS Standards: Functional bonding and shielding

Electromagnetic Interference (EMI) and earthing issues can significantly impact the performance of your network. By following these six proven recommendations, you can ensure that your network is always up and running.

PROFIBUS Standards: Terminations

What is the Volts Direct Current (VDC) required for optimal operation of a PROFIBUS DP termination circuit? Get the answers you need in our latest video on PROFIBUS Standards: Terminations. Don't miss out on this essential knowledge to boost your system's performance!

PROFIBUS Standards: Segment Cable Lengths

Finding the perfect balance between PROFIBUS cable length and network speed is crucial. While longer cable segments might seem tempting, they can lead to signal degradation and slower speeds. Conversely, too short segments might limit your layout options. Striking the...

Sign up for our newsletter

Get exclusive updates and a sneak peek into our world every two months. Stay connected, stay informed.

Related Posts

Our expertise lies in establishing, maintaining, and leveraging plant data for business benefit. Through our innovative solutions, we work alongside you to streamline processes, enhance efficiency, and reduce costly downtime.


Main: +27 (11) 548 9960
Sales: +27 (11) 548 9970


Copyright @2024 Industrial Data Xchange. All rights reserved.