Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘Data Synchronization

Synchronizing Data with Sage 300

with 3 comments

Introduction

Often there is a need to synchronize data from an external source with Sage 300 data. For instance, with Sage CRM we want to keep the list of customers synchronized between the two systems. This way if you enter a new customer in one, it gets automatically added to the other. Similarly, if you update say an address in one, then it is updated in the other. Along the way there have been various techniques to accomplish this. In this blog post I’ll cover how this has been done in the past and present a new idea on how to do this now.

In the past there have been a number of constraints, such as supporting multiple databases like Pervasive.SQL, SQL Server, IBM DB2 and Oracle. But today Sage 300 only supports SQL Server. Similarly, some suggested approaches would be quite expensive to implement inside Sage 300. Tied closely to synchronization is the desire by some customers for database auditing which we will also touch upon since this is closely related.

Sage CRM Synchronization

The first two-way data synchronization we did was the original Sage 300 to Sage CRM integration. The original integration used sub-classed views to capture changes to the data and to then use the Sage CRM API to make the matching change to Sage CRM. Sage CRM then did something similar and would write the change back to Sage 300 via one of its APIs.

The main problem with this integration technique is that its fairly brittle. You can configure the integration to either fail, warn or ignore errors in the other system. If you select error, then both systems need to be running in order for any one to use either. So if Sage CRM is offline, then so is Sage 300. If you select warn or ignore then the record will be updated in one system and not the other. This will put the databases out of sync and a manual full re-sync will need to be performed.

For the most part this system works pretty well, but isn’t ideal due to the trade off of either requiring both systems always be up, or having to run manual re-syncs every now and then. The integration is now built into the Sage 300 business logic; so sub-classed Views are no longer used.

The Sage Data Cloud

The intent of the Sage data cloud was to synchronize data with the cloud but not to require the on premise accounting system be always online. As a consequence, it couldn’t use the same approach as the original Sage CRM integration. In the meantime, Sage CRM added support for vector clock synchronization via SData. The problem with SData synchronization was that it was too expensive to retrofit that into all the accounting packages that needed to work with the Sage Data Cloud.

The approach the Sage Data Cloud connector took was to keep a table that matched the accounting data in a separate database. This table just had the key and a checksum so it could tell what changed in the database by scanning it and re-computing the checksums and if they didn’t match then the record had been modified and needed synching.

This approach didn’t require manual re-synchs or require both systems be online to work. However, it was expensive to keep scanning the database looking for changes, so they may not be reflected terribly quickly or would add unnecessary load to the database server.

What Does Synchronization Need?

The question is then what does a modern synchronization algorithm like vector clock sync require to operate efficiently? It requires the ability to ask the system what has changed since it last ran. This query has to be efficient and reliable.

You could do a CSQRY call and select based on the audit stamps that are newer than our last clock tick (sync time). However, the audit stamp isn’t and index and this query will be slow on larger tables. Further it doesn’t easily give you inserted or deleted records.

Another suggested approach would be to implement database auditing on the Sage 300 application’s tables. Then you get a database audit feature and if done right, you can use this to query for changed records and then base a synchronization algorithm on it. However, this has never been done since it’s a fairly large job and the ROI was never deemed worthwhile.

Another approach that is specific to SQL Server would be to query the database transaction logs. These will tell you what happened in the database. This has a couple of problems namely the queries on the transaction logs aren’t oriented around since the last sync, and so are either slow or return too much information. Further SQL Server manages these logs fairly aggressively so if your synchronization app was offline for too long, SQL Server would recycle the logs and the data wouldn’t be available anymore. Plus, this would force everyone to manage logs, rather than just have them truncated on checkpoint.

SQL Server 2008 to the Rescue

Fortunately, SQL Sever 2008 added some nice change tracking/auditing functionality that does what we need. And fortunately Sage 300 only supports SQL Server so we can use this functionality. There is a very good article on MSDN about this and how it applies to synchronization here. Basically the SQL Server team recognized that both data synchronization and auditing were important and quite time consuming to add at the application level.

Using this functionality is quite simple, you need to turn on change tracking for the database, then you need to turn on change tracking for each table you want to track changes for.

trackchanges

Then there is a SQL function that you can select from to get the data. For instance, I updated a couple of records in ARCUS and then inserted a new one and the result is shown.

changetable

This is giving me the minimal information, which is all I require for synchronization since I really only need to know which records to synchronize and then can get the full information form the main database table.

If you want to use this to audit all the changes in your database, then there are more options you can set to give you more complete information on what happened.

Summary

If you are writing an application that needs to synchronize data with Sage 300 (either one way or two way), consider using these features of SQL Server since you can add them externally to Sage 300 without affecting the application.

Similarly, if you are writing a database logging/auditing application you might want to look at what Microsoft has been adding to SQL Server starting with version 2008.

 

Advertisements

Written by smist08

April 18, 2016 at 10:09 pm

On the Sage GCRM Contract

with 5 comments

Sage has standardized on a REST based Web Services protocol called SData. This protocol is documented at http://sdata.sage.com/. As a first step this will standardize the technology that you use to communicate with Sage applications. You will be able to issue SData requests to multiple Sage products such as Accpac, Abra and SageCRM. As opposed to using a different technology to talk to each, such as COM, .Net, Java, Soap, DLL, etc. All the products will still maintain their existing APIs, but they will all add SData to their repertoire. Even if you are a reporting type application, crossing multiple Sage products can be difficult as they tend to use different databases or have very different schemas. SData is a nice way to hide these differences and to provide a single consistent programming interface.

The next step is to standardize the actual data that is exchanged via SData. Towards this end, Sage is releasing a number of “contracts” that can be used to interface to a given application. For instance the Sage Global CRM (GCRM) Contract (http://sdata.sage.com/sdatacust_GCRM.html) is a standard contract supported by all Sage CRM products including Act!, SalesLogix and SageCRM. If someone integrates to one of these products using SData and the GCRM Contract, then it will be very easy to integrate to any of the other CRM products.

The GCRM contract was created by Sage for mostly selfish reasons. Sage owns dozens of ERP packages in dozens of regions around the world. Having each of these create a point to point integration with one of our CRM packages was proving to be a huge amount of development effort. Plus the results weren’t consistent, some of the integrations were better than others. This started as an attempt to standardize what was being done repeatedly over Sage again and again. However as the work progressed we realized there are a lot of other benefits. Not only does this save Sage some development time and money but it will:

  • Provide a standard interface for others to do similar integrations. For instance an ISV could integrate another CRM package to multiple Sage ERP’s by taking advantage of the GCRM contract.
  • ISVs will be able to integrate to multiple Sage applications at once (usually part of a suite of products) using only 1 technology.
  • The various application contracts are well documented at http://sdata.sage.com/sdatacust.html making life a bit easier.
  • Using REST based web services is an efficient way to produce a nice lightly coupled integrations.

Each application will still have a native SData interface where you can get at the full functionality of that product. But these standard contract interfaces will make it far easier for ISV’s to integrate their vertical solutions to multiple Sage products.

Many applications have a component of their data that is shared and needs to be synchronized. For instance much of the data that CRM maintains for its companies is shared by the ERP package in its customer’s information. In most CRM to ERP integrations a large part of the integration is keeping this company/customer data synchronized in the two applications. Similarly for an HR application to ERP integration, typically the employee database tables need to be synchronized in the two applications. SData provides a standardized protocol to allow multiple applications to synchronize data in a uniform manner (http://sdata.sage.com/sdatasync.html).

At the core of the GCRM Contract is the specification of how to synchronize the tradingAccount (company/customer), contact, postalAddress, phoneNumber and email tables. The GCRM contract specifies the fields that make up these standard contract definitions (how CRM and ERP expose their native tables) and what the applications need to do to provide the synchronization. In this case the hard part is handled by the CRM application which hosts the synchronization engine. The ERP package just needs to provide an SData feed for each of these tables in the correct format. Then CRM will periodically ask ERP if anything has changed and if so to provide the updated records. The protocol then has the details of how to handle conflicts and how to catch up if one application was unavailable for a period of time.

The GCRM Contract really consists of three parts:

  1. The base customer/company tables that need to be synchronized and the protocol to handle that.
  2. A number of real time data SData feeds in a standard format to access much of the data in the CRM product.
  3. A set of SData feed definitions that allow CRM to provide a basic Order Entry screen, to query pricing information from the ERP and feed orders into the ERP package.

These three parts can be used independently. For instance Accpac could use the GCRM contract synchronization contract to replace the current method and start incorporating CRM data into its own UIs that is retrieved by the real time SData feeds, but then rather than using the CRM Order Entry screen, surface the Accpac Order Entry screen inside of CRM (the quote-to-order feature: https://smist08.wordpress.com/2009/12/17/sage-erp-accpac-6-0-quote-to-orders/).

As Sage moves ahead with adopting our REST based Web Services, SData, you will be seeing more and more new technologies based on this. Generally Sage is looking to develop many SData based technologies that can be shared across all Sage products and can be utilized by all Sage development partners.

Written by smist08

May 7, 2010 at 8:35 pm