Stephen Smith's Blog

Musings on Machine Learning…

Archive for March 2012

Learning over the Web

with one comment

Now a days, when people need to know something, they turn to Google. When people need to learn something new, like say a new programming language or application, they turn to the Web for on-line training. But what is the best way to learn things from the Internet? Which tools work well? Which tools end up wasting a lot of time? As a blog writer and Software Architect, I spend some time wondering on the best ways to disseminate information. I’ve tried a lot of the items discussed in this posting both as a teacher and as a student. As usual these represent my own personal biases and opinions.

I think there is a lot to be said for attending real physical classroom or attending conferences. Besides the sessions there is all the networking and sharing of experiences with other attendees. Sage Summit is a great one for learning and networking here in North America. But travel is expensive and time consuming. Often attending classes is too much time commitment. So it’s nice that if you can’t turn to these there are still some good alternatives.

PowerPoint is Evil

The heading refers to a well-known Wired article: PowerPoint is Evil. Many consider PowerPoint to be the worst thing that has ever happened to education. Here is the Gettysburg Address in PowerPoint form  to emphasize what is lost in a PPT presentation.  If you are looking to learn a new topic and you Google, chances are you will find many PPT presentations on the topic. Chances are if you download and read these, you will learn very little and become frustrated. I know I’ve annoyed people by sending them PPTs from old conferences in answer to various questions. Many Universities and Colleges make a big deal on how they publish all their class PPTs on the Web. To me these aren’t very useful, and make me wonder at the value of these institutions. Never mind the horror of death by PowerPoint in a meeting or at a conference.

Khan Academy

Khan Academy is a website that is completely free to use and offers course material from all over elementary school, high school and college. There are over 3000 videos and over 300 math exercise that you can use. All the videos are made by the founder Salman Khan, and are interesting because of their simplicity. Basically you get a virtual blackboard that Salman draws on as he explains a topic. Most of the videos are under ten minutes long and using this site is quite addictive. Although I haven’t done it yet, I feel that producing Khan Academy type videos could be a very efficient way of producing quite effective training material. There are other specialized on-line training sites like Code Academy, but I tend to like Khan the best.


There are some excellent videos of training sessions and lectures on the Internet. InfoQ is always posting quite good lectures. The main problem I have with them is that they are hard to skim through. Often to get the benefit you have to watch a full hour long video. I tend to prefer written material since I can skim through it and process it quite a bit quicker. You also have to watch the quality of the video, some are quite un-watchable. In a way videos are great to watch lectures you missed, perhaps at a conference you couldn’t attend. On the other hand I find it really hard to find that un-interrupted hour in a day to watch a complete video lecture.

I’ve only created a couple of videos, I found it very time consuming, mostly because in the editing process you spend so much time watching and repeating parts. Perhaps I need more practice, but I find it can take a full day to create a decent 20 minute video. I wonder if to practice, I should start doing some of my blogs as vlogs?


I’ve attended some really excellent webinars. I especially like them if they are interactive. If there are a small number of attendees then you can ask questions as you go along. If attendees can’t ask questions I find it isn’t nearly as engaging, if there is a large audience, often a helper can find some good questions off the chat window to interject a little interactivity. Generally a good technology to provide a near classroom experience when you can’t physically meet.  With newer Telepresence technologies, I expect webinars to get better and better and to fulfill the vision of virtual classrooms.

Often when people give webinars, they record them and then post them for people that missed the live webinar. Then you get into all the issues I mentioned in the videos section. I often find recorded webinars quite boring in comparison to the live event.


Often you can find a number of free e-books on any given topic. For that matter you could buy a real physical book or buy an e-book version. For much learning, I still enjoy quietly reading a book, whether a physical book or an e-book on my iPad. There are always promises of more interactive books, but I still like the old fashioned passive variety. Being able to mark up and search e-books is definitely nice. I even wonder if I should one day create a book version of all my blog posts?

I like it that companies post all their instruction manuals as PDFs on the web. So once I’ve lost the manual for my phone and need to change the answering machine, I can find the manual much easier on the web than finding the printed one in my house.


I find blogs a good way to disseminate small amounts of information. However I don’t think a blog is a good vehicle for producing a training course. Cumulatively there are a lot of blogs out there and a lot of good information is available in blogs first before it appears in other media. I know I try to push out information in my blog before other departments have a change to process and publish it. So generally I find blogs best for information at the very bleeding edge, often in quite a raw form. I’m not sure if I would get many viewers, if say I ran a ten part series on say learning SData.


Around the office we have a joke that Wikipedia knows everything and this is pretty much true. If you need quick info on a topic, then Wikipedia or other Wiki’s can be a great source. We do all our technical documentation in Wiki format now, so we can quickly push it from our internal to external Wiki very easily. We find this is a great way to provide technical documentation without any extra overhead. Many companies have Wikis of this nature. These aren’t always the best places to learn from, but they are great for looking things up.

Reference Material

A lot of companies publish all their reference material to the Internet. Often this is in Wiki format as mentioned above. However there are many other tools for generating this. For instance for our Java APIs we insist that all source code has complete JavaDoc, and then we generate this JavaDoc and reference it form the Wiki to provide API reference documentation.


I tend to think the best way to learn, is by doing. The best way to learn is from making mistakes and you need to be doing, to make those mistakes. You can’t learn all the pitfalls of something just by reading or watching. But how do you get started? The web now offers a wonderful variety of resources that are mostly free to get started and to get learning.

Written by smist08

March 31, 2012 at 5:13 pm

Sage Advisor PEP

with 4 comments


Sage Advisor is an umbrella term for a number of technologies and programs that are being rolled into all Sage products over their coming releases. Previously I blogged on the Sage Advisor Update project here. In this blog post I’m going to talk about the Sage Advisor PEP (Product Enhancement Program).  The intent of this program is to actively gather program usage information to help Product Managers and Application Designers better focus their work and to do a better job designing and specifying new features for future versions.

This sort of information gathering is becoming very common in the software industry. Microsoft has a very extensive program that they call their Customer Experience Improvement Program.  Mozilla Firefox has a telemetry program to gather performance data. Cisco has their Smart Call Home functionality. All SaaS applications do this big time. Every SaaS application logs every call to the web server and then can archive and mine this data endlessly. For SaaS applications you don’t have a choice, since you need to talk to the Web server to talk to the application.

It’s important to remember that participation in this Sage program is purely voluntary and easy to opt out of. Further no actual data from your database is ever transmitted. We are also subject to various governmental privacy laws such as HIPAA.

This feature has been around for a while now in one form or another. We introduced the “Call Home” feature in Sage 300 ERP 5.6A. This feature sent back information on which modules a customer had activated. It was a one-time message that was sent a few months after a new version was installed and activated. With version 6.0A we introduced PEP level 1, which sent similar information to Call Home but was sending it to the central Sage collection server rather than a special one only for Sage 300. With the forthcoming Sage 300 ERP 2012 release, we’ll be implementing level 2 which sends more usage data as explained below.

To do detailed user testing, a usability lab gives the best results, but this is quite expensive and time consuming for customers. The hope here is to virtualize some of this process and get a lot of useful data without all the manual work.


The goal of this project is to provide better information to our Product Managers, Usability Analysts, Business Analysts and other developers on how real users use our products. We need to know where users are spending their time, where they are productive and where they aren’t productive. We need to know which parts of the program are working well and which parts are causing problems.

Basically we want to guide our design and efforts based on data and not opinion. This is one of the methods we are using to gather real customer usage data.

Data we are Gathering

One of the things we want to determine is where users spend their time, so we are gathering data on what screens the user starts and how long they are in that screen. From this we can get hard data on which are the really heavily used screens and then spend more effort on improving these screens. Generally we know some screens users spend a lot of time in, like Order Entry, but we are looking for surprises here. Further we can see what combinations of screens people run, so if they always run A/R Customers at the same time as O/E Orders, then we can infer there is information in this screen required by everyone doing Order Entry and that to improve the workflow we should make this information more readily available in Order Entry. Generally this is a matter of simplifying workflows and making our customers more productive.

We want to simplify the parts of the program where users are having difficulty. To do this we are recording the usage of the Help. Basically recording all links to the help, this way we can determine the parts of the program that people are finding difficult and having to consult the help. Then we can work on the associated forms to make them more intuitive, so the user doesn’t need the help anymore.

Along the same lines we are recording all error messages displayed. This is to see if we can change the workflow, so the user doesn’t get errors. Also if we can pro-actively avoid error situations we hope to avoid a lot of support calls. For instance if after installing many people get a certain error that indicates things aren’t setup correctly, can we modify our installation program so people won’t run into this?

Big Data

Many Sage corporate presentations start with a slide proclaiming we have 6.3 million customers. This is great, but now with Sage Advisor PEP that means we now have 6.3 million customers sending usage data to a corporate web server and all this data needs to be recorded and analyzed.

This starts to put us into the world of “Big Data”. I blogged about Big Data and ERP here. Currently we are gathering all the data into SQL Server, but this is already strained with only a few Sage products contributing. We are already moving the data from the SQL Server to a NoSQL database to perform data analysis. As the volume of data continues to grow we will probably need to replace the SQL Server with something more scalable and this is a classic use case for a NoSQL database. To me this is an exciting initiative to use and become familiar with Big Data technology. As Sage moves forward this will become a more and more important technology to gain expertise in.

We do take care that we won’t delay people using their business application to send usage data. We always start a new thread or program to transfer the data so we don’t block the main program for the user as it upload data. Also we don’t consider this data “crucial” so we don’t need to worry too much if some is lost because the system is too busy.


Gathering usage data is becoming more and more common in the software industry. Sage is stepping up our efforts to gather good usage data from all our products. The primary goal of this is to feed this back into the organization to improve our products and processes. To become more scientific in the ways that we improve our products.

Written by smist08

March 25, 2012 at 12:53 am

TPAC 2012

with 2 comments

TPAC (Third Party Advantage Conference) is a conference for Sage 300 ERP third party developers, that is held yearly in Vancouver, Canada. This year it was held at the River Rock Casino Resort. The idea is that this gives a venue for all the Sage 300 ERP ISVs (Independent Software Vendors) to showcase their wares to Sage 300 ERP Business Partners. The conference started with a quick introduction and then an exhibitor flash, where each exhibitor received ten minutes of fame to give a quick introduction to their products. Then there was a trade show in the River Rock Show Theatre where attendees could gain more detailed information on the various solutions. There is plenty of time for networking and education. If you arrived a day early, there was pre-conference training and an organized ski trip to Whistler. This year there were 36 exhibitors. This show is run independently of Sage, though Sage is a sponsor.

Sage’s big North American conference is Sage Summit, which will be in Nashville in August this year. Summit highlights all of Sage’s many products, and although the ISVs have booths and presentations, the main focus of the show is Sage, its direction and announcements. TPAC is nice because it is a more intimate affair with only Sage 300 ERP and Sage CRM partners and ISVs, many who have known each other since the Accpac Plus days. The focus in on the ISVs and they have the spotlight. This makes it a very focused and valuable conference for everyone that attends.

The main development center for Sage 300 ERP is in Richmond, B.C., only a 10 minute drive away from the conference. This meant that many Sage Product Managers and Developers could attend either the whole show, or at least attend for part. We also held a developer open house the following day at the Sage office that any ISV was welcome to attend. This included attending the Sprint Demo for a Sage 300 ERP 6.1A Agile Sprint that happened to fall on the same day, detailed technical information on topics like SData and lots of discussion and Q&A.

Although Sage isn’t the focus of the show, Sage people give a few presentations including a Sage 300 ERP roadmap presentation to highlight the road ahead. This was also an opportunity for partners to meet Sage’s new EVP of Mid-Market Solutions Joe Langner and the new SVP of Product Management and Marketing Doug LaBahn. There was also a very spirited town hall session led by Joe Langner where partners could ask any questions or raise any concerns.

ISVs are of crucial importance to any ERP software vendor. No vendor can do everything on their own. The key goal of each vendor is to provide a compelling software development platform along with a core set of applications developed on that platform.  This is a huge undertaking. Then the ERP vendor relies on ISVs to take the platform and develop all sorts of vertical applications for specific industries, useful add-on programs or integrations to other stand-alone applications. For Sage 300 ERP, the System Manager is the development platform that provides the frameworks, structure and supporting APIs for developing ERP modules. The System Manager is the runtime component and the Sage 300 ERP SDK contains all the samples, documentation and tools to develop ERP modules to run on SM. Then Sage 300 ERP provides the core accounting applications including General Ledger, Accounts Payable, Accounts Receivable, Inventory Control, Order Entry, Purchase Order, Payroll, and Project and Job Costing. Then we rely on ISVs to fill in other industry specific modules like POS (Point of Sale), Manufacturing, Service Management, etc. This is one of the main reasons Sage invests so heavily in technologies like SData that enhance the ability for ISVs to integrate to all Sage products.

To provide some concrete examples of Sage 300 ERP ISVs, here are the ones that exhibited at TPAC. This isn’t an endorsement of any particular vendor; it just means they exhibited at TPAC. This isn’t meant to exclude anyone, if an ISV isn’t on the list it’s only because they didn’t exhibit at TPAC or I made a mistake. Many of the ISVs below have more products than I list, these are just a sampling.


Written by smist08

March 17, 2012 at 7:19 pm

Sage Advisor Update

with 9 comments


There is a lot of debate on the relative benefits of installing software on-premise versus using a cloud or SaaS based product. Generally for SaaS based software you don’t need to manage backups or infrastructure, that is all done for you and you don’t need to perform software updates, they just continually happen in the background.  On-premise software then gives you more control of your environment, allows greater customization and you don’t need to worry about your data being in someone else’s hands.  The team at Software Advice wrote a good article on the benefits of on-premise software here.

One of the advantages of SaaS, is not having to install updates, just having them appear magically, ready for your use. On-premise software can do this as well via auto-updates. For instance the Google Chrome browser will silently update itself whenever it notices a new version is available and usually you never notice that it has updated itself. Other products let you set options on how they update or prompt you whenever an update is available, letting you choose at that point what you want to do it.

Sage is looking to add automatic software updates (similar to Windows Update) to all our products called “Sage Advisor Update”. A couple of Sage products like the Nonprofit Solutions already include this. Most other products will be rolling this out with their next release over the next year. When a Sage product with this capability installs, it will add a new Sage Advisor Update program to your Start Menu. Think of this like the Apple Software Update program. Below is a screen shot with some labels of this program:

This will give you a list of all the updates available for all your Sage products installed on this computer. From this program you can see what is available and then download the update and optionally also install the update. The Sage Advisor Update is aware of client/server type installations as well as security requirements for installation (like being an Administrator).

Starting Slowly

Sage has a large set of business applications, based on all sorts of technologies and using all sorts of customization schemes. Running auto-update on an ERP which has source code customizations will remove all those customizations. Installing a product update that includes Crystal Reports runs the risk of over-writing customized reports. Perhaps all the workstations need an update to match a server update and this needs to all be done at once rather than at the whim of a single workstation user. Plus most large business applications are integrated to all sorts of third party products and compatibility with all these needs to be checked.

These are all the sort of pain points that cause upgrades to be historically expensive. Ultimately we would like to be able to silently and frictionless install updates without any user intervention. But we will start slow. For the initial releases, Sage Advisor Update will notify you of updates and download them for you. However it won’t take the next step of automatically installing them. This then allows you to contact your business partner to confirm whether whatever needs installing is safe for you given the customizations, third party products or integrations you might be running. Additionally Sage Update Advisor is fully configurable so you can tell it what you want done or not done.

Perhaps it will be a while before we can perform a major version upgrade silently behind the scenes. However perhaps we can automatically install some critical hotfixes that we know won’t break customizations. Perhaps we can quickly reach the point of quietly installing product updates behind the scenes. Part of this is discipline on Sage’s part to be very careful in product updates that we aren’t doing anything that will break customizations.


All Sage products tend to have upgrade-safe customization and upgrade-unsafe customizations. Upgrade safe customizations will not be broken by Product Updates and new versions of the software. The other will (might) be broken (or will break the new version). For instance for Sage 300 if you customize a Crystal Report in the directory where its installed, then it is not upgrade safe since it will be over-written by a new version. However if you place that report in a customization directory then it is upgrade-safe since it won’t be over-written. Similarly for Sage 100, if you use the customizer tool, then you are upgrade-safe, if you modify the source code, then you aren’t.

As we move forwards we want to make all customizations upgrade-safe. This way new versions can be installed at a far lower cost than they are today. Ultimately then customizations won’t be an issue to upgrading and won’t be an impediment to the Sage Advisor Update program.

Frictionless Upgrades

Ultimately we would like to make the whole upgrade process frictionless and remove the current large cost in upgrading customer’s business applications. Sage Advisor Upgrade provides a tool that can notify you when upgrades are available, download them and possibly install them. The next step is ensuring that our product updates don’t cause problems with customizations. Either by investing more in upgrade-safe customization technology or by having the product update itself update customizations to work with the new update.

Another aspect of this is to make sure any new features are easily and naturally accessible as well as very easy to learn. To make upgrades truly frictionless, they shouldn’t require a large training effort to get everyone to effectively use them. Other aspects of Sage Advisor technology has the ability to watch how you are using the software and offer training tips to take better advantage of what you have. A very large percentage of new feature requests we receive are for features that are already in the product, so we need to do a better job of helping people make use of these.

Downsides to Avoid

That’s not to say auto-update can’t be annoying. When I returned form African Insights 2012, I booted up my home computer and then it proceeded to install Windows Updates, Apple Updates, Flash Updates, Acrobat Updates, etc., etc. Basically it took over an hour to boot my computer hand have it usable. Perhaps I shouldn’t have clicked the buttons to let all these updates proceed, but I like to stay up to date. Especially annoying are updates that require a computer re-boot to complete.

In Africa, several people I met never run Windows Update. This is because they access the Internet over the cell phone network and have very low download limits. If they let Windows Update run then it uses up their download limit and Internet access becomes quite expensive.

Of course a worse case is having some favorite program break, so you can’t use it until the next update or you have to manually un-install it and then manually install an older version. This has happened to me with Google Chrome a couple of times. Worse of course is when Windows Update messes up and you are stuck with re-installing Windows.

We need to keep all these things in mind when we deploy updates, that we aren’t just putting a lot of noise in front of customers. That we are respectful of people’s time and the fact that they need to get work done with their computer. That they may have restrictions like bandwidth limits and will need ways to work around these. We especially have to increase our product quality so that our automatic updates don’t break things and don’t cause problems. A certain number of current upgrade difficulties is due to Sage pushing upgrade tasks to Business Partners rather than automating the process; now, Sage needs to spend the time and make the investments to automate all these tasks.


I just updated my iPad and iPhone to iOS 5.1 and this was a fairly painless procedure. It took the device a little time to download and install the update, but it did it all without intervention from myself. It will be nice when upgrading large multi-user business applications is just as easy.

Written by smist08

March 10, 2012 at 6:22 pm

Defining SData Feeds for Sage 300 ERP

with 8 comments


We introduced SData support with Sage ERP Accpac 6.0A; however, the product as it shipped only defined a few SData feeds that it needed to support the new Web Portal, Data Snapshots, Inquiry Tool and Quotes to Orders features. But Sage 300’s support for SData is based on converting SData Web Service requests into View calls. So it is possible to expose any View (or collection of composed Views) as an SData feed.

In a future version of Sage 300 ERP we will expose all the relevant Views via SData, but in the meantime if you want to use SData with Sage 300, then you need to provide XML files to define the feeds you need.

All the feed definitions are XML files, which means you can read all the existing ones that come with Sage 300 using a normal text editor. Hence you can use the existing ones either as examples or templates for the definitions you need.

One thing to be careful of is that most of the fields in these XML files are case sensitive. This means they must match exactly or they won’t work. When things don’t work, it’s worth looking in the Tomcat\log folder at the relevant SDataServlet.log as this will often point out errors when parsing the XML files.

Class Map Files

The classmap files are a series of XML files located in the sub-folders under: C:\Program Files (x86)\Common Files\Sage\Sage Accpac\Tomcat6\portal\sageERP. The feeds for a given application are stored under the application’s program id and version id directory such as oe60a. Note that these need to be in a folder for an activated application to be read, but within an application you can define feeds that access Views in any application.

All the configuration XML files are loaded into memory by the SDataServlet on startup. So if make any changes to these files, you need to restart the “Sage Accpac Tomcat6” service for your changes to take effect.

You can use these to define custom Java classes to process the SData requests, I’ve covered this a bit in other blog postings, but won’t go into that here, since this article is only considering what can be done by editing XML files.

The classmap defines each SData feed and specifies the class to handle the feed and then a detailed feed definition file called a resourceMap file.

Example – Currency Codes

The currencyCodes resource is implemented by the Java class: ViewResourceKind and is defined by the resource map file: currencyCodeViewMapping.xml. ViewResourceKind is a system provided Java Class for generically converting SData requests into View calls. You can use this to expose most Views (that have data) as SData feeds.


<contract name=”accpac”>
<resource name=”currencyCodes” className=”com.sage.orion.sdata.servlet.accpac.ViewResourceKind>
<parameter name=”ResourceMapFile“ value=”currencyCodeViewMapping.xml”/>

If you aren’t programming server classes then this is all you need to know about the classmap files.

Resource Map File

Maps an SData resource to a backing family of Accpac Views and fields. These are stored in the resourceMap folder under the folder that holds the classmap file.

By default all fields from the view are exposed as SData Resource Elements.

Has the ability to exclude or include or overwrite Sage 300 fields from the SData resource.

Example – Currency Codes

The currencyCodeViewMapping.xml resource map file contains the following:

<resource name=”currencycode” description=”Currency Codes”>

<resourceViewField viewFieldName=”CURID” />
<resourceViewField viewFieldName=”CURNAME” name=”Description”/>
<resourceViewField viewFieldName=”DECIMALS” />
<resourceViewField viewFieldName=”SYMBOLPOS” />
<resourceViewField viewFieldName=”THOUSSEP” />
<resourceViewField viewFieldName=”DECSEP” />
<resourceViewField viewFieldName=”NEGDISP” />
<resourceViewField viewFieldName=”SYMBOL” />

The key points are the viewID that maps this feed to the currency codes view CS0003. The URI of the feed is the plural name, namely currencyCodes. Then you can specify the list of fields you want included in the feed. You might specify a shorter list of fields to keep the size of the feed to a minimum. The includedFields section is optional.  I tend to prefer using an excludedFields section to just list the fields I don’t want.

Example – Single Level Resource

SData resource “customer” is defined from the view AR0024.

<resource name=”customer” description=”AR Customers”>

Example –  Multi-Level Resource

SData resource ‘arInvoiceBatch’ is defined from a set of composed views – AR0031, AR0032, AR0033, AR0041 and AR0034.

<resource name=”arInvoiceBatch” description=”AR Batches”>
<resource name=”invoice” description=”AR Invoice”>
<resource name=”detail” description=”AR Invoice Details”>
<resource name=”optional” description=”AR Invoice Detail Optional Fields”>
<resource name=”schedule” description=”AR Invoice Payment Schedules”>

Resource Map File Details

Resource Mapping File Attributes / Elements:

name: name of the resource

description: description of the resource.

viewID: The ViewID of the resource. Remember you can get further information on the Views from the Sage 300 ERP Application Object Model (AOM).

pluralName: plural name of the resource.  If undefined then the pluralName = name +”s”.  This will be the URI of the resource.

resources: Collection of sub-resource elements

kind: resource kind name of the resource. Top resource of the resource tree the resource name and the resource kind name should be the same. However for a sub-resource the resource name reflects the name of the property that refers to it in the parent while the kind name is the name that it appears as at the top level of the schema.

includedFields: list of resourceViewFields that are to be included in the resource (by default all fields are included)

excludedFields: list of resourceViewFields that are not to be included in the resource

overridenFields: list of resourceViewFields that are to be overriden in the resource (usually done to change the SData name)

virtualFields: list of resourceViewFields that are to be added to the resource. Note: virtual fields requires extending ViewResourceKind with a custom class that implements these virtual fields.

lookupFields: These are a 6.1A feature that allow you to add fields looked up from another view like to get a description.


Hopefully this article gives an idea of how to setup additional SData feeds for Sage 300 ERP, without requiring any programming.

Written by smist08

March 3, 2012 at 11:57 pm