Stephen Smith's Blog

Musings on Machine Learning…

Archive for January 2014

Branching by Feature

with 8 comments

Introduction

Now that we’ve released Sage 300 Online as well as the on-premise Sage 300 ERP 2014, we need to change the way we develop features going forwards. We would like to develop features and frequently deploy these to Sage 300 Online so that customers can take advantage of these as soon as possible. Plus we would like to start including more features in our Product Updates.

This means that we have two separate products that would like to release features out of the same source code on their own timelines.

This article describes some of the issues with doing this and describes some aspects of the procedures we are following. In some cases there are additional benefits and in some cases there is additional work.

Operating SaaSy

Generally we are switching from releasing big bang product versions every year or so, to releasing updates on a much frequent basis. I’ve blogged about this before in these articles: SaaSifying Sage and SaaSifying Accpac. This article is really just covering one aspect of this process, namely around how we now use our source control system. This is one aspect of a continuous deployment pipeline. It also involves the DepOps team.

Doneness Criteria

A key ingredient into making this all work is having good “Doneness Criteria”, so that when agile stories are completed they are fully done, including all testing, automation, documentation, etc. So that when a “done” story is included in the release it really works. The closer you can come to this the better. If you aren’t very close then you are going to need a lot of integration/regression testing steps between these minor small feature releases to ensure the quality of the product. An important aspect of this is to have good automated tests to find integration problems and reduce manual regression testing.

Source Code Branching

Source code control systems have the concept of branching, where you can create a branch for a feature and then go through the agile process developing the feature and when you are all done then you merge the branch back into the product. This is a very simple branching strategy which works great if you are developing one feature at a time. However in reality there is a lot of work going on simultaneously including multiple features being developed at once, sustainment work (bug fixing) and infrastructure work (like the 64 Bit version).

Modern source control systems like Subversion and Git provide very powerful features to easily create branches and to incorporate changes from other branches and finally merge branches back together. We’ve used Subversion for some time now and have a lot of source code and history stored here. But for new projects we’ve been creating them in Git, partly due to the more powerful branching features.

To work effectively there are usually quite a few branches, but hopefully not as many as on the fractal tree below.

fractal-tree2

So what sort of branches do we need? Potentially we could create a lot of branches in the Source Code. The main root of all the branches is called the trunk and we always want the trunk to be in a releasable state. Perhaps we can have a development branch, and a regression branch. Then from the development branch we fork off the individual features as separate branches. Then when we are happy with them we merge them into the regression branch for more testing and when this is competed they are merged into the trunk.

Generally this works well for one product, say a web product where the DevOps team controls what gets merged from the regression branch into the trunk. However when you have multiple products derived from the same source tree this can be quite complicated.

Another danger of too much branching is that the longer a branch lives, the further it can get from the trunk due to work done on other branches and then merging back into a main branch and the trunk gets more difficult. This can be alleviated by merging changes done elsewhere into your branch, so you have smaller merges along the way rather than a bit difficult merge at the end. Another advantage we have in Sage 300, is that it’s a very large product consisting of hundreds of DLLs, OCXs and EXEs. As a result different teams are often operating in quite different areas of the source code. However there will still be points of contention, one that is happening right now is multiple features being added to the reporting engine that is creating contention on that source code module.

So for a branching strategy we need to strike a compromise between keeping everything perfectly isolated and having a lot of gates for features to pass through versus keeping the management of the system down and reducing the difficulty in merging branches.

So our approach is to limit the number of branches. Trunk is the main branch that is released to production (i.e. customers). Nothing is developed directly on the trunk, everything needs to be developed on a feature branch. Before merging back into trunk, a feature must merge the trunk back into the feature first and the team developing the feature must do some testing (including running our full automated testing suite). Then the feature can be merged back into the trunk and the feature is available for either the Online or on-premise products to consume.

Consequences

Of course this has consequences on other components in the build pipeline. Now rather than just build the trunk with our set of build servers, we have to build all these branches. As part of the build we run a number of unit tests, but we also have a large set of automated tests that run on a separate set of servers, and now we want to run these automated tests on the branches.

Generally this increases the logistics of maintaining all these sets of servers. The build servers need to be configured on what branch to run and then the output of that build needs to be fed into the automated test servers to run all the tests.

Generally this all improves the quality and keeps the trunk in a releasable state since a lot of testing has been done before the feature is merged into the trunk.

For newer components like the Sage 300 Online home page and the Sage 300 ERP provisioning engine, these are built entirely using ASP.Net and then built and deployed using TeamCity. This simplifies the whole process and we can use a more sophisticated branching strategy in conjunction with DevOps where they have a release branch which only they control where they can have complete control on what they merge, build and deploy.

Summary

We’ve been putting this infrastructure in place for some time now and have it operating fairly smoothly. Now we will really start putting it into practice by releasing features frequently on two product lines. The main test that we’ve done it right is that it’s seamless to customers because we’ve been able to maintain high quality with all these processes and technologies in place.

Update 2014/10/04: This scheme has given us some problems. For an update on what we are doing as a result check out this article on the problems with branching by feature.

Moving on to Unicode

with 7 comments

Introduction

My first computer was an Apple II Plus, which didn’t even support lower case characters. Everything was upper case. To do word processing you used special characters to change case. Now we expect our computer to not just handle upper and lower case characters, but accented characters, special symbols, all the Asian language characters, all the Arabic characters and everything else.

In the beginning there was ASCII which allowed computers to encode the alphabet, numbers and the common typewriter characters, all 127 of them. Then we added another 127 characters for accented characters. But there were quite a few different accented characters so we had a standard first 127 characters and then various options for the upper 127 characters. This allowed us to handle most European languages on computers. Then there was the desire to support Chinese characters which number in the tens of thousands. So the idea came along to represent these as two bytes or 16 bits. This worked well, but it still only supported one language at a time and often ran out of characters. In developing this there were quite a few standards and quite a bit of incompatibility of moving files containing these between computers systems. But generally the first 127 characters were the original ASCII characters and then the rest depended on the code page you chose.

unicode

To try to bring some order to this mess and make the whole problem easier, Unicode was invented. The idea here was to have one character set that contained all the characters from all the languages in the world. Sounds like a good idea, but of course Computer Scientists underestimated the problem. They assumed this would be at most 64K characters and that they could use 2 bytes to represent each character. Like the 640K memory barrier, this turned out to be quite a bad idea. In fact there are now about 110,000 Unicode characters and the number is growing.

unicode1

Unicode specifies all the characters, but it allows for different encodings. These days the two most common are UTF8 and UTF16. Both of these have pros and cons. Microsoft chose UTF16 for all their systems. Since I work with Sage 300 and since we are trying to solve this on Windows that is what we will discuss in this article. To convert Sage 300 to Unicode using UTF8 would probably have been easier since UTF8 was designed to give better compatibility with ASCII, but we live in a Windows UTF16 world where we want to interact well with SQL Server and the Windows API.

Microsoft adopted UTF16 because they felt it would be easier, since basically each string became twice as long since every character was represented by 2 bytes. Hence memory doubled everywhere and it was simple to convert. This was fine except that 2 bytes doesn’t hold every Unicode character anymore, so some characters actually take 2 16-byte slots. But generally you can mostly predict the number of characters in a given amount of memory. It also lends itself better to just using array operations rather than having to go through strings with next/previous operations.

Compatibility

Windows took the approach that to maintain compatibility they would offer two APIs, one for ANSI and one for Unicode. So any Windows API call that takes a string as a parameter will have two versions, one ending A (for ANSI) and one ending in W (for Wide). Then in Windows.h if you compile with UNICODE defined then it uses the W version, else it uses the A version. This certainly adds a lot of pollution to the Windows API. But they maintained compatibility with all pre-existing programs. This was all put in place as part of Win32 (since recompiling was necessary).

For Sage 300 we’ve resisted going all in on Unicode, because we don’t want to double the size of our API and maintain that for all time, and if we do release a Unicode version then it will break every third party add-in and customization out there. We have the additional challenge that Unicode doesn’t work very well in VB6.

But with our 64 Bit version, we are not supporting VB6 (which will never be 64Bit) and all third parties have to make changes for 64 Bit anyway, so why not take advantage of this and introduce Unicode at the same time? This would make the move to 64 Bit more work, but hopefully will be worth it.

Why Switch to Unicode

Converting a large C/C++ application to Unicode is a lot of work. Why go to the effort? Sage 300 has had a traditional and simplified Chinese versions for a long time. What benefits does Unicode give us over the current double byte system we support?

One is that in double byte, only one character set can be installed on Windows at a time. This means for our online version we need separate servers to host the Chinese version. With Unicode we can support all languages from one set of servers, we don’t need separate sets of servers for each language group. This makes managing the online server farm much easier and much more uniform for upgrading and such. Besides our online offerings, we have had customers complain that when running Terminal Server they need separate ones for various branch offices in different parts of the world using different languages.

Another advantage that now we can support mixtures of script, so users can enter Thai in one field, Arabic in another and Chinese in another. Perhaps a bit esoteric, but it could have uses for optional fields where there are different ones for different locales.

Another problem we tend to have is with sort orders in all these different incompatible multi-byte character systems. With Unicode this becomes much more uniform (although there are still multiple of these) and much easier to deal with. Right now we avoid the problem by limiting key fields to upper case alphanumeric. But perhaps down the road with Unicode we can relax this.

A bit advantage is ease of setup. Getting the current multi-byte systems working requires some care in setting up the Windows server that often challenges people and causes problems. With Unicode, things are already setup correctly so this is much less of a problem.

Converting Sage 300

SQL Server already supports Unicode. Any UI technology newer than VB6 will also support Unicode. So that leaves our Business Logic layer, database driver and supporting DLLs. These are all written in C/C++ and so have to be converted to Unicode.

We still need to maintain our 32-Bit non-Unicode version and we don’t want two sets of source code, so we want to do this in such a way that we can compile the code either way and it will work correctly.

At the lower levels we have to use Microsoft’s tchar.h file which provides defines that will compile one way when _UNICODE is defined and another when it isn’t. This is similar to how Windows.h works for the Windows runtime, only it does it for the C runtime. For C++ you need a little extra for the string class, but we can handle that in plustype.h.

One annoying thing is that to specify a Unicode string in C, you do l”abc”, and with the macro in tchar.h, you change it to _T(“abc”). Changing all the strings in the system this way is certainly a real pain. Especially since 99.99% of these will never contain a non-ASCII character because they are for debugging or logging. If Microsoft had adopted UTF8 this wouldn’t have been necessary since the ASCII characters are the same, but with UTF16 this, to me is the big downside. But then it’s pretty mechanical work and a lot of it can be automated.

At higher levels of Sage 300, we rely more on the types defined in plustype.h and tend to use routines form a4wapi.dll rather than using the C runtime directly. This is good, since we can change these places to compile for either and hide a lot of the details from the application programmer. The other is that we only need to convert the parts of the system that deal with the database and the parts that deal with string handling (like error messages).

One question that comes up is what will be the length on fields in the database? Right now if it’s 60 characters then its 60 bytes. Under this method of converting the application the field will be 60 UTF16 characters for 120 bytes. (This is true if you don’t use the special characters that require 4 bytes, but most characters are in the standard 64K block).

Summary

Moving to both 64 Bits and Unicode is quite an exciting prospect. It will open up the doors to all sorts of advanced features, and really move our application ahead in a major way. It will revitalize the C/C++ code base and allow some quite powerful capabilities.

As a usual disclaimer, this article is about some research and proof of concept work we are doing and doesn’t represent a commitment as to which future version or edition this will surface in.

 

Data Entry Through the Ages

with 4 comments

Introduction

Last weekend I visited my parents in Victoria and my mom mentioned that she had finally used up all the computer punch cards I had left her when I graduated U-Vic. She likes them because they are more solid than paper but lighter than cardboard and are ideal for using as shopping lists and such. To have lasted this long shows how many cards I needed to do all my first and second year Computer Science courses back then at U-Vic.

This got me to thinking on how entering data into computers has changed over my career. Data entry is changing at an even faster rate these days, so I thought it might be fun to look back and to look forwards as well.

I’m not sure if this makes me appear very old, or shows how slow educational institutions adopt new technology. Not only was I the last first year computer science class to have to use punch cards, but I was also the last year when you weren’t allowed to use calculators in the Provincial exams and had to use a slide rule.

slideruls

Dec LA36

My first experience of computers was in Computer Science 11 at Oak Bay Secondary in Victoria. The school had a Dec LA36 terminal connected via a 300 baud modem to the PDP-11 at Camosun College.

la36

Basically the terminal printed what you typed and sent it to the computer when you hit enter and then would echo anything sent back. Rather primitive. Certainly was different editing files this way. Back then Basic used line numbers and you edit lines by specifying what you wanted done to a specific line by number.

IBM Punched Cards

Then I went to the University of Victoria, which was a step backwards. Rather than a nice online terminal like the LA36, we had to enter data via punched cards and then receive the output later from a managed line printer.

IBM_card_punch_029

You had to be careful what you typed since each run took quite a bit of time and used up money from your account. You got good at using functions like duplicating cards up to a point and were always very careful not to drop them. Given the nature of the medium, it was surprisingly robust, in that the cards were actually pretty reliable.

Video Terminals

Once I hit third year, we were allowed to use video terminals to do our Computer Science work. Some people were lucky enough to use very compact languages like APL to program. Others of us had to manage rather slow editors using cursor keys. Admittedly a huge improvement over the LA36 or punch cards.

vt100

Personal Computers

For my first Co-op work term, I worked at Island Medical Labs and programmed a Radio Shack TRS-80 computer to do a number of calculations and to print a number of reports for the lab. This was my introduction to personal computing and I was so happy to have a computer all to myself, rather than the time sharing systems I was used to. It had disk drives and a daisy wheel printer. Lots of fun programming in Basic for this.

trs80

After my first co-op work term I used much of my earnings to buy a brand new Apple II+. Since I mostly took Numerical Analysis type CS courses, I was actually able to do quite a few labs off my Apple II+. I had to take a cassette tape downtown to get a print out at a computer store since I didn’t have a printer yet (or a disk drive).

Apple-II

Lisa/Mac

A big innovation came when the Apple Lisa came out and introduced the world to the mouse and the GUI Operating System. This was a huge leap forward. I never owned a Lisa or Mac, but eventually started using Windows, a rather pale copy in those days.

lisamac

Touch

Along the way there were quite a few devices that used touch as an input mechanism. But none of them were popular until the iPhone came along. Like GUIs and the mouse, Apple brought this into the mainstream. I have an iPhone 4s and really love it. Using this device is very easy and once used to it, I don’t miss the keyboard from my previous Blackberry at all.

smokedemo

Voice

Voice input is finally starting to work properly. Tools like Apple’s Siri are actually starting to be useful. I blogged on this previously here. Certainly people are relying on this in their cars to dial phone and to select music. Even to ask Siri trivia questions as you drive along.

scotty

Gesture

Gesture is still fairly controversial. It’s not clear whether Kinect helps or hinders Xbox. People like the concept but are put off by an always on video camera into their living room. We aren’t quite at the level of Minority Report yet, but we are getting there. I’m not sure what this will do to the cube office environment once this goes mainstream.

minreport

VR

Although not really an input device, Virtual Reality and VR Goggles are closely related. In these immersive worlds they combine voice and gesture input with providing an immersive complete visual view. The Oculus Rift was quite popular at CES this year. It will be interesting to see if these can successfully be productized and achieve a mass appeal.

oculus

Glasses

I blogged previously on Google Glasses here. These are fairly controversial. Google is just in the process of releasing these into the mainstream market. It will be interesting to see if they are accepted. They are expensive, and wearers are commonly called glassholes. I’m not sure everyone else likes being filmed all the time, so it will be interesting to see how this evolves.

google-glasses-2

Mind Reading

We are starting to see devices that can interpret and act on the electrical signals generated from the brain. Right now it takes a fair bit of concentration and training to use these, but as these get more refined, how long before we can practically control our computers via thinking? How long before we have a USB port embedding into our neck where we can read USB sticks directly?

mindcontrol

Summary

We’ve come a long way from punched cards to Google Glasses. We’ve adapted input devices from all sorts of innovative techniques from keyboards to mice to touch to voice to gestures and R&D into new techniques is progressing at a breakneck pace. It will be really amazing what comes out over the next few years. Which experimental technologies go mainstream, which mainstream technologies die out?

Written by smist08

January 11, 2014 at 9:14 pm

Sage 300 ERP Optional Fields

with 4 comments

Introduction

Optional Fields were added to Sage 300 ERP as the major feature for version 5.3A. They are a great way to add custom data fields to many master and transaction screens in Sage 300. They also have the benefit that we can flow them with the transactions through the system, for instance from an O/E Order to the corresponding A/R Invoice to the G/L Batch. This opens up a lot of power for tracking extra data in the system and to do sophisticated reporting based on it.

In this article we will look at some of the programming consideration when dealing with Optional Fields at the API level. We’ll use the Sage 300 .Net API to explore how to deal with a few things in that come up with Optional Fields.

Many programmers feel they can ignore Optional Fields and their program works perfectly for sample data, but as soon as they install it at a customer site, it fails. A common cause of this is that the customer has required optional fields that cannot be ignored and the API program needs to be updated. (Another common failure at customer sites is caused by not dealing with locked fiscal periods).

optionalfields

Ordered Header/Detail

From my previous article on the View Protocols, any optional field view is an ordered detail where its header is whatever it is an optional field for. An ordered header/detail means that you set the value of the key. But you do have to be careful to setup your optional fields correctly in the application’s setup UI since there is validation on these.

In the sample program I’ll give a complete set of steps. In some situations some of the steps can be skipped, for instance you don’t really need to call RecordClear and RecordGenerate for the header optional fields, but I’ll leave them in since sometimes you do, and it’s easier to just use a formula that always works rather than re-thinking everything each time.

Revision Lists

Since Optional Fields are often sub-details of details that are stored in revision lists, you sometimes need to be careful that things exist before using them. Revision Lists are our mechanism to store things in memory before they are written to the database in a single database transaction. So until you save the header, nothing is in the database and everything is in memory. The Revision List store the list of details that have been manipulated so far. The Optional Fields are themselves stored in Revision Lists until the big database transaction happens.

In the sample program we insert the detail before adding this optional fields. Until the detail is inserted there is nothing to attach the optional fields to, so we need to do that first. The Insert operation only adds the detail record to a revision list in memory, but once that is done we can add sub-details (ie Optional Fields) to it (which are also stored in memory).

Sample Program

For a sample program, I modified the ASP.Net MVC sample ARInvEntry to save a couple of optional fields for the header and a couple of optional fields for the detail. Below is a subset of the code to insert the detail optional fields:

   // 9. Insert detail.
   arInvoiceDetail.Insert();   // Insert the detail line (only in memory at this point).

   // 10. Add a couple of detail optional fields.
   arInvoiceDetailOptFields.RecordClear();
   arInvoiceDetailOptFields.RecordGenerate(false);
   arInvoiceDetailOptFields.Fields.FieldByName("OPTFIELD").SetValue("EXTWARRANTY", false);
   arInvoiceDetailOptFields.Fields.FieldByName("VALIFBOOL").SetValue(true, false);
   arInvoiceDetailOptFields.Insert();
   arInvoiceDetailOptFields.RecordClear();
   arInvoiceDetailOptFields.RecordGenerate(false);
   arInvoiceDetailOptFields.Fields.FieldByName("OPTFIELD").SetValue("WARRANTYPRD", false);
   arInvoiceDetailOptFields.Fields.FieldByName("VALIFTEXT").SetValue("180 Days", false);
   arInvoiceDetailOptFields.Insert();

   // 10.5 Register the changes for the detail.
   arInvoiceDetail.Update();

   // 11. Insert header. (This will do a Post of the details.)             
   arInvoiceHeader.Insert();

Different Field Types

If you look in the database you will see that the Optional Field tables don’t hold that many fields, but any Optional Field View has quite a few fields, but many are calculated. Optional Fields can be all sorts of different types, but in the database all the values are stored in a single VALUE field regardless. So there has to be a conversion to/from this text field and the real type. This is the job of all the VALIFtype fields. Basically you use the VALIF field based on the type of the Optional Field and then the View will handle the conversion to and from the type as stored in the VALUE database field. That is why we used the fields VALIFTEXT and VALIFBOOL above.

Auto Insert

In the sample program I just inserted the optional fields myself. However there is another way. Views that have optional fields usually have a field called PROCESSCMD (or something similar) where you can set a value with a title like “Insert Optional Fields” which will insert the optional fields that are needed. You can set this field and call Process on the View and it will insert these optional fields for you. Then you can read the optional field, set its value and update it. Some people find this an easier way to do things and you get all the optional fields with their default values as a bonus. (Note that this only applies to Optional Fields that are set to auto insert in the applications Optional Fields setup screen).

Similarly for Views that will transfer in Optional Fields (like from an Order to a Shipment), you will see PROCESSCMD’s like “Default and Transfer Optional Fields”. If you are doing API programming and want to preserve the flow of Optional Fields, you will occasionally have to set one of these and call Process.

When dealing with Optional Fields, it’s worth checking out the PROCESSCMD functions of the main View to see if they will help you do your job and save you a fair bit of coding.

Lookup Values

Many optional fields have a list of valid values. If you don’t set it as one of these you will get an error message to this effect. When dealing with optional fields make sure you have an error handler to show the errors after an exception as explained here. If you want to get at these values they are in CSOPTFD CS0012 (a detail of CSOPTFH CS0011). You can read through these views like any other as explained here.

Summary

Optional Fields are a powerful feature in Sage 300 ERP. Many customers use these in a fundamental way to support their businesses. This means that developers working in the Sage 300 world have to be cognizant of these and make sure their programs are compatible with their use.

Written by smist08

January 4, 2014 at 3:54 am