Stephen Smith's Blog

Musings on Machine Learning…

Archive for April 2010

Diagnosing Problems Running Accpac Part 3

with 4 comments

In the first part of this series we looked in detail at what happens from double clicking on the Sage Accpac ERP icon through to seeing the signon dialog being displayed. In the second part we looked at the steps that happen from hitting ok on the signon dialog through to the desktop being running displaying the tree of icons. In this installment we will look into what happens when you double click on an application screen on the desktop.

Each application form is an ActiveX control (http://en.wikipedia.org/wiki/ActiveX). Every ActiveX control needs to be registered. When you first double click on an icon we will use the a4wInstaller component to see if that control is registered and if not, register it. You can register all the screens ahead of time using the regacc.exe tool. If running web deployed (or a screen launched from SageCRM) it will first check to see if the control exists on the current computer and if it doesn’t it will download it. Once the control is registered, Accpac will run a copy of a4wContainerXP.exe to host this ActiveX screen control. ActiveX controls can’t run on their own, they must be hosted inside a COM Container like a4wContainerXP, Internet Explorer or Visual Basic. A4wContainerXP will use the mechanisms of COM to load the ActiveX screen and display it.

Now the screen will actually run. At this point it checks that a lanpak is being used and that this application is licensed; then opens a connection to the database and load any Views (Business Logic Objects) that it requires. It also creates all its screen controls (which are ActiveX controls themselves) and connects them to the Views. Each View is wrapped by a Data Source control which provided additional COM services for each Views (like events).

When diagnosing when a UI won’t start, there are three very powerful spying utilities included with Accpac. These include Accpac Spy which spies on everything that happens in the COM components, RVSpy which spies on the business logic layer and DBSpy which spies on the database layer. The User Interface form consists of a number of COM components that provide all the UI controls and services. These communicate with the Business Logic layer which communicates with the database layer. Below is a diagram of the 3 tier stack along with the Spy tools that spy on each layer.

Using Accpac Spy you can Spy on the various components as they initialize to see if anything goes wrong. The usual order to spy is as follows:

  • Container – Window that hosts the application screen (a4wContainerXP.exe)
  • Installer – Automatic registration of UI OCX (a4wInstaller.dll)
  • Session Management – Auto-creation of session if already signed on, etc.
  • Datasource controls – used by UI to access application views
  • Edit and list controls, finder, import/export – Advanced

Accpac Spy should be able to diagnose problems with the various COM and ActiveX controls that make up a User Interface form.

RVSpy records everything that happens at the Business Logic Layer:

  • Records operations on all application views, including 3rd party applications
  • Which record is read, field values retrieved, when update happens, etc.
  • Output is nested if one view calls another
  • Can include DBSpy output to show what database calls are made for a particular view call
  • Views are identified by View IDs
    • AP0015: APVEN
      • AP Vendors
    • First 2 characters are always the application prefix so you can determine if a view is Sage Accpac’s or a 3rd party application
  • Sage Accpac Application Object Models

DBSpy records everything that happens at the Database Layer. It is good for diagnosing problems with the database server. It records every error from the database and sometimes these don’t make it up to be displayed at the UI level.

All the preferences for things like grids, like the column order, column sizes and such are stored in the shareddatafolder\user\userid folder as orgid_p.ism. If a screen won’ start it is often worthwhile to rename this file to see if that helps (it will be re-created when you run Accpac). ScanIsam should find any structural problems with this file and possible fix them.

One common cause of problems with the UI COM components is having had multiple versions of System Manager installed at different locations on your hard disk. Often then the wrong component will be in the PATH or the wrong one will be referenced by the Window’s registry. If you have this case, its best to delete all the old copies of System Manager, clean up your PATH environment variable and re-run regacc.exe to make sure all the correct objects are registered. Similarly moving the programs between a file server and a local hard drive can cause the same sort of mismatches. Best is to only have one copy visible so you know you are running the correct files. You can also use Process Explorer from System Internals (now part of Microsoft) to select the a4wcontainerxp.exe process and check that all the DLLs it has loaded come from where you expect System Manager to be installed. If you do want to run multiple versions of Accpac, its far more trouble free to use Virtual images with products like VirtualPC or VMWare; these guarantee that the multiple versions won’t interfere with each other.

When you report problems to technical support, they will often request all these logs so they can isolate in the code where things are going wrong. Then they can diagnose the problem. Sometimes they will go to development, where hopefully the spy log has isolated where the problem is occurring so the programmer can look in the code and figure out what is going on. If the log doesn’t isolate things sufficiently, development can produce a “patch” DLL with additional logging to isolate the problem further. As a first step they will try to reproduce the problem in support and only pass on the “patch” DLL to the customer if they can’t reproduce the problem in support.

This concludes this installment of diagnosing problems, next week we’ll look at problems running Crystal Reports.

Written by smist08

April 24, 2010 at 7:55 pm

Posted in sage 300

Tagged with , ,

Diagnosing Problems Running Accpac Part 2

with 27 comments

Last time we looked at what happens from when you double click on the “Sage Accpac ERP” icon to the sign-on dialog appearing. This time we will look at what happens when you click ok on the sign-on dialog through to when the desktop is displayed and ready to go.

When Accpac loaded the sign-on dialog with the company list, it kept track of whether security is turned on for that company/system database. This information is stored in orgs.ism. Accpac will then validate the user name against accpacshareddata\site\users.ism. If security is enabled it will also validate the password.

If you are having trouble signing on, first run DBSpy. If nothing appears in DBSpy then the problem is before opening the database links and will be to do with something in the accpacshareddata\site directory. Run ScanIsam to ensure the various ism files are ok. Sometimes ScanIsam can even fix them. Also remember that every user must have full control security access to all the files in the site directory.

Next Accpac will try to reserve a lanpak slot for this workstation/company/userid combination. If there is a free slot then you will be allowed to proceed, if not you will get an error message that all the lanpaks are in use. You can use the Desktop menus to see how many lanpaks are available and how many are used and which userid/company combinations are using them. If you think they should be available make sure the lanpaks aren’t expired 30 day licenses. If everyone is out of the system, it is safe to delete the site\lanpak.bin file. It will be re-created the next time you run Accpac. To do all this the desktop runs the a4wLPMgr.exe program which then tracks the lanpak usage for this workstation.

Accac will now run the Accpac Signon Manager (a4wSignonMgr.exe). This isn’t required by the Desktop, but is run so if anyone starts a UI program from outside the desktop, it can connect to this to share the signon info. You see this  running in the icon tray in the bottom right of your screen. If the signon manager doesn’t run as part of the desktop startup process, you may have a problem with the Accpac COM components. The best bet is to re-register these by running accpac\runtime\regacc.exe. This program will register all the Accpac COM components, normally this is run by the Accpac installation program, but sometimes parts of this can fail.

Accpac then needs to register with semaphore.bin that you are using this company. If someone has the company open exclusively (for something like Database Load) then you will get an error to that effect. Otherwise this is accomplished by using some more byte locks on this file.

Now Accpac is ready to actually open a database connection. With 5.5A and later we just open a connection to the company database. Prior versions will also open a connection to the system database. However the steps taken are similar for both. Encrypted within the records in orgs.ism is sufficient information for Accpac to sign-on to the database. This is the information you entered in Database Setup. Depending on the database, Accpac will use database client software to login to the Database Server. For Pervasive we connect to both the transactional engine using the Pervasive client as well as to the relational engine using ODBC. For SQL Server and Oracle we connect using ODBC. For newer versions of Accpac we login to SQL Server using the SQL Native Client OBBC driver. All of these require that the database’s client software be installed on the workstation and that the directory where the client DLLs are located are in the PATH environment variable. This is true even if using an ODBC only connection. For SQL Server, you used to not see this since MDAC and many other pieces of Microsoft software install the SQL Server client software. However with the introduction of the SQL Native Client, this wasn’t included in MDAC and now requires the client software be installed. As a best practice its best to install the client software from the DVD that you installed the database server with, on every workstation. Accpac 5.6 will now install the SQL Native Client as part of its install to save hassle. But say if once we support SQL Server 2010, then you would want that client software installed on every workstation rather than the SQL Server 2008 client installed by 5.6A. A very common cause of support calls are the result of people running client software that is older than the version of the database server. Its seems to usually be ok to run new versions of the client software, but older versions seem to always lead to problems. If you are having database connection problems, try running the “Enterprise Manager” software that came with the database to see if it can connect ok, if not often it has some helpful advice (or at least an error message you can Google). For database connection problems its often useful to look through the database vendor’s knowledgebase to see if someone else has experienced and solved a similar problem. For Pervasive problems are often associated with setting up the workstation and server ODBC datasources correctly. Setting up Oracle can be tricky, but usually if you can test the ODBC connection successfully then Accpac will work. The first thing Accpac checks when connecting to the database is whether the version of the database client and server software is supported. Getting back strange errors from these calls will often result in a “Database version is unsupported” error. If you are sure the version is correct, then this is probably a result of a connectivity problem that confused Accpac checking the version.

Once you start connecting to the database, DBSpy becomes the tool of choice. It will record all diagnostics from the database to hopefully help you diagnose the problem. Googling any error messages usually yields good clues as to what is going wrong. Most databases also have tracing facilities that can be quite helpful in diagnosing problems. If you are running on Terminal Server and have run the Web Deployment Wizard and chosen the DCOM protocol, then you have to run DBSpy (or RVSpy) from the console, they will always come up blank from a terminal server client, best solution to this is to run the Web Deployment Wizard again and choose .Net Remoting as this doesn’t have the problem and works much better.

Once a database connection is successfully established, Accpac will read through the CSAPP table. This contains the list of application (like AP, AR, etc.) along with their version and datalevel. For AS and CS it will see if these need to be activated to the current version of System Manager or if these are ok. Perhaps we’ll talk about data activation in a later posting. But basically we get the list of applications and their versions that are activated for this company. You can see this occur in DBSpy.

Now Accpac will go through this list and construct the path accpacprograms\applicationversion (say accpac\ar56a) and try to load the roto.dat file there. This file contains the list of all the Views (Accpac Business Logic Objects) and User Interface objects for a given application. (There are some variants on roto.dat for applications that sub-class other applications that will also be read at this time). With these loaded, Accpac now knows how to run any UI or View for that application.

Now that a user is signed-on to a company, Accpac will read the accpacshareddirectory\user\userid\orgid.wsp file. This contains the layout of the various desktop panes from when the user was last signed on. This file can be safely deleted and will be re-created when the user next signs on.

The desktop is now ready to start constructing the tree of icons it displays. Again the desktop goes through the list of activated applications and for each one looks in its language specific directory for grp.dat (perhaps ar56a\eng\grp.dat). This file contains all the icons to display on the desktop along with their captions and which program to run when they are double clicked.

As the desktop loads the icons from the group file, the accounting application has the option to hide some of the icons. The desktop looks in the application’s xx.ini file (say ar.ini) and in the section [callbacks], if there is a startupIcon entry then the desktop will call the specified DLL for each icon, so the application can hide it. This is why the icons for multi-currency related activities like revaluation don’t show up when you sign into a single currency company.

One problem that we see now and then is people installing a larger product update that includes updated grp.dat and roto.dat files. If the base application wasn’t installed, but the application was activated (like they all are in sample data) then the desktop will pick up the roto.dat and grp.dat and display the icons for this application. But since it’s a product update, it isn’t complete, so clicking on the icons leads to all sorts of missing file errors.

Once the standard desktop icons are loaded, the desktop will load all desktop customizations. These are stored in the accpacshareddata\user\userid\orgid_c.ism. This file contains any icons that the user has added to the desktop for this company. There is also a accpacshareddata\user\_EVRYONE\orgid_c.ism file that contains icons added to the desktop and set for all users. Since these are Accpac ISAM files, if there are problems with loading custom icons, run ScanIsam to see if there are problems. If so perhaps ScanIsam can fix them. If you don’t mind losing your customizations you can delete these files and the will be re-created the next time you run Accpac.

If you are running multiple Window’s users on a workstation and are running Vista or Windows 7, you might run into problems with the operating systems file shadowing feature. The ideas of this was to shield the changes to files that one user makes from other users. However in multi-user applications this tends to cause havoc. Your best bet is to install Accpac to a location where this feature isn’t enabled (ie anywhere except under Program Files). Otherwise changes one user makes, say be running Database Setup won’t be visible to other users. This is further complicated by Windows Services which run as another user and installation programs which get aliased to run as the Administrator account.

Now the desktop should be signed in. The next step is to run an Accpac screen by double clicking on an icon. Next week we’ll look in detail at the steps that happen when you run an Accpac VB screen.

Written by smist08

April 17, 2010 at 8:37 pm

Posted in sage 300

Tagged with , ,

Diagnosing Problems Running Accpac Part 1

with 38 comments

This blog post is going to go into quite a bit of detail on what happens when you double-click on the “Sage Accpac ERP” desktop icon. Many of you have had conversations with our technical support department and many have wondered why they ask you to do the things they ask. Why do they ask you to run RVSpy/DBSpy? Does it tell them anything useful if it comes up blank? What do all those options in Accpac Spy mean? A lot of times what they are trying to do is diagnose where in the startup process things have gone wrong and what area do they need to concentrate on. A lot of times the net starts wide and then gets drawn in around the problem.

So to start with you double click on the “Sage Accpac ERP” icon. This then runs the Accpac.exe program in the runtime directory of where ever you installed Accpac. When you double click on the icon, Windows will load that EXE and it will find and load all the DLL’s it is statically linked to. Some of these DLLs are part of Windows itself, some of these are part of the Microsoft C/C++ runtime and MFC library, and many are DLLs installed in the Accpac runtime directory. All these DLLs might be statically linked to other DLLs. In total 39 DLLs are usually loaded with Accpac.exe as a result of double clicking on the icon. When we install Accpac we try to give Windows the best chance of finding all these DLLs that we can. Windows looks for DLLs in the following places:

  1. The same directory as the EXE (ie accpac\runtime).
  2. The current working directory (the “start in” field in the icon properties, usually accpac\runtime).
  3. The Windows System Directory (windows\system32).
  4. The Windows Directory (windows).
  5. The directories listed in the PATH environment variable (should include accpac\runtime).

Some things that can go wrong with this process include:

  1. Not having execute and/or read security access to a directory where one of these is installed.
  2. Having a corrupted PATH environment variable. If something didn’t handle a path with spaces properly or didn’t separate things properly with semi-colons, it can cause loading problems for all sorts of things.
  3. On terminal server a user has a local windows directory as well as the main windows directory. Sometime things can be installed properly for one user but not another because things installed into their local windows directory rather than the global one. This was a problem with the older Crystal 8.5 runtime, but newer Crystal runtimes handle this ok. (Crystal actually isn’t required at this point).
  4. Having multiple versions of System Manager installed at once in different locations. This can happen if you use programs like “UniLauncher” to run multiple versions of Accpac. This often causes multiple accpac\runtime directories to be in the PATH, or the Accpac icon points at once place and then a different accpac\runtime is used to load the DLLs. Running the wrong version of these DLLs will cause all sorts of problems. If you need to use multiple versions of Accpac System Manager, it is far safer and less hassle to use virtual machines like Microsoft’s VirtualPC to run the various configurations each in their own VM. This also allows you to run several at once and avoid them getting mixed up.
  5. Viruses can cause all sorts of strange problems.
  6. Disk drives that are starting to fail and causing some files to fail to be read.
  7. Note also that Accpac.exe and all these DLLs are 32-Bit. They will run fine under 64-Bit Windows. But if you are a developer and want to use these DLLs yourself then you need to compile for 32-bit in order to load them. 32-Bit DLLs can only be loaded by a 32-bit process, not a 64-bit process. 64-Bit IIS will spawn 32-Bit processes for this reason if configured to do so. .Net programs and Java programs can be compiled for either.

Besides outright not working, it could be that it takes quite a long time to reach the sign-on dialog after clicking the icon. These are usually related to networking issues. Most customers install the Accpac programs on a shared file server, so they don’t need to be installed on every workstation and so they can apply product Updates to one place. Some things to note:

  1. Use UNC paths (\\myserver\myvolume\accpac) rather than mapped network drives. There are many Microsoft knowledgebase articles of various bugs in mapped drives. They may work fine for you or you might run into one of these depending on your environment. Search the Microsoft knowledge base to see if you fall into one of these scenarios.
  2. Bad DNS entries. Sometime if your network isn’t configured properly network requests can takes quite a long time. Try “ping myserver” to ensure this is fast (<1ms). If it isn’t then you will take a long time to load due to the number of DLLs that must be loaded.
  3. Running too many virus checkers. We’ve seen people running 5 different virus scanners at once. This sort of thing can really slow things down. Also if you know your server is running a virus scanner then have the workstation exclude network shares/drives since this is just double work.

Rather than figure these problems out, many people find a Terminal Server configuration more convenient since everything is installed locally on the terminal server, so you don’t have the network problems. You also maintain the advantage of having Accpac only installed in one place so you don’t need to apply Product Updates all over the place. Other people, if they don’t want to use Terminal Server, will install the Accpac programs locally on each workstation. This is more work, but again avoids some network speed problems, you may still have some problems when you access the shared data directory or the database server when you finally sign-on.

Accpac.EXE is a Windows executable program that is written in C++ using the Microsoft Foundation Classes (MFC) library. Once Accpac.exe is loaded and running then our code is finally executing and we will use the MFC library to create the desktop window layout. MFC stores its configuration stuff in the registry. These are things like the sizes of the various panels the last time you exited Accpac. These are stored under:

HKEY_CURRENT_USER\Software\ACCPAC INTERNATIONAL, INC.\ACCPAC

There are 5 keys that all start with the word “FrameState”. If these registry entries become corrupted for some reason, then Accpac will not start. These registry entries can be safely deleted, you will just set the frame sizes back their defaults and they will be re-created the next time you run Accpac.

Accpac splits its files into two categories the “Programs” directory and the “Shared Data” directory. The programs are all the executable programs and DLLs that make up the Accpac program. The “Shared Data” directory are various non-database files that store useful information relevant to Accpac. The “Shared Data” directory must be shared by all users on the system, they must all look at the same directory or there could be problems. The Programs can either be common, or people can look at their own copy. You must be able to read and execute the programs directory, and you must have full control over the shared data directory. These directories are stored in the registry under:

HKEY_LOCAL_MACHINE\Software\ACCPAC International, Inc.\ACCPAC\Configuration

These are stored in the Programs and SharedData keys.

Accpac will check if you are licensed to run this version of System Manager. It will access accpacshareddata\sm56a0.lic to validate that you are properly licensed. If you have an expired 30 day (or other period) license, are missing this file, etc. Then you won’t be allowed to run Accpac. The desktop will also do a quick scan of your other license files to see if there are any inconsistencies or suspicious entries. If so then you will be re-directed to the Sage Anti-Piracy web page.

Accpac will also open accpacshareddata\site\semaphor.bin to see what others are doing in Accpac. This file can be deleted if causing problems (when everyone is out of the system) and will be re-created the next time you run Accpac. Again security access rights are the most common cause of problems here since Accpac.exe needs full control of this file. Accpac controls access to different parts of the program by placing byte locks in this file. Usually these are cleaned up when Accpac exits, or are cleaned up by Windows if Accpac terminates abnormally, or if a workstation disconnects unexpectedly from the server. However sometime Windows gets confused and these locks need to be cleared by re-booting the file server where this file resides. If you have to do this frequently then you have some other problem going on and need to investigate with FileMon (discussed below).

Accpac will access accpacshareddata\site\orgs.ism to load the list of companies you can logon to into the sign-on dialog. Orgs.ism is a simple isam file. It needs to be stored outside of the database since it contains the information required to sign-on to the database. If this file is corrupted then you may not be able to get to the sign-on dialog. Run scanisam to see if there are problems. It might also be able to fix them. Otherwise get a fresh copy from another install and re-populate it (admittedly a pain).

The best way to diagnose problems that happen before the sign-on screen is reached is to use the “FileMon” program from SystemInternals (now owned by Microsoft). Their web site is: http://technet.microsoft.com/en-ca/sysinternals/default.aspx. Its worth having a look through their utilities, they are really useful. Regmon is another very useful program to monitor for registry errors. Often these programs will show quite useful information, but they will fill up fast. Often worth stopping any windows services not associated with Accpac when you run these to reduce the noise.

This blog post is getting quite long, and we’ve only just got to the sign-on dialog. If I’ve forgotten something, I’ll update this post. Next week we’ll look at what happens when you click OK on the sign-on dialog.

Written by smist08

April 11, 2010 at 8:07 pm

Posted in sage 300

Tagged with , ,

Automated Testing in Sage ERP Accpac Development

with 7 comments

All modern products rely heavily on automated testing to maintain quality during development. Accpac follows an Agile development methodology where development happens in short three week sprints where at the end of every sprint you want the product at a shippable level of quality. This doesn’t mean you would ship, that would depend on Product Management which determines the level of new features required, but it means that quality and/or bugs wouldn’t be a factor. When performing these short development sprints, the development team wants to know about problems as quickly as possible so any problems can be resolved quickly and a backlog of bugs doesn’t accumulate.

The goal is that as soon as a developer checks in changes, these are immediately built into the full product on a build server and a sequence of automated tests are run on that build to catch any introduced problems. There are other longer running automated tests that are run less frequently to also catch problems.

To perform the continuous builds and to run much of our automated tests we use “Hudson” (http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson). Hudson is an extremely powerful and configurable system to continuously build the complete product. It knows all the project dependencies and knows what to build when things change.  Hudson builds on many other tools like Ant (similar to make or nmake) and Ivy among others to get things done (http://java-source.net/open-source/build-systems). The key thing is that it builds the complete Accpac system, not just a single module. Hudson also has the ability to track metrics so we get a graph of how the tests are running, performance metrics, lines of code metrics, etc. And this is updated on every build. Invaluable information for us to review.

The first level of automated testing are the “unit tests” (http://en.wikipedia.org/wiki/Unit_testing). These are tests that are included with the source code. They are short tests where each test should run in under 1 second. They are also self contained and don’t rely on the rest of the system being present. If other components are required, they are “mocked” with tools like easy-mock (http://easymock.org/). Mocking is a process of simulating the presence of other system components. One nice thing about “mocking” is that it makes it easy to introduce error conditions, since the mocking component can easily just return error codes. The unit tests are run as part of building each and every module. They provide a good level of confidence that a programmer hasn’t completely broken a module with the changes they are checking in to source control.

The next level of automated tests are longer running. When the Quality Assurance (QA) department first tests a new module, they write a series of test plans, the automation team takes these and transfers as many of these as possible to automated test scripts. We use Selenium (http://en.wikipedia.org/wiki/Selenium_(software)), which is a powerful scripting engine that drivers an Internet Browser simulating actual users. These tests are run over night to ensure everything is fine. We have a subset of these call the BVT (Build Validation Test) that runs against every build as a further smoke test to ensure things are ok.

All the tests so far are functional. They test whether the program is functioning properly. But further testing is required to ensure performance, scalability, reliability and multi-user are fine. We record the time taken for all the previous tests, so sometime they can detect performance problems, but they aren’t the main line of defense. We use the tool JMeter (http://jakarta.apache.org/jmeter/) to test multi-user scalability. JMeter is a powerful tool that simulates any number of client’s accessing a server. This tool tests the server by generating SData HTTP requests from a number of workstations and bombarding the server with them. A very powerful tool. For straight performance we use VBA macros that access the Accpac Business Logic Layer to write large numbers of transaction or very large transactions, which are all timed to make sure performance is fine.

We still rely heavily on manual QA testers to devise new tests and to find unique ways to break the software, but once they develop the tests we look to automate them so they can be performed over and over without boring a QA tester to death. Automated testing is a vibrant area of computer science where new techniques are always being devised. For instance we are looking at incorporating “fuzz testing” (http://en.wikipedia.org/wiki/Fuzz_testing) into our automated test suites. This technique if often associated with security testing, but it is proving quite useful for generalized testing as well. Basically fuzz testing takes the more standard automated test suites and adds variation, either by knowing how to vary input to cause problems, or just running for a long time trying all possible combinations.

As we incorporate all these testing strategies into our SDLC (Software Development Life Cycle) we hope to make Accpac more reliable and to make each release more trouble free than the previous.

Written by smist08

April 5, 2010 at 5:24 pm