Archive for February 2010
Web based applications can be quite complex and good tools are required to simulate heavy user loads, automate functional testing and diagnose/troubleshoot problems once they are discovered. We have to ensure that Accpac will keep running reliably under heavy user loads as users are performing quite a diverse set of activities. With Accpac 6 there are quite a few new components like the dashboard data servlets that need to have good performance and not adversely affect people performing other tasks. We want to ensure that sales people entering sales orders inside CRM are as productive as possible.
We are fundamentally using SData for all our Web Service communications. This involved a lot of building and parsing XML. How efficient is this? Fortunately all web based development systems are highly optimized for performing these functions. So far SData has been a general performance boost and the overhead of the XML has been surprisingly light.
Towards ensuring this goal we employ a number of open source or free testing tools. Some of these are used repeatedly run automated tests to ensure our performance goals are being met. Some are used to diagnose problems when they are discovered. Here is a quick list of some of the tools we are using and to what end.
Selenium (http://seleniumhq.org/) is a User Interface scripting/testing engine for performing automated testing of Web Based applications. It drives the Browser as an end user would to allow automated testing. We run a battery of tests using Selenium and as well as looking for functionality breakages, we record the times all the tests take to run to watch for performance changes.
JMeter (http://jakarta.apache.org/jmeter/) is a load testing tool for Web Based applications. It basically simulates a large number of Browsers sending HTTP requests to a server. Since SData just uses HTTP, we can use JMeter to test our SData services. This is a very effective test tool for ensuring we run under heavy multiuser loads. You just type in the number of users you want JMeter to simulate and let it go.
Fiddler2 (http://www.fiddler2.com/fiddler2/) is a Web Debugging Proxy that records all the HTTP traffic between your computer and the Internet. We can use this to record the number of network calls we make and the time each one takes. One cool thing Fiddler does is record your network performance and then calculates the time it would have taken for users in other locations or other bandwidths would have taken. So we get the time for our network, but also estimates of the time someone using DSL in California or a modem in China would have taken to load our web page.
Firebug (http://getfirebug.com/) is a great general purpose profiling, measuring, debugging tool for the Firefox browser. Although our main target for release if Internet Explorer, Firebug is such a useful tool, that we find we are often drawn back to doing our testing in Firefox for the great tools like Firebug that are available as add-ins.
Using these tools we’ve been able to successfully stress the Accpac system and to find and fix many bugs that have caused the system to crash, lockup or drastically slow down. Our automation group runs performance tests on our nightly builds and posts the results to an internal web site for all our developers to track. As we develop out all the accounting applications in the new SWT (Sage Web Toolkit), we will aggressively be developing new automated tests to ensure performance is acceptable and look to keep expanding the performance of Accpac.
As Chief Architect for the Accpac product line, I usually write about technology topics, especially the new technologies that are going into version 6. However I do get requests to write about other application features that are being enhanced.
As we developed the plans for version 6, we are taking on a number of big challenges to update the Accpac technology to modernize our User Interface. This includes improving TCO by being fully Web-based so no Workstation Setup is required and investing a lot of time and money in User Centered Design to improve the look, usability and workflow of all our screens.
However, we are still investing time and effort to address the main pain points in our Accounting applications. So for this blog posting, I thought I’d cover the new “Locking Fiscal Periods by Module” feature.
From our user surveys, site visits and business partner feedback, a much requested feature is the ability to lock fiscal periods by accounting module. Suppose as we pass month end, we might want to ensure all Order Entry clerks are entering transactions in the new month, since they are live and online; but, in A/R we may still be processing some paper documents that still need to be entered. So for data entry accuracy we would like to lock the last month for O/E but still keep using it for a few days for A/R. Similarly when A/P finishes the last month they want to lock the period independently from whether A/R is finished or not.
We expanded the Common Services Fiscal Calendar function, to rather than lock/unlock a fiscal period for everyone at once, to having a matrix of locking of period versus application. Then you can check/uncheck for each application/period independently.
This looks intimidating, but isn’t too bad in real use, since practically you are only dealing with one column at a time. Plus we provide buttons to lock/unlock whole columns and rows at a time.
Then all the Accpac Accounting modules have been modified to use this new data and behave accordingly.
Notice that the screenshot is of one of our standard VB screens, showing we are still doing development here, even as we are developing out all our new User Interfaces.
SData is the new Sage standard for Web Services. It is a REST based protocol similar to GData. Typically REST based protocols are “stateless”, meaning that any request doesn’t rely on knowledge of other requests. An SData request might be something like add an order where the order is provided as an XML payload; update a customer record where the new fields are in the XML payload. This is a very efficient way to do things. Each document entered just requires one round trip request from the client to the server.
“Stateless” processes are very desirable in large Web based applications. It means each request can be directed (load balanced) to one of a set of servers in a server farm. History doesn’t matter; each request can go to any server. If a process is “stateful” then requests from a given client have to be directed always to the same server. “Stateless” is also desirable since the server doesn’t keep any knowledge of things in progress, so no server resources are committed between calls. Also if the server is restarted, or a process on the server is restarted, besides perhaps a slight delay, the clients never know, since no info was kept in the process that was stopped.
However this typically isn’t how we enter orders from a User Interface form. From the User Interface we build up an order a piece at a time. First we input the customer number or an order template, from which we fill in a number of field defaults. Then typically we update some order header fields, and then enter a number of detail lines. As we enter things they are validated and often entering one field like a currency code or item number causes a number of other fields to be updated.
One way to do this would be to send the order to the server each time something happens and have it updated and sent back. This would keep things stateless, but would tend to become unwieldy as the order gets large. Another consideration is that we want to keep using the Accpac Business logic more or less unchanged.
The Accpac Business logic builds up the order inside the Business logic (View) as it goes along. Then when you say save, it commits everything to the database as one transaction. In this model as parts of the order are completed they get applied to the in-memory order, defaults are looked up and validations are performed. So typically an UI will issue a command to create a new order and receive back the default values and states for all header fields. Then the UI will enter the customer number or a template code, this will go to the business logic which will then validate this value and lookup the defaults, apply them to its copy of the order and pass them back to the UI. At each step the UI is really updating the server memory part of the order. The UI doesn’t need to keep all entered details in memory since it can ask the server for them at any time. When the order is saved, the UI just sends a save request (with no data) and the server commits the order from memory to the database in one database transaction.
So how do we reconcile stateless SData with stateful Accpac? Generally we try to keep as many things stateless as possible, but for some operations, mostly entering documents, we need to introduce state to SData. We add an SData call that will start “stateful” processing on a given SData resource like an order. This means then that SData will keep state on the resource, which under the hood means it will tie a View instance to the client and preserve it for the client. Then the UI rather than send the whole order, just sends the parts that the user changed (the UI has to know when to trigger this) and then SData applies these changes to the Accpac Views, and then gets all the fields that changed as a result (either in value or in state) and returns them as the return payload to the SData call. This then goes on until an SData call like a header navigate or save happens to clear the state. This means that while the Order is being compiled server resources are being used and all requests for this client have to be routed to this particular server.
In a Web Application like Facebook with say 500,000 users online at once, keeping any state on the server is incredibly expensive. Plus things need to be load balanced in the most flexible manner possible. But I think Accpac has a bit of breathing room before we reach this kind of volume. At least we are moving in the right direction, having many things stateless such as inquiries and simple file updates/inserts. We just keep state during document entry of header/detail type documents. But if Accpac does start having hundreds of thousands of users online then we will need to start evolving our business logic to become truly stateless. But for now we have a nice compromise of using our existing business logic, but becoming stateless in a lot of what we do.
With the next generation of Accpac’s Web based screens, we wanted to greatly improve the customizability of Accpac. We have a User Centered Design team that is working hard to improve the general usability of the product with the hope that fewer customizations are required. But as a mid market product we know that in most cases the 80 – 20 rule still applies, to ensure we deliver at least 80% of the customer’s required functionality, but be flexible enough that the remaining 20% can be filled by ISV partners and customizations.
With-in Accpac 6 there are three categories of screen customization:
- In product. Basically screens and dialogs that allow users to customize their own screens and workspace. This includes things like adding shortcuts to the shortcuts bar and choosing their Data Portlets for their Portal home page. Configuring the Data Portlets to display what the customer requires.
- Codeless customization. These are customization that can be performed exclusively inside the screen/form designer. These customizations don’t require any Java programming.
- Coded customization. Here you are writing Java code to extend and modify the shipped forms.
For any customization we want to ensure that the user can easily upgrade to the next version without having to re-create the customizations from scratch. Ideally this should be handled transparently via the upgrade process. But once coding is involved then there is quite a bit of room to break this, we will publish best practices, so your code will just need to be re-compiled on the new version, but it’s up to the partner performing the customizations to follow these guidelines.
For codeless customizations, all the User Interface screens are saved on the server as XML files separate from the UI program. This means you can edit/customize these XML files without needing to use programming code or development tools. However we do provide a screen designer to edit these XML files in a friendlier manner. In Sage ERP Accpac 6.0, this tool will be available for developers via the SDK, and in 6.1 will be part of the standard product for use by anyone with sufficient security rights. Below is a screen shot of the UI designer editing the A/R Billing Cycles form. You can see some of the controls you can select as well as some of the control properties that can be set.
Some typical customization you can do here include:
- Changing text. The form will automatically resize as you change text lengths.
- Adding controls. Controls are controlled via layout, hence you can’t put a control on top of another control. The controls will move out of the way to allow you to insert an new control. You can attach the control to fields in the SData feeds.
- Adding data sources. You can add SData feeds to the form.
- Re-arrange all the control. Change the layout.
- Remove unwanted elements from the form.
- Change the tab order.
- Change existing control properties and CSS styles.
- Change the columns in list controls.
- Assign actions (like button presses) to standard actions provided by the system.
On the server we have classes that receive the SData requests from the clients, and translate these requests into Accpac View (Business Logic) calls. There are a number of classes that do this in a standard manner. However applications can extend the default class for this to do things like add extra fields to the feed. These extra fields may be to provide additional database fields from other places or to perform calculations on the server rather than the client. These classes can also change things like the read-only status of a field. ISV’s or sophisticated customizers can also added classes that extend the existing classes to add additional fields, say for their customizations or change the behavior/values of the existing fields. Since Java fully supports object oriented class inheritance, creating these classes is far easier than creating View subclasses (which you can still do).
On the client, you can extend the classes that comprise the User Interface program that runs in the browser. This is very much like how you wrap our current ActiveX VB UIs. You basically extend the main UI class and get access to a collection of all the controls and SData feeds. You can also add listeners to receive any event you are interested in. Then programming is very similar to how you customize the current UIs. The difference being that you now program in Java rather than VB or VBA, and you have to feed your program through the GWT Compiler to generate a new extended UI.
Hopefully this gives an overview of how customizations work in the new Sage ERP Accpac 6 Web-based world. Generally we are looking to continuously improve our customization model to make it easier to accomplish the customizations that customers require.