Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘jmeter

Stability Testing of Sage 300 ERP

leave a comment »

Introduction

As we roll out Web versions of the Sage 300 ERP (Accpac) accounting modules, we need to ensure that the system can run for a long time without requiring a server re-boot or other manual corrective action. This is far more important in a Web environment than client/server. In client/server the Accpac Business Logic typically runs on the client and if it crashes (which is bad), it only affects one person. In the Web if something crashes then it could affect all users of the system. Many regular GPFs can be captured and handled just causing one person to redo their work; however for more severe errors they could crash a server process or even the server itself. Rebooting a server is a long time for people to wait to continue their work (especially if there is no one around to do the reboot).

As we develop Sage 300 ERP we want to ensure that the system is completely stable and can run for months and months under heavy multi-user load. Hopefully the only time you need to reboot your server is when it installs the monthly patch Tuesday Microsoft Updates.

Why Do Things Crash?

In both client/server and web computing we need to deal with basic GPFs which are usually caused by programming mistakes like memory overruns. In the early days of Windows these were very prevalent, but in more modern times these can often be caught and handled within the program. Plus modern programming systems like Java and .Net work to prevent these sorts of problems by not supporting the programming paradigms that lead to them. Even for C programs, there are better static code analysis programs and methods to test and eliminate these sorts of bugs. Generally these sorts of problems are fairly reproducible and easily diagnosed and solved in a source code debugger.

The harder to reproduce and diagnose problems usually involve running the system for a long time with a lot of users before something bad suddenly happens. The problem can’t be reproduced with just one user. It’s not known how to reproduce the problem in a shorter time. Often once we know what the problem really is, then we can construct an easier way to reproduce it, but of course now we don’t need it, on the other hand throwing in a unit test to make sure this doesn’t happen again is a good idea.

These sorts of problems usually start to occur when the system is stressed and low on resources. Sometimes the problem is that our program has a resource leak and is the cause of the system being low on resources. Another cause is multi-user. We run as a multi-threaded process on the server where the individual threads handle individual web requests. If these aren’t coded properly they could conflict and cause problems. The Java and .Net environments don’t really help with these situations and often make things harder by hiding the details of what is going on making these harder to solve. These are the problems we really need to avoid.

How to Find These Problems

How do we solve these problems? How do we test for them? We use a combination of manual and automated testing to find if we have these problems and then employ a number of tools to track what happened.

Usually every Friday we get a large number of people to participate in manual multi-user testing where everyone accesses a common server and performs transactions, either as part of functional testing that needs to be done anyway or following scripts that concentrate effort in areas we are concerned about. Then anything weird that happens is recorded for later investigation.

We also use automated testing. We extensively use Fitnesse and Selenium as I blogged about here and here. We can string all these tests together, or run them repeatedly to create long running tests that the system is stable and that nothing bad happens over time. Another very powerful tool is JMeter which performs load testing on Web Servers. Here we record the HTTP requests that the browser makes to the server as users perform their tasks, then we build these into tests that we can run for as long as we want. JMeter will then play back the HTTP requests to the server, but it can simulate any number of users doing this. This gives us a way to easily automate multi-user testing and run with far more users and for longer periods of time than manual testing allows. Below is a screen shot of JMeter with one of our tests loaded. It might be hard to read the screen shot, but notice under the Runtime item is a Pick One Randomly item. This item will pick one of several tests randomly and run it. Then we simulate a large number of users doing this for long periods of time.

 

How to Fix These Problems

The key to fixing these sorts of problems is diagnosing what is going on. Usually if you can find the correct place in the code where things are breaking down, then fixing the problem is usually fairly easy. The real trick is finding that one line of code that requires fixing. Though major refactoring’s are required now and then, and then you need to re-test to ensure you have really fixed the problem and not introduced any new problems.

At the most basic level we include a lot of logging in the product, so when we run these tests we have logging turned on and can study the logs to see what happened. This only gives clues of places to look and often we have to add more logging and then re-run the test. Logging isn’t great, but it’s very tried and true and often gets you there in the end. We use log4j for our server logging.

For memory leaks, the Java JDK gives a number of good tools to diagnose problems. The Java JDK includes the tool JConsole which you can connect to the JVM used by Tomcat and monitor the memory usage. It also records the number of classes loaded, CPU utilization and a few other interesting statistics. But one really cool feature is the ability to generate a dump file of the Java Heap at any time. From the MBean tab in JConsole you can choose com.sun.management – HotSpotDiagnostic – Operations – dumpHeap. The p0 parameter is the name of the dump file and should have an .hprof extension. By default the file will end up in the Tomcat folder. After you invoke this you get a binary dump of the Java heap, now what to do with it? Fortunately the JDK includes another tool called jhat which is used to analyze the heap and then run a web server that lets you browse the contents of the heap from a web browser. From this you can see all the Java objects in the heap, and hopefully determine what is leaking (usually one of the objects with a very high instance count). You can also see the contents of each object and who references it.

One good tool for solving these problems is ReplayDirector from Replay Solutions. This is a commercial product that records everything that happens in the JVM. You can configure your Tomcat service, so that ReplayDirector records everything that happens as you run. It doesn’t matter if you call outside of Java since it will record the call and the result. Then you can play back the recording in the regular Eclipse Java debugger as often as you like to figure out the problem. When testing, the testers can put marks in the recording to indicate where and what happened. You can set breakpoints on any REST or other HTTP call into Tomcat, you can expand log4j messages to see them even if they aren’t logged. Below is a screenshot of the main ReplayDirector Eclipse plugin. This was recorded on a server being driven by JMeter. The green line is CPU usage; the blue line is memory usage. From the looks of the blue line, you can probably guess that I’m looking for a memory leak. I can set a breakpoint at a later SData request and then break into the program and look around in the debugger to see what is going on.

Summary

Ensuring a Web Server can keep on running reliably for long periods of time can sometimes be quite a difficult task. Fortunately the tools that help with this are getting better and better, so we can monitor, record and reproduce all the weird things that testing reveals and ensure that our customers never see these sorts of things themselves.

Written by smist08

November 12, 2011 at 5:45 pm

Automated Testing in Sage ERP Accpac Development

with 7 comments

All modern products rely heavily on automated testing to maintain quality during development. Accpac follows an Agile development methodology where development happens in short three week sprints where at the end of every sprint you want the product at a shippable level of quality. This doesn’t mean you would ship, that would depend on Product Management which determines the level of new features required, but it means that quality and/or bugs wouldn’t be a factor. When performing these short development sprints, the development team wants to know about problems as quickly as possible so any problems can be resolved quickly and a backlog of bugs doesn’t accumulate.

The goal is that as soon as a developer checks in changes, these are immediately built into the full product on a build server and a sequence of automated tests are run on that build to catch any introduced problems. There are other longer running automated tests that are run less frequently to also catch problems.

To perform the continuous builds and to run much of our automated tests we use “Hudson” (http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson). Hudson is an extremely powerful and configurable system to continuously build the complete product. It knows all the project dependencies and knows what to build when things change.  Hudson builds on many other tools like Ant (similar to make or nmake) and Ivy among others to get things done (http://java-source.net/open-source/build-systems). The key thing is that it builds the complete Accpac system, not just a single module. Hudson also has the ability to track metrics so we get a graph of how the tests are running, performance metrics, lines of code metrics, etc. And this is updated on every build. Invaluable information for us to review.

The first level of automated testing are the “unit tests” (http://en.wikipedia.org/wiki/Unit_testing). These are tests that are included with the source code. They are short tests where each test should run in under 1 second. They are also self contained and don’t rely on the rest of the system being present. If other components are required, they are “mocked” with tools like easy-mock (http://easymock.org/). Mocking is a process of simulating the presence of other system components. One nice thing about “mocking” is that it makes it easy to introduce error conditions, since the mocking component can easily just return error codes. The unit tests are run as part of building each and every module. They provide a good level of confidence that a programmer hasn’t completely broken a module with the changes they are checking in to source control.

The next level of automated tests are longer running. When the Quality Assurance (QA) department first tests a new module, they write a series of test plans, the automation team takes these and transfers as many of these as possible to automated test scripts. We use Selenium (http://en.wikipedia.org/wiki/Selenium_(software)), which is a powerful scripting engine that drivers an Internet Browser simulating actual users. These tests are run over night to ensure everything is fine. We have a subset of these call the BVT (Build Validation Test) that runs against every build as a further smoke test to ensure things are ok.

All the tests so far are functional. They test whether the program is functioning properly. But further testing is required to ensure performance, scalability, reliability and multi-user are fine. We record the time taken for all the previous tests, so sometime they can detect performance problems, but they aren’t the main line of defense. We use the tool JMeter (http://jakarta.apache.org/jmeter/) to test multi-user scalability. JMeter is a powerful tool that simulates any number of client’s accessing a server. This tool tests the server by generating SData HTTP requests from a number of workstations and bombarding the server with them. A very powerful tool. For straight performance we use VBA macros that access the Accpac Business Logic Layer to write large numbers of transaction or very large transactions, which are all timed to make sure performance is fine.

We still rely heavily on manual QA testers to devise new tests and to find unique ways to break the software, but once they develop the tests we look to automate them so they can be performed over and over without boring a QA tester to death. Automated testing is a vibrant area of computer science where new techniques are always being devised. For instance we are looking at incorporating “fuzz testing” (http://en.wikipedia.org/wiki/Fuzz_testing) into our automated test suites. This technique if often associated with security testing, but it is proving quite useful for generalized testing as well. Basically fuzz testing takes the more standard automated test suites and adds variation, either by knowing how to vary input to cause problems, or just running for a long time trying all possible combinations.

As we incorporate all these testing strategies into our SDLC (Software Development Life Cycle) we hope to make Accpac more reliable and to make each release more trouble free than the previous.

Written by smist08

April 5, 2010 at 5:24 pm

Sage ERP Accpac 6 Performance Testing

with 4 comments

As Accpac moves to becoming a Web based application, we want to ensure that performance for the end user is at least as good as it was in the Accpac 5.x days running client/server with VB User Interface programs. Specifically can you have as many users running Accpac off one Web Server as you currently have running off one Citrix Server today?  In some regards Accpac 6 has an advantage over Citrix, in that the User Interface programs do not run on the server. On Citrix all the VB User Interface programs run on the Citrix server and consume a lot of memory (the VB runtime isn’t lightweight). In Accpac 6, we are only running the Business Logic (Views) on the Server and the UIs run as JavaScript programs on the client’s Browser. So we should be able to handle more users than Citrix, but we have to set the acceptance criteria somewhere. Performance is one of those things that you can never have enough of, and tends to be a journey that you keep working on to extend the reach of your product.

Web based applications can be quite complex and good tools are required to simulate heavy user loads, automate functional testing and diagnose/troubleshoot problems once they are discovered. We have to ensure that Accpac will keep running reliably under heavy user loads as users are performing quite a diverse set of activities. With Accpac 6 there are quite a few new components like the dashboard data servlets that need to have good performance and not adversely affect people performing other tasks. We want to ensure that sales people entering sales orders inside CRM are as productive as possible.

We are fundamentally using SData for all our Web Service communications. This involved a lot of building and parsing XML. How efficient is this? Fortunately all web based development systems are highly optimized for performing these functions. So far SData has been a general performance boost and the overhead of the XML has been surprisingly light.

Towards ensuring this goal we employ a number of open source or free testing tools. Some of these are used repeatedly run automated tests to ensure our performance goals are being met. Some are used to diagnose problems when they are discovered. Here is a quick list of some of the tools we are using and to what end.

Selenium (http://seleniumhq.org/) is a User Interface scripting/testing engine for performing automated testing of Web Based applications. It drives the Browser as an end user would to allow automated testing. We run a battery of tests using Selenium and as well as looking for functionality breakages, we record the times all the tests take to run to watch for performance changes.

JMeter (http://jakarta.apache.org/jmeter/) is a load testing tool for Web Based applications. It basically simulates a large number of Browsers sending HTTP requests to a server. Since SData just uses HTTP, we can use JMeter to test our SData services. This is a very effective test tool for ensuring we run under heavy multiuser loads. You just type in the number of users you want JMeter to simulate and let it go.

Fiddler2 (http://www.fiddler2.com/fiddler2/) is a Web Debugging Proxy that records all the HTTP traffic between your computer and the Internet. We can use this to record the number of network calls we make and the time each one takes. One cool thing Fiddler does is record your network performance and then calculates the time it would have taken for users in other locations or other bandwidths would have taken. So we get the time for our network, but also estimates of the time someone using DSL in California or a modem in China would have taken to load our web page.

dynaTrace AJAX Edition (http://ajax.dynatrace.com/pages/) is a great tool that monitors where all the time is going in the Browser. It tells you how much time was spent waiting for the network, executing JavaScript, processing CSS, parsing JavaScript, rendering HTML pages. A good tool to tell you where to start diving in if things are slow.

Firebug (http://getfirebug.com/) is a great general purpose profiling, measuring, debugging tool for the Firefox browser. Although our main target for release if Internet Explorer, Firebug is such a useful tool, that we find we are often drawn back to doing our testing in Firefox for the great tools like Firebug that are available as add-ins.

IE Developer Tools (http://msdn.microsoft.com/en-us/library/dd565628(VS.85).aspx) are built into IE 8 and are a very useful collection of tools for analyzing things inside IE. The JavaScript profiler is quite useful for seeing why things are slow in IE. It is necessary to use this tool, since often things that work fine in Firefox or Chrome are very slow in IE and hence you need to use an IE based tool to diagnose the problem.

Using these tools we’ve been able to successfully stress the Accpac system and to find and fix many bugs that have caused the system to crash, lockup or drastically slow down. Our automation group runs performance tests on our nightly builds and posts the results to an internal web site for all our developers to track. As we develop out all the accounting applications in the new SWT (Sage Web Toolkit), we will aggressively be developing new automated tests to ensure performance is acceptable and look to keep expanding the performance of Accpac.

Written by smist08

February 26, 2010 at 9:33 pm