Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘tomcat

Accpac 6 Online

with 6 comments

As we move to finish and release Sage ERP Accpac 6.1A, we are also working to host the Web Version of Accpac at AccpacOnline.com. I talked a bit about AccpacOnline and how it hosts Accpac currently as well as some of our plans here, this article looks a little deeper into how we will be deploying the Web version of Accpac Online, how we will be scaling it and how we will manage reliability. Of course as we develop this out, there will likely be some changes from what I am describing here.

The goal is to have an economically profitable hosted solution that can scale with our anticipated growth. Although it would be nice to assume we will be as successful as Facebook and quickly have 750,000,000 users, realistically we won’t and engineering a solution for 750,000,000 users doesn’t make sense. Like most companies you want to engineer your solution for the number of users you anticipate over the next few years and then as that materializes, you engineer for the next step.

Our initial goal is fairly conservative, and that is to host as many users on a web server as we currently host on a Citrix server. The Web server has a number of advantages over the Citrix server; namely it doesn’t run the UIs, so we save all the memory and processing by not having all those VB programs running, there is just the View portion then further the Web Server runs fewer processes and instead individual sessions are threads within a process, so this further saves resources.

Then we have the ability to scale out as many web servers as we need to accommodate our user load. As we move forward then we will work to increase the number of users that run efficiently on each server to make our solution better economically trying to scale to more users per server faster than we need to add servers.

To compare to how we deploy for on-premise installations, have a look at this article.

Below is a high level block diagram of how we will be running. This just shows the main components. I’m not covering everything, for instance this doesn’t show firewalls or other security layers that are present.

Hardware Load Balancer

At the left is a hardware load balancer. Its first job is handle encryption. All external requests come in using SSL. It is the job of this piece of hardware to decrypt all incoming communications and encrypt all outgoing communication. This offloads this onerous function from the actual Web Servers. Then its second job is to distribute incoming connections between however many Web Servers we happen to be running. Due to the way Accpac works we require that once a new connection is sent to one particular Web Server then all subsequent requests for that connection go to the same server, this is called operating with sticky sessions.

IIS

From the load balancer un-encrypted requests are routed to one of the IIS’s. If the request is for static content like bitmap images or HTML files then IIS just provides these. If the request is an SData request then it is passed onto Jakarta.

Jakarta

Jakarta is the connector which connects IIS to Tomcat. Besides connecting IIS to Tomcat, this connector has a number of very important jobs in a hosted environment. In our hosted environment we run a number of Tomcat server processes; each one running as a Windows service. Jakarta then acts as a software load balancer distributing requests across the Tomcat processes using sticky sessions again.

So why do we want to run multiple Tomcat processes and why do we want to operate a software load balancer on each server in addition to the main hardware one? The two big reasons are reliability and memory usage.

Accpac is still a 32-Bit Windows application. It runs fine on 64-Bit versions of Windows, but a 32-Bit process can only address 2Gig of RAM. As a consequence each Tomcat process can only allocate up to 2Gig of RAM and this is shared by all the threads running inside that Tomcat process. It’s easy and cheap now to get servers with far more RAM than this. So the way to utilize that RAM is to run multiple Tomcat processes each of which can utilize up to 2Gig of RAM.

The second issue is reliability. Of course we write our software to never crash and to never leak resources or do anything else harmful or bad. But the reality is that bugs happen and they have to be accounted for. If your Order Entry screen crashes, it only affects you and usually you just re-run the screen or at worst re-boot your computer. In a Web environment, something crashing on the server affects all the users on that server and re-booting a server is considered a very long time for a service to be down. Newer versions of Windows Server have gotten very good at isolating problems to an individual process, so when that process crashes, Windows can recover all its resources and happily continue working. So if a thread inside Tomcat (representing one user) crashes then the Tomcat process will crash and all the users using that Tomcat will be affected. Jakarta monitors this and if a Tomcat process crashes it will restart it. It also notes when a Tomcat process isn’t responding and will stop directing new requests to that Tomcat. Additionally you can configure Jakarta to stop using a Tomcat after some period of time (usually some number of days) and then when everyone is logged off it, restart it to recover any leaked resources. Generally the overall health and availability of the Web Server is being monitored and controlled by Jakarta. Of course we do extensive testing so these sort of events are infrequent, but they still need to be handled and recovered from.

Tomcat

Tomcat is a Java application server that runs our Java program that interprets SData requests. Then this Java application will call into the Accpac Business Logic (Views) to do the actual processing. Tomcat runs as a multi-threaded program in which each new request is handled from a pool of worker threads which do their job and then return to the pool. All the classic Accpac Business Logic then runs in the context of this program. As far as Tomcat can tell at this point, it is running as if it was on-premise and it doesn’t know about the other entire infrastructure that is going on around it.

SQL Server

Then off to the right of the diagram we have our pool of SQL Servers. Basically an individual customer’s company will exist on one specific one of these. Basically we manage these in exactly the same way we do for the current Accpac Online. In fact for a given customer, some users could be accessing the SQL Server through all the items in this blog and other users could be accessing it via Citrix and our current VB screens. All co-existing and sharing nicely.

Summary

This blog is just meant to give an idea of how we are setting up AccpacOnline.com for running the web version of Sage ERP Accpac, hopefully you found it interesting.

Written by smist08

September 17, 2011 at 11:33 pm

Posted in sage 300

Tagged with , , , , , ,

Diagnosing Problems with Sage ERP Accpac 6.0A

with 24 comments

Now that people are installing and deploying Accpac 6, hopefully, to get the new Web technologies working you just need to install, play with Database Setup and then away you go with no problems. Or at least hopefully this is how it works out for most people. But for those that it doesn’t, this blog posting is to help you figure out what has gone wrong. The main purpose of this blog posting is to help you track down and solve unforeseen problems on your own, hopefully giving you tools that will give you clues to what is wrong and allow you to come up with solutions.

This blog posting assumes that you can run both regular Database Setup and the classic Accpac Desktop. If you are having trouble at this step then check out these blog postings on diagnosing problems in Accpac part 1, part 2, part 3 and part 4.

Database Setup

The first time you hit some of the new technologies is when you configure the Portal’s database from Database Setup’s Portal… button. When you click here it runs a Java program that configures the portal for JDBC and tests the connection. Often if you are going to have trouble with the Java runtime or with accessing the database server from Java, this is where you will run into problems.

Bad Java Runtime

The Accpac 6 installation installs the Java Runtime version 6 update 17. If you previously have a later version of the Java runtime or let the Java Updater run then you will have a newer version. If you get an error about having trouble with a Java class such as com.sage.orion.connection.services.CommandLine, then chances are you have a problem with the runtime. We’ve seen systems where many versions of the Java runtime are installed and even though they are supposed to co-exist, there appear to be problems, plus now and then a bad Java update gets installed from the Java Updater. This was especially a problem shortly after Oracle bought Sun, at which point Oracle renamed quite a few things from Sun to Oracle and put out a couple of bad updates. They seem to be getting better at this again, but this is something to watch out for. A good remedy for this is to un-install all versions of the Java runtime and delete the program files\java directories (or the x86 variant if running 64 bit). Then either re-install Accpac or install a good known version of Java directly. Remember that Accpac is a 32 bit application and as such we require the 32 bit version of Java.

Troubles Connecting to SQL Server from Java

If your Java runtime is ok, then the next thing is: the Java classes will test your settings by connecting to the Portal database. If you get an error about “can’t locate server or user id/password are incorrect”, it might be a SQL Server configuration problem. The configuration of SQL Server from Database Setup for normal Accpac companies is fairly forgiving; it is quite smart about finding SQL Server’s and SQL Server instances and can use any communications protocol that is configured. But we access the Portal database using JDBC which is a bit more stringent. If you can setup regular Accpac companies, but not the Portal database then it is usually due to a few SQL Server configuration properties:

  1. In the SQL Server Configuration Manager – SQL Server Network Configuration – Protocols for serverinstance: make sure TCP/IP is enabled (usually it isn’t for SQL Server express and a few other varieties). Make note of the Port number used to ensure you are using the correct one (right click on TCP/IP and choose properties).
  2. Use the servername alone and the correct port number. Do NOT use server/instancename, this isn’t currently recognized (we will probably fix this for Product Update 1).
  3. In SQL Server Management Studio: in the properties for the server, security tab, ensure “SQL Server and Windows authentication mode” is selected for Server authentication.
  4. In SQL Server Management Studio: in the properties for the server, connections tab, make sure “Allow remote connections to this server” is checked.

If none of this works, then you are going to have to run a SQL Trace to see what is failing.

Hopefully this helps you get past Database Setup.

Tracing Web Requests

Below is a deployment diagram of the new Web components included with Sage ERP Accpac 6.0A. For more details look here.

As you can see requests come from the Internet and enter the Web Server via Microsoft Internet Information Services (IIS). Static content like HTML files and bitmap images will be served up directly from here. Requests for data are sent through the Jakarta ISAPI re-director to Tomcat which runs our SDataServlet which in turn calls the Accpac business logic objects (Views) to get the actual accounting data.

The general approach to troubleshooting here is to use logging to figure out how far requests from the Browser are getting and then to see why are failing at that point. All the components in the above deployment diagram have logging capabilities. Usually by default they only log errors which at this point are often all you need, however they can all be configured to provide much more detail on everything that is going on in the system.

Logging

First I’ll quickly summarize how to turn up and control the logging levels for the various services.  When running normally you won’t want to leave all these logging levels high since it wastes a lot of disk space and causes a performance drag writing out all this information. Usually you want to leave IIS logging on for every request so you can audit and keep an eye out for malicious attacks. But the other logs may as well be left at the error level.

IIS’s logging is controlled from the IIS Manager via the Logging icon for the web server. By default it’s enabled and the logs are stored in: C:\Inetpub\logs\LogFiles\W3SVC1. Note that the various versions of IIS are different so you might need to look around a bit for the settings.

The Jakarta IIS redirector is configured via the file: C:\Program Files (x86)\Common Files\Sage\Sage Accpac\Tomcat6\Jakarta\isapi_redirect.properties. Change the log_level from error to debug. Note that this setting causes an extremely large amount of output and will affect server performance, so remember to set this back to error when you are done. The log file is AccpacRedirector.log in the same C:\Program Files (x86)\Common Files\Sage\Sage Accpac\Tomcat6\Jakarta folder.

For Tomcat and SDataServlet, the log files are stored in: C:\Program Files (x86)\Common Files\Sage\Sage Accpac\Tomcat6\logs . To configure the logging level edit: C:\Program Files (x86)\Common Files\Sage\Sage Accpac\Tomcat6\webapps\SDataServlet\WEB-INF\classes\log4j.properties and change “log4j.rootLogger=ERROR, logfile” to “log4j.rootLogger=ALL, logfile”.

Diagnosing Problems

The general approach is to follow messages through the system by looking at the log files. Having a set of log files from a working system can help greatly since then you can compare them, since differences often give a clue to the cause of problems.

IIS

First, do the Browser requests even get to IIS? If not try using the server’s IP address instead of the URL. Try Browsing from the server using localhost. This can give an indication if IIS just isn’t working at all or you have DNS problems or there is some problem resolving your URL. Obvious things to check are that the IIS service is running, check the Windows Event Log if it won’t start or that IIS is configured on port 80 (HTTP) and 443 (HTTPS). Note that only one program can use a given port so if something else is using port 80, say Apache as a web server for something else then only one of them can have port 80 and one process will fail to start. If you have to use a different port, this is ok, just remember to put :portnumber after the servername in your URLs.

Next, what if the browser requests get to IIS but are rejected? What if you get a 403 error return (access forbidden)? This also manifests itself in that you will see “isapi_redirect.dll” as part of URLs in the log file. This means you don’t have sufficient permissions to execute the Jakarta redirector or there is some other sort of access problem. If you get general 403 errors on requests for static content like HTML files or image files, then the file permissions on the WebUIs folder under the Accpac directory are too strong. This folder only contains HTML, JavaScript and image files. Neither data, nor anything valuable is stored in this folder, so it is ok to relax the permissions and grant read-only access for everyone (don’t grant write since you certainly don’t want to risk a hacker writing files here). The other case is if you see isap_redirect.dll in the URL, this means you don’t have sufficient permissions to load this DLL. We install IIS with all our processes running as the local system account, this account should have permission to do anything on the machine, but it won’t have access to network resources so the most common cause of this is having key files installed on a network share. The best solution here is to ensure the Accpac programs are installed on the web server so it only needs to access the LAN to get to the database server. You can also change the user the Sage Accpac Application pool runs under to something with domain privileges.

Another problem we’ve seen with IIS is that in some cases the Accpac install fails to create the various virtual directories and application pools inside IIS. We haven’t figured out why this happens yet, but it’s worth quickly checking that the SageERPAccpac and SDataServlet applications are added to the default web site and that Sage ERP Accpac App Pool is added to the application pools. Right now the only solution we have for this is to create these entries manually, copying from a good system. I’ll update this blog posting with more details once we figure this out. It’s strange because we don’t get any errors back from IIS when this happens.

Jakarta

So far we haven’t seen problems with Jakarta. This is an ISAPI module that loads into IIS and redirects all the Accpac SData requests to the Tomcat Server. I would first look in Tomcat’s logs and then only come back here if nothing is appearing in the Tomcat logs.

Generally here you need to turn on the logging and then look to ensure messages are making it to Jakarta (i.e. the log isn’t empty). Then usually Jakarta provides fairly good diagnostics in its log when things don’t work. These are usually along the lines that the Tomcat service isn’t running, there is a port conflict or something of this nature.

Tomcat/SDataServlet

Apache Tomcat is a Java application server that runs the server side of Java web applications. The Accpac SData server is written in Java and runs under this. Tomcat’s own log files give good information if there are problems with Tomcat. Tomcat can have problems if a bad version of the Java runtime is installed, hopefully this was resolved back when you ran Database Setup, but if you get a bunch of weird Java errors in the Tomcat logs then check out the info in the Database Setup section above on cleaning up the Java runtimes.

Tomcat is also the point where we need to load DLLs from the Accpac\runtime directory. If you see errors about loading Java classes and you see a4wapiShim.dll mentioned, then you probably don’t have the Accpac\runtime folder in your system PATH. Note that since Tomcat runs as a system service under Windows, this must be in the system PATH and not your user PATH.

Next when you look in the SDataServlet log files you will see various Accpac error messages. Make sure you can login to the standard Windows desktop and access the company you are using. You can’t perform data activation from the web portal, so this needs to be done from the Windows desktop.

If you installed an alpha, beta or release candidate of Accpac 6, try uninstalling, deleting the C:\Program Files (x86)\Common Files\Sage\Sage Accpac\Tomcat6 folder and then re-installing. There might be some old leftover files that need deleting.

Watch out for IE6

We see people trying to use IE6 a surprising amount. It’s always surprising to me that people are still using IE 6, but they are. If the reported symptom is that the browser is hanging, then it is probably IE6. We don’t support IE6 and IE6 definitely doesn’t work in anyway. Worse the IE6 JavaScript engine can completely lock up Windows causing you to need to re-boot your workstation. The usual symptom of IE6 is that the first time you hit the portal it creates the uiContent table and then locks up, not creating the other three tables in the PORTAL database.

Remember we only officially support IE 7 and 8 for Sage ERP Accpac 6.0A. We will add support for Safari, Chrome and Firefox in Sage ERP Accpac 6.1A. However we won’t ever be adding support for IE 6. We will validate IE 9 once it’s released. Also if you are running IE 7, although it works, why not upgrade to IE 8? It does work quite a bit better.

Quotes to Orders

Diagnosing problems with Quotes to Orders is done the same as above. But just a couple of notes:

  • Ensure you use .Net remoting as the protocol from the Web Deployment Manager.
  • Make sure that you have activated and setup the SageCRM (EW) module within Accpac.

Fiddler

Fiddler is a great tool for tracking down problems. It spies on TCP/IP traffic through IE. Note that you have to use the proper server name in the URL, you can’t use localhost. This will show you all the requests and responses made from the Browser, often giving important clues when solving problems.

Summary

I hope as you deploy Sage ERP Accpac 6.0A, that you don’t need to refer to this blog posting very often. Further I hope that this blog posting doesn’t need to be read much. But if you do run into problems, I hope you find this information helpful.

Now off to TPAC.

Update 2011/03/29: If you have an individual workstation that is having trouble loading the Portal, try clearing the Browser cache. Sometimes if the Portal failed to load half way through then IE caches that and won’t finish until the cache is cleared and it can start clean.

Update 2014/10/02: For SQL Server Express, by default it uses dynamic ports. To work with the Portal you need to configure it to use a specific port. To do this run the SQL Server Configuration Manager. Choose “SQL Server Network Configuration”. First ensure TCP/IP is enabled, then right click on it and choose properties. Choose the IP Addresses tab, scroll to the bottom and in the IPAall setting, blank out the TCP Dynamic Ports entry and set the TCP Port to 1433 (or whatever you need).

Written by smist08

March 5, 2011 at 5:24 pm

Posted in sage 300

Tagged with , , , ,

Installing and Deploying Sage ERP Accpac 6.0A

with 23 comments

We’re only a month or two away from the launch of Sage ERP Accpac 6.0A (for details see: https://smist08.wordpress.com/2010/09/25/preparing-for-the-sage-erp-accpac-6-0a-launch/). A major goal of this release is to provide a smooth upgrade transition from 5.x to 6.x. There has been some concern that the new Web based components will make the product harder to install or increase the hardware requirements of the product. This blog posting will go into some detail of the parts that are the same, the parts that are new, what is optional and what you need to watch out for. We’ve put a lot of effort into the Accpac installation program to make the process as transparent and simple as possible. But sometime it’s helpful to know what’s going on behind the scenes and what is being setup for you.

Accpac Classic

If you don’t want to worry about the new Web technologies, you don’t have to. From Accpac’s installation screen you can de-select the “Portal” feature and install just like you always have.

Of course you can always add the Portal feature at any later time.

Accpac Web Portal

Whether you are doing a Workstation Setup, Terminal Server or Local programs installation, you only want to install the portal in one place on the server that will be your IIS Web Server. On all other computers if doing a full installation, be sure to un-check the Portal as an installation option. If you want to run VB UIs from the new Portal, you do need to have the ability to run the classic Windows Desktop via a workstation setup or local install.

Hopefully you will want to use the new Web Base Portal with all the new functionality available there. If you have worked with SageCRM 7.0 then much of the deployment and installation is the same. We both use a combination of IIS and Apache Tomcat to process Web Requests. In fact, when we wrote our installation script, we copied all the sections for Tomcat, Jakarta and IIS from the SageCRM installation script and just modified them for Accpac. So much of the installation is actually field tested since it has been used for many SageCRM installations.

Below is a diagram showing the main components involved in a Web Portal deployment.

IIS

If you are installing the new Web Based Portal or using Quotes to Orders from SageCRM, you need a Web Server. Hopefully you already have such a server being used as a file server or as a SageCRM server. You will install the Accpac Portal here. On this server computer you need to install IIS before you install Accpac. Since IIS is a Windows component, we can’t do that for you from our installation program. You need to either add this as a role to your server or add the Windows component from the control panel. Note that this excludes you running the new Portal on a laptop running Windows Home edition, since it does not have IIS as an option.

You don’t need to do anything beyond adding IIS, our installation program will add all our components to IIS and will configure IIS to work with Accpac. This is true if you are running internally, if you expose this Web server to the Internet or you want better internal security then you have to do a small amount of work to configure the Web Server for HTTPS and add a digital certificate to correctly identify the server to clients.

Below are the virtual directories and application pools added by the Accpac and SageCRM installation programs to IIS.

Tomcat

Our new SData services that process the various SData Web Service requests are programs written in Java (https://smist08.wordpress.com/2009/11/24/sdata-in-sage-erp-accpac-6/). To run this Java program we require a Java Application Server, which is a program that receives web requests and passes them to Java classes to be processed. The Java application server that we use is Apache Tomcat (http://tomcat.apache.org/). Behind the scenes our installation program installs Tomcat in C:\Program Files (x86)\Common Files\Sage\Sage Accpac\Tomcat6 and adds it as a Windows Service. Hopefully you never need to know this program is even here. Hopefully it is all setup perfectly and you will never need to touch it.

Below is the Windows Services dialog showing our usual service for handling .Net Remoting requests and our the new service that is the Accpac instance of Apache Tomcat.

Jakarta

Our installation program adds another component called Jakarta to connect IIS up to Tomcat. All the installation and configuration of this component is done automatically by the installation program and you don’t even have to know it is there. Basically we want IIS to serve up all static web content like HTML pages, JavaScript pages, cascading style sheets and various bitmaps. But we want any SData requests passed on to the Apache Tomcat Java application server running the Accpac SData processor.

Portal Database Configuration

The new Accpac Web Portal has its own database that it uses to store various information like you shortcuts and preferences. So you need to configure this database from Database Setup which now has a “Portal…” button which leads to the following dialog.

HTTPS

If you want to expose your Accpac Portal to the Internet or just want extra internal security then you should configure IIS to use HTTPS rather than HTTP. To do this you add an https binding to the default web site bindings. When doing this it wants a digital certificate to identify the server. You need to get this issued from Verisign (www.VeriSign.com)  or another vendor. Otherwise the Browser will complain about accessing this site, since it won’t be able to prove who it’s talking to. Additionally you will then want to disable HTTP so access only happens through a secure connection. Then you would follow all the other security best practices for managing a Windows Server (like be behind a firewall and have all unused services turned off). Often it’s worth running a tool like NMap/Zenmap (http://nmap.org/zenmap/) to confirm that you aren’t running anything unexpected that could be hacked.

Summary

Hopefully this gives some idea of the various issues and considerations installing Sage ERP Accpac 6.0A. If you are familiar with SageCRM 7.0 then this should all be old hat. Otherwise give it a try; it’s not nearly as hard as it might look.

Written by smist08

October 2, 2010 at 4:15 pm