Stephen Smith's Blog

Musings on Machine Learning…

Archive for the ‘Software Development Lifecycle’ Category

Performance Testing in Swift

with one comment

Introduction

A couple of blog posts ago I covered writing my first Swift program for iOS so that I could draw a Koch Snowflake on an iPad or an iPhone. Then last time I covered adding unit tests to that project. This time I’m going to add performance tests.

In the process of adding performance tests, I had to refactor the test project and we’ll also look at why that was and how it makes it better going forwards as more tests are added. I’ll also mention a few things that should be done if this project gets a bit bigger.

I put an updated version of the Koch Snowflake project on Google Drive here.

Performance Tests in XCode

Of course you could instrument your program yourself and perhaps write the performance results out to a file or something, for that matter you can drill down into the Swift test case class and have a look at their implementation. But XCode gives you a bit of support so you generally don’t need to. If you add self.measureBlock {} around code in a unit test then the time taken of the code inside the measureBlock will be recorded and reported inside XCode as shown in the following screenshot.

Screen Shot 2016-06-01 at 3.32.53 PM

Actually it does a bit more than that. When you add measureBlock to a unit test, then when you run that unit test, it won’t just be run once, but will be run ten times, so that the average and standard deviation will be recorded. Due to this it is crucial that any performance tests are idempotent. You can also set a baseline, so the percentage deviation from the baseline gets recorded. This is shown in the following screenshot that is a drill down from the previous screen shot.

Screen Shot 2016-06-01 at 3.33.04 PM

Hence XCode gives a fairly painless way to add some performance metrics to your unit tests.

Test Case Organization

Generally, you want your unit tests to run against every build or your product, so you want then to run in a second or two. Once the performance tests get longer, you will probably want to separate them off into a separate test group and then run this test group perhaps once over night. I haven’t done that, but if the project gets any bigger then I will.

In fact, the test framework inside XCode is quite good for performing integration tests (which would run against real databases and real servers), but since these may require some setup or be quite time consuming, you could also set these to run once per night.

There is also a separate framework for UI testing, which again is too slow to run against every build, but makes sense to run every night.

Refactoring the Unit Tests

For the performance test, I wanted to record the time it takes to draw the Koch Snowflake at various fractal levels. To do this I wanted to do something like the previous testInitialViewController routine, but it contained a lot of setup code. So first the unit test framework includes setUp function that is called before the unit tests are run and a tearDown routine that is called after then finished. So I moved the creating of the graphic context to this routine, along with the code to get the view controller started. Then it was fairly easy to add tests for fractal levels 3 through 7.

Last time I just had 2 unit tests, each was quite large and performed multiple things. Now we’ve split things up into more unit tests that do less, which is generally a better practice. This was actually forced on me since you can only have one measureBlock in any unit test, so I couldn’t performance test the different fractal levels in the same unit test (at least with separate timings). Really I should break up the turtle graphics tests into multiple unit tests, perhaps next time.

The reason I went all the way to fractal level 7, was that the performance reports in XCode are often 2 decimal places (or sometimes 3 decimal places) on the number of seconds the test takes. For my fractal, the drawing is quite quick so I needed go this high to get some longer test times recorded (kind of a good problem to have). I could have gone higher or put them in an additional loop, but thought this was sufficient.

//
//  KochSnowFlakeTests.swift
//  KochSnowFlakeTests
//
//  Created by Stephen Smith on 2016-05-13.
//  Copyright © 2016 Stephen Smith. All rights reserved.
//

import XCTest

@testable import KochSnowFlake

class KochSnowFlakeTests: XCTestCase {
    var storyboard:UIStoryboard!
    var viewController:ViewController!

    override func setUp() {
        super.setUp()
        // Put setup code here. This method is called before the invocation of each test method in the class.

        UIGraphicsBeginImageContextWithOptions(CGSize(width: 50, height: 50), false, 20);

        self.storyboard = UIStoryboard(name: "Main", bundle: nil)

        self.viewController = storyboard.instantiateInitialViewController() as! ViewController
        _ = viewController.view
        viewController.viewDidLoad()
    }

    override func tearDown() {
        // Put teardown code here. This method is called after the invocation of each test method in the class.
        UIGraphicsEndImageContext();
        super.tearDown()
    }

    func testTurtleGraphics() {
        // Test the turtle graphics library.
        // Note we need a valid graphics context to do this.

        let context = UIGraphicsGetCurrentContext();
        let tg = TurtleGraphics(inContext: context!);
        XCTAssert(tg.x == 50, "Initial X value should be 50");
        XCTAssertEqual(tg.y, 150, "Initial Y value should be 150");
        XCTAssertEqual(tg.angle, 0, "Initial angle should be 0");
        tg.move(10);
        XCTAssertEqual(tg.x, 60, "X should be incremented to 60");
        XCTAssertEqual(tg.y, 150, "Initial Y value should be 150");
        XCTAssertEqual(tg.angle, 0, "Initial angle should be 0");
        tg.turn(90);
        tg.move(10);
        XCTAssertEqualWithAccuracy(tg.x, 60, accuracy: 0.0001, "X should be o 60");
        XCTAssertEqualWithAccuracy(tg.y, 160, accuracy: 0.0001, "Initial Y value should be 160");
        XCTAssertEqual(tg.angle, 90, "Initial angle should be 90");
        tg.turn(-45);
        tg.move(10);
        XCTAssertEqualWithAccuracy(tg.x, 60 + 10 * sqrt(2) / 2, accuracy: 0.0001, "X should be o 60+10*sqrt(2)/2");
        XCTAssertEqualWithAccuracy(tg.y, 160 + 10 * sqrt(2) / 2, accuracy: 0.0001, "Initial Y value should be 160+10*sqrt(2)/2");
        XCTAssertEqual(tg.angle, 45, "Initial angle should be 45");
    }

    func testPerformanceLevel3()
    {
        // Test that the storyboard is connected to the view controller and
        // that we can create and use the view and controls.

        viewController.fractalLevelTextField.text = "3"
        self.measureBlock {
            self.viewController.textChangeNot("dummy")
            self.viewController.fracView.drawRect(CGRect(x:0, y:0, width: 50, height: 50))
        }

        XCTAssertTrue(viewController.fracView.level == 3)
        // This next line is just to get 100% code coverage.
        viewController.didReceiveMemoryWarning()
    }

    func testPerformanceLevel4() {
        // This is an example of a performance test case.
        viewController.fractalLevelTextField.text = "4"
        self.measureBlock {
            self.viewController.textChangeNot("dummy")
            self.viewController.fracView.drawRect(CGRect(x:0, y:0, width: 50, height: 50))
        }
    }

    func testPerformanceLevel5() {
        // This is an example of a performance test case.
        viewController.fractalLevelTextField.text = "5"
        self.measureBlock {
            self.viewController.textChangeNot("dummy")
            self.viewController.fracView.drawRect(CGRect(x:0, y:0, width: 50, height: 50))
        }
    }

    func testPerformanceLevel6() {
        // This is an example of a performance test case.
        viewController.fractalLevelTextField.text = "6"
        self.measureBlock {
            self.viewController.textChangeNot("dummy")
            self.viewController.fracView.drawRect(CGRect(x:0, y:0, width: 50, height: 50))
        }
    }

    func testPerformanceLevel7() {
        // This is an example of a performance test case.
        viewController.fractalLevelTextField.text = "7"
        self.measureBlock {
            self.viewController.textChangeNot("dummy")
            self.viewController.fracView.drawRect(CGRect(x:0, y:0, width: 50, height: 50))
        }
    }
}

 

Summary

I found adding performance tests to my fractal iOS application quite easy. XCode gives quite nice support to perform these tests painlessly, hopefully motivating more programmers to include them.

At this point I’m not going to optimize the code as it is running fast enough. But if I ever take on drawing more sophisticated or complicated fractals, then drawing speed becomes really important. Some things to consider would be how efficient in the recursive algorithm used, and whether I’m efficiently using floating point and integer arithmetic (or are there unnecessary conversions or perhaps too much precision being used).

Written by smist08

June 7, 2016 at 2:15 am

Adding Unit Tests to My Swift Application

with 3 comments

Introduction

Last time I blogged on creating my first Swift application that would display a Koch Snowflake fractal on an iPad or and iPhone. This time I’m going to look at adding some unit tests to that program to see how that works in XCode and Swift. It appears that Apple has built some nice unit testing support into XCode so doing this was actually quite easy.

I put the Koch Snowflake Swift project on Google Drive here. Feel free to download it and play with it. Refer to this project or the previous blog post for the code we are actually testing in this article.

Test Driven Development

If I was following proper Test Driven Development (TDD) then I should have written the unit tests before I wrote the actual code. This would force me to think about the design of the APIs and the testability of the program first, and then would force quick cycles of writing tests, writing code so the tests pass and then refactoring the code to make it all better. However, in this case I ported an old Objective-C program to Swift to play with Swift and now that I’m getting more familiar with Swift, I’m finally looking at the unit testing framework. So what I’m doing is perhaps not a good practice, but adding the unit tests later is better than not adding them at all.

The danger I could have run into is that the program could have turned out to be quite hard to test due to perhaps some bad design decisions and then required quite a bit of refactoring to make it properly unit testable. Fortunately, in this case the program is fairly simple and adding a good set of unit tests was fairly straight forward and didn’t require any changes to the program.

One criticism of TDD is that there is a perception that it interferes with good program design and architecture. This isn’t true. The misconception is that the first tests need to be truly useful, whereas the first tests usually just call the API and set how the API to the various classes is used. The classes are just skeletons and the unit tests pass as long as everything compiles. Getting everything to compile is the first step and thinking about all the APIs before writing the real code that does stuff is the first step to getting a good modular design.

Apple has adopted TDD for a lot of their own projects, for instance their Core Data module was developed this way. Since Apple developers use XCode and now many are using TDD this means that a lot of good support for TDD is being baked into XCode. It also means that newer Apple technologies are much easier to use in a TDD environment than some of their older ones. But even for the older ones there are lots of clever examples on the web of how to work around the various challenges they introduce.

Screen Shot 2016-05-31 at 10.41.21 AM

Testing Turtle Graphics

When you create a project with XCode you have the option to include unit tests (which I did) and this gives you a test project that you can run with some dummy tests in it. So I figured first let’s add some unit tests to my TurtleGraphics class. This is pretty simple you move the turtle and turn the turtle and it leaves a trail behind it. For drawing fractals turtle graphics makes the algorithms very easy since it fits in well with the recursive algorithms typically used.

The turtle graphics class uses Apple’s Quartz2D library to draw on the screen. Perhaps the first possible design flaw is that when I instantiate the turtle graphics library I need to pass its constructor a valid graphics context. The underlying Quartz2D library routines certainly don’t like this being nil and throw an exception if you try that (ie my first try). So I thought I would just create a graphic context object but after battling with Swift a bit, I found I couldn’t do this because the underlying class doesn’t have any constructors. This also prevented me from mocking the class (at least using the real class and replacing just a few methods). This is because Apple wants to make sure you get a valid graphics context from one of their factory methods. If you are actually in the drawing routine this is passed to you by the system, but as a unit test, we aren’t running in a screen drawing context. Fortunately there is a way to get a graphics context to draw on an in-memory image file, so I used that.

To solve this, I could have refactored the drawing routines into another class and had this class accept nil for the graphics context and just return when this happens. This is basically just making a new class that pass through to the real class when the program runs, or acts as a stub class when running unit tests. I didn’t like this so much since then if the real program did pass nil, it wouldn’t get caught and the program just wouldn’t work properly. Even though I’m not testing the actual placement of pixels on the screen, it makes me feel good to know the actual real drawing routines are being exercised during the unit tests, so if they do get an invalid argument or something the problem will be caught.

The unit test framework lets unit tests get at variables in a class to help you set up the asserts to prove correctness (you import the module via @testable import KochSnowFlake rather than just import), so to test it, I just did some simple move and turns and then tested that the internal state of the turtle graphics module was correct. Notice when I turned 45 degrees I used trigonometry to calculate the new position, for testing its usually good to test against a calculation, rather than running the program to see what the value is and then using that. Notice the use of XCTAssertEqualWithAccuracy, since we are using floating point arithmetic, rounding errors creep in, so the equals will never be exact. You can configure XCode to either stop on each error or run to completion just recording all the errors, its up to you. For this I usually just stopped on an error and then fixed the problem.

func testTurtleGraphics() {
 // Test the turtle graphics library.
 // Note we need a valid graphics context to do this.
 
 UIGraphicsBeginImageContextWithOptions(CGSize(width: 50, height: 50), false, 20);
 let context = UIGraphicsGetCurrentContext();
 let tg = TurtleGraphics(inContext: context!);
 XCTAssert(tg.x == 50, "Initial X value should be 50");
 XCTAssertEqual(tg.y, 150, "Initial Y value should be 150");
 XCTAssertEqual(tg.angle, 0, "Initial angle should be 0");
 tg.move(10);
 XCTAssertEqual(tg.x, 60, "X should be incremented to 60");
 XCTAssertEqual(tg.y, 150, "Initial Y value should be 150");
 XCTAssertEqual(tg.angle, 0, "Initial angle should be 0");
 tg.turn(90);
 tg.move(10);
 XCTAssertEqualWithAccuracy(tg.x, 60, accuracy: 0.0001, "X should be o 60");
 XCTAssertEqualWithAccuracy(tg.y, 160, accuracy: 0.0001, "Initial Y value should be 160");
 XCTAssertEqual(tg.angle, 90, "Initial angle should be 90");
 tg.turn(-45);
 tg.move(10);
 XCTAssertEqualWithAccuracy(tg.x, 60 + 10 * sqrt(2) / 2, accuracy: 0.0001, "X should be o 60+10*sqrt(2)/2");
 XCTAssertEqualWithAccuracy(tg.y, 160 + 10 * sqrt(2) / 2, accuracy: 0.0001, "Initial Y value should be 160+10*sqrt(2)/2");
 XCTAssertEqual(tg.angle, 45, "Initial angle should be 45");
 
 UIGraphicsEndImageContext();
 
 }

Testing the View Controller

Now let’s test things at a higher level. Generally, we don’t do UI testing in unit tests, but we can do some basic tests to ensure everything is created properly and is more or less hooked up correctly. XCode does have built in support for UI testing and perhaps we’ll have a look at that in a later blog post. In this case we are going to create the storyboard (where the UI is defined) and from that instantiate our View Controller. Then we do a dummy access for the view to get the delay loaded elements loaded and start calling the View Controllers methods starting with viewDidLoad(). For our test we set the fractal level to 3 in the text box and then call the notification methods and see if it then updated properly in our View.

func testInitialViewController()
 {
 // Test that the storyboard is connected to the view controller and
 // that we can create and use the view and controls.
 
 let storyboard = UIStoryboard(name: "Main", bundle: nil)
 
 let viewController = storyboard.instantiateInitialViewController() as! ViewController
 _ = viewController.view
 viewController.viewDidLoad()
 
 viewController.fractalLevelTextField.text = "3"
 viewController.textChangeNot("dummy")
 XCTAssertTrue(viewController.fracView.level == 3)
 // This next line is just to get 100% code coverage.
 viewController.didReceiveMemoryWarning()
 }

Test Coverage

The big questions that is usually asked about unit tests, is how much code to they cover (or exercise). XCode can give this report which you can see in the screenshot:

Screen Shot 2016-05-31 at 10.42.53 AM

Amazingly we nearly get 100% coverage. The only module with poor coverage is AppDelegate.swift which is code generated by XCode and then I never use it. The reason we get such good coverage is that this is a fairly simple program and it draws the snowflake when it initializes. But the key point here is that I am testing a lot of things fairly easily and fairly quickly. It tests the functionality of the turtle graphics library and it tests that the UI is all hooked up properly and working.

A true TDD adherent would delete all the code in AppDelegate.swift, since code shouldn’t be written unless it is to make a unit test pass. But I tend to give an exception to code generated by the tools like XCode, to me this code is Apple’s responsibility and not mine. Plus, if I ever do add code to this class then I would need to start adding unit tests and take it a bit more seriously.

Harder Cases

This program is pretty simple, so I didn’t need to do anything too fancy to unit test it. This is mainly because there is no database access, no hardware access (beyond drawing and one text field) and no communication with a server. If you add these then you would need to add mock, stub and/or fake classes to assist in testing. You want your unit tests to be fast and run on every compile. You don’t want to require databases be setup or that servers are running somewhere in a certain state. Fortunately, Swift is a very powerful object oriented language and creating all of these is fairly easy (extensions are especially helpful). The main problems you run into are in using the Objective-C libraries like UIKit, but again you aren’t the first to run into these and there are tons of sample code, blogs and lessons on how to deal with these.

Another tricky area is with asynchronous calls. When you make a call to the server, the response will be returned asynchronously. What this means that as soon as your unit test sends the server request, it will complete. Since the asynchronous request comes back on a separate thread later, you need to ensure your main thread doesn’t just end and you wait for the results, or the tests in the callback will never be executed. There are lots of examples on how to do this, but it definitely adds some complexity to your tests.

Continuous Build Server

If you have a team, then you will probably want a continuous build server that monitors your source code repository (probably Git) and then does a build and test each time something new is checked in. Apple has an XCode Server that you can purchase if you are a registered developer. Or you could use the Jenkins Build Server which is open source and free. Jenkins also has an XCode plugin to make life a bit easier. Though XCode is quite good at letting you control it from the command line. This way you can be notified as soon as something breaks and ensure problems get fixed right when they are caused, which is the easiest time to fix them.

Summary

That turned out to be quite a long posting on unit tests. XCode and Swift have a very powerful and easy to work with unit testing framework built in. If you are developing a new application, certainly consider using TDD for your development methodology. If you are porting or doing a new version of an app, then certainly try adding some unit tests. Unit tests will help you greatly over the lifetime of your product, making it more reliable, bug free and easier to maintain.

Written by smist08

May 31, 2016 at 8:17 pm

Accessing Sage 300c’s Business Logic from the Web UIs

with 5 comments

Introduction

In the Sage 300 VB UIs, a user would do something in the UI (press a button or tab out of a field) and then the VB UI would be notified of this and would possibly execute a number of Sage 300 Business Logic (View) calls and based on their results update various other fields and possibly provide user feedback via a message box.

In the Web UIs we want to do similar processing since we want to re-use the tried and true Sage 300 Business Logic, but we have to be careful since now the Web UI is half running as JavaScript in the Browser and half running as .Net assemblies on the server. We have to be careful of the communication between the Browser and the server since there will be quite a bit of latency in each call over the Internet. Generally, we never want one user action to result in more than one call to the server (and ideally most user actions shouldn’t result in any calls to the server).

This blog post talks about where you put your code to access the Sage 300 Business Logic and how a UI interaction in the Browser flows through the system to execute this business logic.

Architecture

In the new Web UI architecture, we access the Sage 300 Business logic from our Business Repository classes. The base classes for these provide a wrapper of the Sage 300 .Net API to actually access the Views, but hiding the details of things like session and database link management. Then above this layer are the usual ASP.Net MVC Models and Controllers.

cna2arch

The Sage 300c Web UI Architecture

Generally, we want to put all this logic in the Business Repository so it can be used by multiple higher level clients including the Web UIs, our new RESTful WebAPI and services which are available for other applications to utilize.

Some of the layering is in place ready for additional functionality like customization. We need provide the common interfaces that can act as the basis for programmatic customization by inserting custom modules into the processing flow via Unity Interception.

Moving VB Code

In VB we often make lots of Business Logic (View) calls all interspersed with lots of interactions with various UI controls. This code has to be separated where the Business Logic (View) calls will go in the Business Repository which runs on the server and then the part that interacts with the controls has to move to the JavaScript code running in the Browser. The Business Repository has to provide the necessary data in a single payload which the mode/controller will transport to the Browser for processing.

The easiest way for the repository to transfer data is to have the model provide extra fields for this communication. This way no extra layers need to be involved, the business repository just populates these fields and the JavaScript layers pull them out of the returned JSON object and use them.

But you only want to add so much to the model, since you don’t want it to be too cumbersome to move around and you might want more focused calls. For these we usually define special calls in the controller and these go through a services layer to execute the code in the repository. The service call only passes the exact data needed (like parameters to a function) and knows what data to expect back.

Example

Adding extra fields to the model is fairly straight forward, so let’s trace through the logic of making a services call. In this example we’ll look at the simple case of checking a customer’s credit limit in A/R Invoice Entry (which is using a stateful business repository). We’ll start up in the JavaScript code and work our way down through the layers to get an idea of who does what.

So let’s start near the top. In the A/R Invoice Entry UI there are various times when the credit limit needs to be looked up. So the JavaScript code in the InvoiceEntryBehaviour.js file has a routine to initiate this process. Note that server calls are asynchronous so the response is handled in a callback function.

    showCreditLimit: function (result) {
        // Open Credit Check pop up window
        if (result) {
            var jsonResult = JSON.parse(result);
            if (jsonResult.ShowCreditCheck) {
                arInvoiceEntryRepository.getCreditCheck(jsonResult.id,
                    sg.utls.kndoUI.getFormattedDate(jsonResult.docDate),
                    sg.utls.kndoUI.getFormattedDate(jsonResult.dueDate),
                    "n" + invoiceEntryUI.CurrencyDecimals, jsonResult.totalPaymentAmountScheduled,
                    jsonResult.prepaymentAmount);
            } else {
                onSuccess.onCreditClose();
            }
        }
        invoiceEntryUI.ModelData.isModelDirty.reset();
    },

This calls a function in the InvoiceEntryRepository.js file to actually make the call to the server:

    getCreditCheck: function (customerNumber, documentDate, dueDate, decimals, invoiceAmount,
        prepaymentAmount) {
        var data = {
            id: customerNumber,
            docDate: documentDate,
            dueDate: dueDate,
            decimals: decimals,
            totalPaymentAmountScheduled: invoiceAmount,
            prepaymentAmount: prepaymentAmount
        };
        sg.utls.ajaxPostHtml(sg.utls.url.buildUrl("AR", "InvoiceEntry", "GetCreditLimit"), data,
              onSuccess.loadCreditLimit);
    },

This will initiate the call to the server. The URL will be built something like servername/Sage300/AR/InvoiceEntry/GetCreditLimit. The ASP.Net MVC infrastructure will use configuration by convention to look for a matching entry point in a loaded controller and hence call the  GetCreditLimit method in the InvoiceEntryController.cs file:

        [HttpPost]
        public virtual ActionResult GetCreditLimit(string id, string docDate, string dueDate,
               string decimals,decimal totalPaymentAmountScheduled, decimal prepaymentAmount)
        {
            try
            {
                return PartialView(AccountReceivable.ARInvoiceCreditCheck,
                      ControllerInternal.GetCreditLimit(id, docDate, dueDate, decimals,
                      totalPaymentAmountScheduled, prepaymentAmount));
            }
            catch (BusinessException businessException)
            {
                return JsonNet(BuildErrorModelBase(CommonResx.NotFoundMessage, businessException,
                    InvoiceEntryResx.Entity));
            }
        }

Which will call the InvoiceControllerInternal.cs GetCreditLimit method:

        internal ViewModelBase<CustomerBalance> GetCreditLimit(string customerNumber,
            string documentDate,
            string dueDate, string decimals
            , decimal totalPaymentAmountScheduled, decimal prepaymentAmount)
        {
            var creditBalance = Service.GetCreditLimit(customerNumber, totalPaymentAmountScheduled,
                 prepaymentAmount);

            if (creditBalance.CalcCustomerOverdue == CalcCustomerOverdue.Yes &&
                creditBalance.CustomerBalanceOverdue > creditBalance.CustomerAmountOverdue)
            {
                creditBalance.CustomerCreditMessage = string.Format(
                        InvoiceEntryResx.CustCreditDaysOverdue,
                        creditBalance.CustomerDaysOverdue,
                        creditBalance.CustomerBalanceOverdue.ToString(decimals),
                        creditBalance.CustomerAmountOverdue.ToString(decimals));
            }

            if (creditBalance.CalcNatAcctOverdue == CalcNatAcctOverdue.Yes &&
                creditBalance.NatAcctBalanceOverdue > creditBalance.NatAcctAmountOverdue)
            {
                creditBalance.NationalCreditMessage = string.Format(
                        InvoiceEntryResx.NatCreditDaysOverdue,
                        creditBalance.NatAcctDaysOverdue,
                        creditBalance.NatAcctBalanceOverdue.ToString(decimals),
                        creditBalance.NatAcctAmountOverdue.ToString(decimals));
            }

            creditBalance.CustomerCreditLimit =
                 Convert.ToDecimal(creditBalance.CustomerCreditLimit.ToString(decimals));
            creditBalance.CustomerBalanceVal =
                 Convert.ToDecimal(creditBalance.CustomerBalanceVal.ToString(decimals));
            creditBalance.PendingARAmount =
                 Convert.ToDecimal(creditBalance.PendingARAmount.ToString(decimals));
            creditBalance.PendingOEAmount =
                 Convert.ToDecimal(creditBalance.PendingOEAmount.ToString(decimals));
            creditBalance.PendingOtherAmount =
                 Convert.ToDecimal(creditBalance.PendingOtherAmount.ToString(decimals));
            creditBalance.CurrentARInvoiceAmount =
                 Convert.ToDecimal(creditBalance.CurrentARInvoiceAmount.ToString(decimals));
            creditBalance.CurrentARPrepaymentAmount =
                 Convert.ToDecimal(creditBalance.CurrentARPrepaymentAmount.ToString(decimals));
            creditBalance.CustomerOutstanding =
                 Convert.ToDecimal(creditBalance.CustomerOutstanding.ToString(decimals));
            creditBalance.CustomerLimitExceeded =
                 Convert.ToDecimal(creditBalance.CustomerLimitExceeded.ToString(decimals));
            creditBalance.NatAcctCreditLimit =
                 Convert.ToDecimal(creditBalance.NatAcctCreditLimit.ToString(decimals));
            creditBalance.NationalAccountBalance =
                 Convert.ToDecimal(creditBalance.NationalAccountBalance.ToString(decimals));
            creditBalance.NatAcctOutstanding =
                 Convert.ToDecimal(creditBalance.NatAcctOutstanding.ToString(decimals));
            creditBalance.NatAcctLimitLeft =
                 Convert.ToDecimal(creditBalance.NatAcctLimitLeft.ToString(decimals));
            creditBalance.NatAcctLimitExceeded =
                 Convert.ToDecimal(creditBalance.NatAcctLimitExceeded.ToString(decimals));

            return new ViewModelBase<CustomerBalance> { Data = creditBalance };
        }

This routine first calls the GetCreditLimit service in InvoiceEntryEntityService.cs:

        public virtual CustomerBalance GetCreditLimit(string customerNumber,
            decimal totalPaymentAmountScheduled, decimal prepaymentAmount)
        {
            var repository = Resolve<IInvoiceEntryEntity<TBatch, THeader,
                 TDetail, TPayment, TDetailOptional>>();
            return repository.GetCreditLimit(customerNumber,
                 totalPaymentAmountScheduled, prepaymentAmount);
        }

Who then calls the repository GetCreditLimit routine in InvoiceEntryRepository.cs. This routine then appears to do regular View processing using the base repository wrapper routines that insulate us from the session/dblink handling logic as well as do some basic error processing:

        public virtual CustomerBalance GetCreditLimit(string customerNum,
            decimal totalPaymentAmountScheduled, decimal prepaymentAmount)
        {
            _header.Read(false);
            _creditCheck.SetValue(CustomerBalance.Fields.CustomerNumber, customerNum);
            _creditCheck.SetValue(CustomerBalance.Fields.CurrentARInvoiceAmount,
                totalPaymentAmountScheduled);
            _creditCheck.SetValue(CustomerBalance.Fields.CurrentARPrepaymentAmount,
                prepaymentAmount);
            _creditCheck.Process();
            return _creditCheckMapper.Map(_creditCheck);
        }

Finally, down in the business repository, the code should look fairly familiar to anyone you has done any C# coding using our Sage 300 .Net API. Further this code should also appear somewhere in the matching VB code, and besides being translated to using the .Net API, its become quite separated from the UI control code (in this case the JavaScript).

At the end of this all the calls return propagating the returned data back to the Browser in answer to the AJAX call that it made.

It might look like a lot of code here, but remember the business repository and JavaScript bits have corresponding VB code. Then the other layers are there to make all the code more re-usable so that it can be used in contexts like WebAPIs and allow interfaces to provide the hooks needed for customization.

Summary

This article is intended to give you an idea of where to put your code that accesses the Sage 300 Business Logic and then how to call that from the Web UIs. There are a lot of layers but individually most of the layers are fairly simple and most of the code will appear in the Business Repository and the JavaScript behavior code.

Written by smist08

February 12, 2016 at 3:27 am

Stateless Versus Stateful Sage 300c Web UIs

with 2 comments

Introduction

When two computers communicate they use a well-defined communications protocol. In Browser to Server communications these are often broadly categorized as either stateless or stateful. A stateless communications protocol doesn’t require that anything be remembered between calls. In web applications this is desirable for a number of reasons:

  • often calls are load balanced across multiple servers (perhaps even in different locations).
  • storing state in memory can be expensive and will limit how many users can be accessing a web server at once.

On the other hand, maintaining server state has some advantages:

  • less information needs to be transferred between the client and server since the server knows what has gone before.
  • having things handy in memory can make operations faster and more context aware.

Often in the real world these components are combined or stacked on each other, for instance the TCP protocol is stateful and then the HTTP stateless protocol is layered on top of TCP.

The Sage 300c Web UIs use both stateless and stateful technology. This article will talk about when we use each, how to program either case and what are the advantages and disadvantages of each method. Like many things in programming the lines between these two things can become quite blurred.

Since we do have stateful UIs, this does require that if you scale out to multiple web servers then the load balancer must use sticky sessions as we explained in this blog posting. Basically guarantee that all the requests from a given client go to the same web server in case they are running a stateful UI.

Stateless Sage 300 Web UIs

Most of the Sage 300c setup UIs, processing UIs and report UIs are all stateless. This means IIS could be reset between requests and things would still work. It would just be quite slow since it would need to re-open a Sage 300 session and re-open the Views to process the next request. To avoid this we keep a cache of open sessions along with their open Views. This way when a stateless request comes in we match up a session for the right company, user and with the right Views from the session pool and use that to handle the request. If a session in the pool hasn’t been used for a while we will release it and if too many sessions are in use, we will release the oldest if we need another one. This way practically speaking you usually open a session for your first stateless request and then keep using it from the cache of the duration of your work.

schedules

A Typical Stateless UI

Every Sage 300c business logic entity (View) will have a stateless repository defined for it. This is because only the document entry UI needs stateful operation, everything else like finders, export and other UIs will want to use the stateless version. You get all the default behavior for your stateless repository by having your specific repository inherit from one of the stateless base repositories in Sage.CA.SBS.ERP.Sage300.Common.BusinessRepository.Base like FlatRepository, ProcessingRepository or InquiryRepository.

Generally stateless operation fits very well with ASP.Net MVC since this is the natural way that it works. There is a lot of infrastructure to pass the data model back and forth between the client and server in a stateless manner. Add to that our use of knockout data binding and this makes most basic CRUD operations all handled by the framework.

This doesn’t mean that a stateless UI can’t dynamically interact with the user. By default, you have the CRUD operation off the basic navigation and save/delete buttons. But you can certainly add your own custom AJAX calls to dynamically update areas of your form. We provide support in the framework to update things like descriptions of external keys, but in fact you have a lot of power to make your UI very interactive and polished.

Stateful Sage 300 Web UIs

Most of the main Accounting document entry UIs are stateful. This includes UIs like Order Entry, Purchase Orders, G/L Journal Entry and A/R Invoice Entry. These UIs have a lot of sophistication in building up the Accounting document and the size of these documents can be quite large. Sending this entire document back and forth between the browser and the server as a JSON object is quite impractical. When one of these UIs is started, a session and set of Views are assigned to it for the duration that it runs. As changes are made to the UIs they are sent to the server to make the attendant changes to the server model.

oe1100new

A Typical Stateful UI

To create a stateful business repository, just inherit from one of the stateful base repositories in Sage.CA.SBS.ERP.Sage300.Common.BusinessRepository.Base.Statefull like BatchHeaderDetailRepository or SequencedHeaderDetailRepository.

When you exchange information between the server and browser rather than send the entire document back and forth, you tend to send one component back and forth like one order detail line or the order header. This then reduces the amount of data transmitted, but more importantly greatly reduces the size of the JSON object that JavaScript needs to process. If the transferred document gets too large then JavaScript processing speed can become a real bottleneck.

You don’t need to send something to the server every time every field changes, only the fields that cause some business logic to execute. So you don’t need to do anything say when the user changes a description field. When a field changes that causes business logic on the server to execute then you need to send that to the server and get back all the fields that changed as a result. We tend to mostly do this using the default ASP.Net MVC mechanisms along with the knockout data binding. But only doing it for the component record. So say the item number in a detail record is changed, then we would refresh that record by sending it to the server, which would set the changed fields, do the Sage 300 business logic operations (puts and gets) and return the newly updated record where a number of fields have changed.

This works for most cases, but there is room for optimization. If something is rather a major operation, you might want a more tailored AJAX call for the processing. Similarly, if multiple records are affected you might want to denormalize a bit to reduce the dataflow. For instance, when dealing with details, often header totals change, but you might want to associate these with the detail, rather than also refreshing the header.

Summary

This was a quick overview of the main modes or how our Sage 300 Web UIs operation. Stateless operation reduces server overhead, while stateful operation lets us fully leverage our existing business logic, while reducing our bandwidth requirements.

Written by smist08

February 9, 2016 at 8:44 pm

Adding a Grid to Your Sage 300 Web UI

with one comment

Introduction

The grid or table control is a key element for data entry in any Accounting application. With Sage 300 we use the grid control to enter things like Order or Invoice details. Interactions with a grid control tend to be quite complex. The data has to be managed so it is loaded only a page at a time (often called virtual scrolling), since there could be thousands of detail lines and loading them all at once would be quite slow. There is the ability to edit, delete and add lines. Tabbing has to be handled well to enhance data entry. People also have the ability to re-arrange the grid columns, hide columns and then expect these changes to be remembered.

icreceipts

This blog article will talk about the key elements to adding a grid control to your Sage 300 Web UI and what sort of support you need in your UI to support all the desired functionality. A fair bit is handled for you in the Sage 300 Web UI Framework, however you have to handle various events and there is a lot of power to add your own programming.

Configuration

There is a lot of support for standard grid operations in the Sage 300 Web UI framework. Much of this is controlled by a config JSON object which is passed to our @Html.KoKendoGrid function that defines the grid in the Razor View. This file defines a number of properties of the grid along with a number of standard callout functions you can define to add your custom processing. The good news is we have a utility to generate much of this from the ASP.Net MVC Model.

JavaScript Generation Utility

To generate this code we provide a utility which will generate the Razor View code and a lot of the standard JavaScript code that you need. So for instance the code for the Razor View might be:

jsgen1

And then some of the JavaScript code for the config object might be:

jsgen2

Server Side Pagination

In our VB UIs, we had virtual scrolling in our grids, which would basically bring in a page or two at a time. It supported scrolling one page ahead or one page back, go to the top or bottom but you couldn’t go to an arbitrary point in the file without searching (in fact the scroll bar would always be at the top, bottom or right in the middle). In the Web UIs we use the Kendo UI Grid control and try to keep the scrolling mechanism standard for the Web, which means the control tells you how many pages there are and lets you go to any page you like as well as going to the next or previous one.

We provide a lot of the support for doing this in our business repository base classes which expose a get method which takes the page number, page size, filter and order as parameters. Then as long as you match and set the configuration data in the grid’s JavaScript config JSON object, you get the pagination support. There are a couple of things to keep in mind, one is that we rely on our filterCount API call, which translates directly to a SQL statement, which means it can only count based on database fields and not calculated fields, so you can’t restrict the records in your grid based on any calculated fields or the count will be wrong (if you really need this then you need to disable the ability to go to a specific page). You also need to have a hidden SerialNumber column in the grid which contains the record number.

ViewListControl vs AccpacGrid

In our VB UIs, we actually had two grid controls. One was the ViewListControl which would show a separate View record in each line and supported virtual scrolling. Then we had the AccpacGrid control which would usually show an array of fields from a single record (like tax information, or perhaps item structure information).

In the Web UIs we only have one Grid control. It naturally works more like VB’s ViewListControl. So how do we handle the other case of the AccpacGrid? We do this in our controller by translating the array of fields into what looks like a list of details. This way to the Grid control, it doesn’t really see a difference. Usually you don’t need to enable virtual scrolling in this case since there is typically 5 or 10 records and you just provide them all at once. So typically your ViewModel will have a list of records which the controller will populate and then this is set as the Grid’s data source.

Editing

Like VB, the intent of editing cells is to place the correct edit control over the grid cell to perform the edit. There is a lot of framework support for this as well as lots of callouts for you to do your own custom processing. The same is true for adding a new line and deleting a set of lines (note that the Web UI grid supports multiple selection). Also note that the add line, delete line, edit columns buttons aren’t part of the Grid, these are separate buttons styled to look like part of the grid in a region just above the Grid. This means you can easily add your own buttons and controls to this area if you wish.

Saving Preferences

We have API support to help with loading and saving grid column preferences. In VB these are stored in the *_p.ism files, in the Web UIs these are stored in the SQL database in the new USRPROP table. So emptying USRPROP is the Web UI equivalent of deleting the *_p.ism files. Generally, we want to move everything into the database and remove our reliance on the shared data folder over time.

Summary

This article was just a quick introduction to adding a Grid control to a Web UI. Similar to the VB UIs, the grid control is potentially quite complicated as it supports a lot of diverse functionality. But, if you are doing fairly standard functionality, look for a lot of support in the Web UI framework to help you get the job done.

Written by smist08

February 3, 2016 at 6:29 pm

Adding Your Application to the Home Page

with 6 comments

Introduction

We’ve been talking about how to develop Web UIs using our beta SDK. But so far we’ve only been running these in Visual Studio, we haven’t talked about how to deploy them in a production system. With this article we’ll discuss how to add your menus to the home page. Which files need to be installed where and a few configuration notes to observe.

We’ll continue playing with Project and Job Costing, so first we’ll add PJC on to the end of the More menu in the Home Page:

pjc1

As part of this we’ll build and deploy the PJC application so that we can run its UIs in a deployed environment, rather than just running the screens individually in Visual Studio like we have been.

pjc2

The Code Generation Wizard

When you create your solution, you get a starting skeleton Sage.PM.BusinessRepository/Menu/PMMenuModuleHelper.cs. I’m using PM since I’m playing at creating PJC Web UIs, but instead of PM you will get whatever your application’s two letter prefix is. If you don’t have such a prefix, remember to register one with Sage to avoid conflicts with other Sage 300 SDK developers. Similarly, I use Sage as my company, but in reality this will be whatever company name you specified when you created the solution. This MenuModuleHelper.cs file specifies the name of the XML file that specifies your application’s Sage 300 Home Page menu structure. This C# source file is also where you put code to dynamically hide and show your program menu items, so for instance if you have some multi-currency only UIs then this is where you would put the code to hide them in the case of a single currency database (or application).

The solution wizard creates a starting PMMenuDetails.xml file in the root of the Sage.Web project. Then each time you run the code generation wizard it will add another item for the UI just generated. This will produce rather a flat structure so you need to polish this a bit as well as fix up the strings in the generated MenuResx.resx file in the Sage.PM.Resources project. This resource file contains all the strings that are displayed in the menu. Further you can optionally update all the generated files for the other supported languages.

One caveat on the MenuDetails.xml file is that you must give a security resource that the user has rights to or nothing will display. Leaving this out or putting N/A won’t work. One good comparison is that since these are XML files you can see all of Sage’s MenuDetails.xml files by looking in the Sage 300\Online\Web\App_Data\MenuDetail folder. Note that the way the customize screen works, it removes items and puts them in a company folder under these. It will regenerate them if the file changes, but if you have troubles you might try clearing these folders to force them to be regenerated.

Below is a sample XML element for a single UI to give a bit of flavor of what the MenuDetails.xml file contains.

 

  <item>
    <MenuID>PM4001</MenuID>
    <MenuName>PM4001</MenuName>
    <ResourceKey>PMCostType</ResourceKey>
    <ParentMenuID>PM2000</ParentMenuID>
    <IsGroupHeader>false</IsGroupHeader>
    <ScreenURL>PM/CostType</ScreenURL>
    <MenuItemLevel>4</MenuItemLevel>
    <MenuItemOrder>2</MenuItemOrder>
    <ColOrder>1</ColOrder>
    <SecurityResourceKey>PMCostType</SecurityResourceKey>
    <IsReport>false</IsReport>
    <IsActive>true</IsActive>
    <IsGroupEnd>false</IsGroupEnd>
    <IsWidget>false</IsWidget>
    <Isintelligence>false</Isintelligence>
    <ModuleName>PM</ModuleName>
  </item>

Post Build Utility

Now that we have our menu defined and our application screens running individually in debug mode inside Visual Studio, how do we deploy it to run inside IIS as a part of the Sage 300 system? Which DLLs need to be copied, which configuration files need to be copied and where do they all go? To try these steps, make sure you have the latest version of the Sage 300 SDK Wizards and the matching newest beta build.

The Wizard adds a post build event to the Web project that will deploy all the right files to the local Sage 300 running in IIS. The MergeISVProject.exe utility can also be run standalone outside of VS, its a handy mechanism to copy your files. Its usually a good idea to restart IIS before testing this way to ensure all the new files are loaded.

postbuild1

This utility basically copies the following files to places under the Sage300\online\web folder:

  • xml is the configuration file which defines your application to Sage 300. Think of this as like roto.dat for the Web. It defines which are your DLLs to load using unity dependency injection when we start up.
  • App_Data\MenuDetail\PMMenuDetails.xml is your menu definition that we talked about earlier.
  • Areas\PM\*.* area all your Razor Views and their accompanying JavaScript. Basically anything that needs to go to the Browser.
  • Bin\Sage.PM.*.dll and Bin\Sage.Web.DLL are the DLLs that you need to run. (Keep in mind that I’m using Sage for my company name, you will get whatever your company is instead of Sage in these).

With these in place your application is ready to roll.

Update 2016/01/20: This tool was updated to support compiled razor view and the command line is now:

Call "$(ProjectDir)MergeISVProject.exe" "$(SolutionPath)"  "$(ProjectDir)\"
     {ModuleName}MenuDetails.xml $(ConfigurationName) $(FrameworkDir)

Plus it is only run when the solution (not an individual project) is set for a “Release” build.

Compiled Views

When we ship Sage 300, all our Razor Views are pre-compiled. This means the screens start much faster. If you don’t compile them, the when first accessed, IIS needs to invoke the C# compiler to compile them and we’ve found that this process can take ten seconds or so. Plus, the results are cached and the cache is reset every few days causing this to have to happen over again. Another plus is that when pre-compiled the Views can’t easily be edited, which means people can’t arbitrarily change these causing a problem for future upgrades.

Strangely Visual Studio doesn’t have a dialog where you can set whether you want your Views pre-compiled, you have to edit the Sage.Web.csproj file directly. And you need to change the XML value:

<MvcBuildViews>false</MvcBuildViews>

Between true and false yourself in a text editor.

The Sage 300 system is set so that it only runs compiled Razor Views. If you do want to run un-compiled Razor Views, then you need to edit Sage300\online\web\precompiledapp.config and change updatable from false to true.

Beta note: As I write this the MergeISVProject utility doesn’t copy compiled Views properly. This will be fixed shortly, but in the meantime if you want to test with compiled Views you need to copy these over by hand.

New Beta note: This tool now fully supports compiled razor views.

Beta note: The previous beta wouldn’t successfully compile if you were set to use compiled Views, this has been fixed and the solution wizard now includes all the references necessary to do this.

Summary

This article was meant to give a quick overview of the steps to take once you have a screen working in Visual Studio debug mode, to being able to test it running in the Sage 300 Home Page as part of a proper deployment. Generally, the tools help a lot and hopefully you don’t find this process too difficult.

 

Written by smist08

December 29, 2015 at 4:26 pm

Sage 300 Web UI SDK – Adding JavaScript

with one comment

Introduction

Last week we talked about adding UI controls to our Web UI project. In the early days of the Web, this would be all you needed to do. The user would just enter the data and hit save (or submit). However modern Web applications are expected to be more responsive and interactive. This is achieved by adding JavaScript code which can perform local processing or make calls to the server for additional data or to perform additional actions. Most modern Web applications contain quite a bit of JavaScript and achieve quite a high degree of user interaction.

jscalm

In our source code for the new Web UIs, it turns out that 80% of the application code is C# and 20% is JavaScript. Although it is only 20%, this is still a lot of code and care needs to be taken with it. Some programmers really love JavaScript, some really hate it. JavaScript is more of a free form interpreted language that is very forgiving. This allows a lot more freedom in how you structure your programs. Debugging and writing JavaScript is quick and easy since there is no big build/compile step. On the other hand if you capitalize or spell something wrong, there is no compiler to catch the mistake and debugging these errors can be quite frustrating.

We use JQuery to isolate ourselves from Browser differences, for the most part with newer versions of all the Browsers the differences in dialects of JavaScript is much smaller than it used to be. But it still has to be taken into account. Usually the differences manifest themselves in error handling and security concerns. Sometimes one Browser might ignore an error and keep on going whereas another Browser will throw an exception. Similarly, as some practices that lead to security problems are tightened, the implementation can be a bit uneven across the Browsers. Fortunately, most of our application programming doesn’t hit these cases, but it does highlight one the importance of testing in all the supported Browsers.

Another good thing about JavaScript is that tools have all gotten better. You can debug your code in Visual Studio while it runs in Internet Explorer quite seamlessly with debugging you C# code. Further the other Browser all have great debugging tools that can really help. A lot of times you can get several views on something to see what is going wrong. Plus there are compilers for JavaScript to help find some common typos and to enforce programming standards you might want to adopt.

Frameworks

We provide a global framework for doing things like popping up Finders. We also have lots of routines to handle our standard data like dates and numbers. We use a number of open source libraries like JQuery and Knockout.js as well as the (partly commercial) Kendo UI library. Generally, if you know our framework, you don’t need to know that much from the other libraries, but as you get more sophisticated, you will want to leverage them as well. There is also nothing stopping you including further libraries, just that we won’t be able to support you in their use.

Server Independent

We don’t use any libraries with matching client and server components. Hence all communications are handled by our JavaScript code, which means we have full control over all communications. This means it’s our responsibility to not be chatty. It also means we write all the code to handle any Ajax calls from the Browser. ASP.Net MVC makes this pretty easy. It also means that if you want to add your own components (say another UI control) then you can do so as long as you follow this same basic approach.

Example Code

The Web UI code generation wizard creates three JavaScript files for each UI. These are:

  1. screennameKoExtn.js – This is an extension to the knockout data binding where you can add some JavaScript generated calculated fields to the model.
  2. screennameRepository.js – This file contains a method for each RPC call that can be made. We start with the basic CRUD operations, but you would add your screen specific ones here.
  3. screennameBehaviour.js – This is all the other JavaScript code. You may want to break this file up for larger UIs. But for smaller UIs this is a handy place to contain all the JavaScript for working with the controls on your form.

The Behaviour.js file contains a large object for the screen with functions to bind controls to methods and to do various processing when that control is activated. For instance, for our sample PJC CostType UI, in the init routine it will call initButtons which will bind all the buttons to code including the Save button:

       // Save Button
       $("#btnSave").bind('click', function () {
              sg.utls.SyncExecute(costTypeUI.saveCostType);
       });

Then the saveCostType routine is:

saveCostType: function () {
     if ($("#frmCostType").valid()) {
         var data = sg.utls.ko.toJS(modelData, costTypeUI.computedProperties);
         if (modelData.UIMode() === sg.utls.OperationMode.SAVE) {
              costTypeRepository.update(data, costTypeUISuccess.update);
         } else {
         costTypeRepository.add(data, costTypeUISuccess.update);
      }
   }
},

The costTypeRepository.update routine is in the Repository.js file and performs the Ajax call to do the save:

       // Update
       update: function(data, callbackMethod) {
              costTypeAjax.call("Save", data, callbackMethod);
       }

This Ajax.call routine will actually transmit the data model to the server where it is picked up by a controller and processed. We talk about the server side of things in a later article. The entire file is a bit on the long side to include in the blog, but hopefully this give a bit of flavour of what it contains.

The dataModel.UIMode is a predefined calculated field in the screennameKoExtn.js file and keeps track of whether the UI is in Add or Save mode.

Summary

This was a quick glimpse of the JavaScript that is always running behind the scenes in our new Sage 300 Web UIs. Once we cover the server side, we’ll have to consider a lot of cases that go from the HTML to the JavaScript to the C# to the Sage 300 Business Logic to the SQL database and then all the way back. This is just the second step on this journey after last week’s article.

 

Written by smist08

December 12, 2015 at 8:22 pm