6. Evaluation

6.1 Overview

Object-oriented frameworks like the solutions discussed in the previous chapter are difficult to evaluate.

One initial difficulty is to understand the intended domain of the framework and its applicability to the application under construction [BMMB97].

The frameworks which are relevant for this thesis are known as application frameworks [GB01]. Their purpose is to provide all the domain-independent functionality needed in an application. The evaluated solutions extend another application framework, the .NET Framework. In comparison to the .NET Framework they provide additional functionality for more specific domains:

In the evaluation, the Composite UI Application Block is always used with the extensions provided by the Smart Client Software Factory (Chapter 5.3).

The evaluation of the applicability of these frameworks for the Test Suite application is done by checking the fulfillment of the requirements. This is dealt in the next chapter. The applicability is the most important part in this evaluation. Only suitable frameworks are considered in the next two evaluation parts which are about further quality issues (chapter 6.3) and strategic aspects (chapter 6.4).

In all three evaluation parts the appraisal is done by using three grades:

6.2 Fulfillment of the Requirements

This chapter presents the first and most important evaluation part. It checks the fulfillment of the requirements which are defined in chapter 2. Table 2 shows a summary of this evaluation which is discussed afterwards.

Requirement CAB/SCSF May 2007 Spring .NET Version 1.1 SharpDevelop Version 2.1
Runtime platform:
Minimum .NET Framework version
Open source and programmed in C# (+) (+) (+)
Test modules
Define test modules (+) (+) (+)
External configuration (+) (o) (+)
Loose coupling (+) (o) (-)
Lazy loading of modules (o) (+) (+)
Modules deployment (o) (o) (+)
GUI integration
Support for GUI extension (+) (-) (o)
Command service (+) (-) (o)
Loosely coupled events (+) (o) (-)

Table 2: Checking the fulfillment of the requirements which are defined in chapter 2.

Runtime platform

All solutions run on the .NET Framework 2.0 as it is specified in the requirements.

Open source and programmed in C#

The source code of the three frameworks is available and the used language is C#.

Define test modules

The .NET assemblies are an ideal candidate for representing the test modules. Assemblies are the basic unit for versioning, security, and deployment. Hence, they fulfill the requirements. An assembly can physically be a standalone application (.exe) or a class library (.dll) [Löwy05, p. 23]. In the case of a test module the choice would be a class library. By using assemblies the requirement is already dealt by the .NET Framework. Therefore, the investigated frameworks do not have to provide an own solution for defining the test modules.
CAB extends the .NET assemblies by the introduction of the abstract ModuleInit class. A concrete implementation of this class is used to initialize the module. During module loading this concrete class is searched by reflection and initialized through the framework.
Spring .NET does not extend the functionality of the .NET assemblies. Because the .NET Framework already fulfills this requirement, Spring .NET gets full grade too.
In SharpDevelop a module is defined by the .addin xml file. This file refers to a .NET assembly by using the Identity tag. The .addin file extends the meta-data of the assembly with additional information.

External configuration

The configuration for the module loading has to be in an external human-readable file. Changes of the configuration need to be done without recompiling the application. All investigated solutions use an adoption of the Plugin pattern to fulfill this requirement. A short introduction of the Plugin pattern can be found in chapter 3.
The module loader of CAB can be configured via a single XML file. This file contains the module assemblies and the dependencies between them. The information about dependencies assures that the modules are loaded in the correct order.
Spring .NET does not support the loading of modules in the same way as the other solutions do. It can configure the wiring of single components in XML files. For the reason that every module can contain many components the configuration gets extensive. Moreover, the one who is configuring the application needs in-depth knowledge about the dependencies of each component in the module. A way to reduce the amount of configuration is to use the autowiring function of Spring .NET. The setter injection with activated autowiring is limited because it is not possible to define that some properties of an object need to be injected and others not. Autowiring only saves the writing of the dependency information. The components still have to be defined in the configuration file. That is the reason why Spring .NET does not get full mark for this requirement.
SharpDevelop scans special directories for .addin files and interprets them. The .addin files can define dependencies to other modules in a similar way as in CAB. The main difference to CAB is that in SharpDevelop every module has its own configuration file whereas in CAB a central configuration file is used.
All three frameworks ship XML schema files (XSD) for the configuration. This simplifies the writing and editing of the configuration in an XML schema aware editor. The requirements of this thesis also specify the configuration via command line arguments. The frameworks do not provide direct support for this requirement but the application can extend them to provide this functionality. A possible solution with the CAB framework can be seen in chapter 7.5.

Loose coupling

Loose coupling between the modules is essential for fulfilling the requirements which are defined in chapter 2. It allows the developing of modules by different teams. Furthermore the modules can be isolated for testing. This simplifies the test procedure since the dependent components can be replaced by mock objects. Loose coupling can be achieved by programming to an interface (see chapter 3). Nevertheless, the loosely coupled modules have to work together in a coherent application. This means that the components of the modules have to be weird up.
CAB and Spring .NET support the wiring by the Dependency Injection and the Service Locator implementation. In both frameworks the Dependency Injection and Service Locator implementation work hand in hand together and thus, they can even be mixed in one application. The Service Locator implementation in Spring .NET has one drawback. It does not allow the replacement of a service instance during runtime.
SharpDevelop had a Service Locator implementation called ServiceManager in a previous version [HKS03, p. 109], but the current version 2.1 has replaced the ServiceManager with static service classes. This step simplifies the service usage but it does not allow the replacement of the services any more. The isolating of components during a test is impossible without replacing the dependent services. A solution for implementing the Service Locator on top of the add-in tree is demonstrated in chapter 5.5. The framework still gets the grade (-) because it does not support loose coupling by itself.

Lazy loading of modules

The Test Suite is an extensive application. For keeping the start-up time at a minimum, the modules have to be lazy loaded. This strategy handles the resources in a smart way, as only the needed ones are allocated. If a user does not use some modules during work these modules will never be loaded. Therefore, unused modules do not waste any resources.
The Composite UI Application Block is not able to load the modules on demand. This is due to the fact that the application integration is done in the module initializing code. However, CAB investigates only the modules via reflection and instantiates the subclass of ModuleInit. The services that the modules provide can be loaded on demand with the ServiceCollection.AddOnDemand method.
Spring .NET does not support modules in a special way. Therefore, it has no need to load the assemblies at the start-up process. The components, which are needed by other components, are instantiated on demand [Spring07, p. 29]. If more components have to use the same instance, the dependent component can be defined as singleton. By default, singletons are instantiated during the start-up sequence of the container. The lazy-init attribute allows delaying the creation until the component requested for is the first time [Spring07, p. 18].
SharpDevelop defines the extension points in the .addin configuration files. These configuration files are read at application start-up only. The loading of the add-in assemblies is delayed until one of its extension points is accessed [Grunwald06a].

Modules deployment

Deployment is a topic that is well supported by the .NET Framework. .NET provides version control for the modules and allows side-by-side execution of different module versions in the same process.
The Smart Client Software Factory includes help topics and a reference application for using ClickOnce deployment. ClickOnce simplifies the deployment tasks for the end user and the manufacturer but it has its limitations.

(…) if a program needs to carry out privileged operations that could affect other applications or data on the target machine, such as performing unrestricted file access or accessing the registry, then it may not be suitable for deployment using ClickOnce [Noyes04].

The Test Suite requires unrestricted access to the file system and it has to install native components like system drivers for the test devices. Therefore, ClickOnce is not an option.
Spring .NET does not provide any special deployment features.
SharpDevelop simplifies the deployment of add-ins since the add-in files only need to be copied into one of the specified directories. In contrast to CAB and Spring .NET it is not necessary to modify a configuration file for installing and uninstalling an add-in. A prefabricated add-in named AddIn Manager allows the end user to control the add-ins. A limitation of the AddIn Manager is that the add-ins can be installed into the user profile directory only. This issue is caused by a security restriction of the operating system because it cannot be guaranteed that the end user has write access in the application directory.
It is required to do the deployment tasks without restarting the application which is not supported by all three solutions. The reason is that the .NET Framework cannot unload .NET modules or assemblies. However, it is possible to load every module in a different application domain. All application domains except of the default one can be unloaded by the .NET Framework [Löwy05, p. 322]. The drawback of this strategy is that the modules have to communicate through remoting with each other. Solutions, which are using this strategy, are the System.Addin namespace introduced in the .NET Framework 3.5 (Chapter 8.2) and the CAP .NET project [Dhungana06]. If this requirement is not fulfilled, it is acceptable since it is just defined as a nice-to-have requirement.

Support for GUI extensions

The test modules have to be integrated into the Test Suite user interface. For example, a test module needs to add a new menu item in the menu bar of the Test Suite. The challenge is to create the extension without having a dependency on a concrete UI technology. Furthermore, a test module also needs also the possibility to define own GUI extensions for other modules.
CAB provides a flexible mechanism to extend the user interface. This mechanism consists of two parts which are integrated into the WorkItem. The first part is the Workspace. It is used for hosting UI controls of other modules. The second part is the UIExtensionSite. The extension site allows the extension of exposed UI elements. Every module can use these parts to provide own GUI extensions. Both parts are independent of the UI technology. The Composite UI Application Block supports Windows Forms controls and allows the hosting of WPF controls in the Workspaces. If other requirements occur, the framework can be extended.
The Spring .NET framework does not support the building of Windows-based applications out of the box. In version 1.1 the support is limited on Web-based applications which are using ASP .NET. However, the framework can be extended with the required functionality.
SharpDevelop provides GUI extensions through the .addin configuration files. This extension mechanism is independent of the UI technology. The core supports the handling of Windows Forms controls. If other technologies must be used, a rewriting of the core is necessary. The rewriting of code is not the best strategy to extend the functionality. After modifying the core it has to be accurately tested to assure that no side-effects occur. An advantage of SharpDevelop is that every module can register its own GUI extensions by defining a new AddInTree path. A reusable mechanism for hosting of UI elements like the Workspace of CAB is missing. Instead, SharpDevelop uses the specific WorkspaceSingleton class to host the Windows Forms controls.

Command service

An implementation of the Command design pattern [GHJV95, p. 233] is necessary for the Test Suite. It is required to decouple the UI elements from the command handlers.
The Composite UI Application Block contains a command service. It is managed by the WorkItem container. It is possible that different UI elements can register themselves as command invoker to the same command as required. An adequate CommandAdapter is necessary for registering a UI element. If a UI element type is not known by the framework, a new adapter can be registered in the ICommandAdapterMapService. Furthermore, the command supports the notification of more than one command handler. The defining of a command handler is simple because the CommandHandler attribute just needs to be attached to the method.
Spring .NET does not have a command implementation for user interface elements.
SharpDevelop provides a command implementation. The commands are defined in the .addin file as an attribute of the associated UI element. Different UI elements can use the same command class. The command class has to implement the ICommand interface. From this it follows that the command class is already the command handler. It is not possible that a second command handler can handle the same command. Another drawback is that the command implementation is not able to handle the command state. For example, the state is responsible for deactivating all associated UI elements if the command cannot be executed. In SharpDevelop this is done by Conditions. Nevertheless, the Conditions are associated directly to the UI elements instead of associating to the commands. If more UI elements do the same task the Conditions have to be applied to all of them. This means code duplication in the .addin file.

 1  ...
 3  <Condition name = "ActiveWindowState" windowstate="Dirty,Untitled"
 4      nowindowstate="ViewOnly" action="Disable">
 5    <ToolbarItem id = "Save"
 6      ...
 8  <Condition name = "ActiveWindowState" windowstate="Dirty,Untitled"
 9      nowindowstate="ViewOnly" action="Disable">
10    <MenuItem id = "Save"
11      ...
13  ...

Listing 11: An extract of the ICSharpCode.SharpDevelop.addin file that shows code duplication.

Listing 11 shows the save ToolbarItem and the save MenuItem. Both items require the same condition statement since the items have the identical meaning. In the SharpDevelop .addin file the condition statement is duplicated.

Loosely coupled events

The framework has to support loosely coupled events for communication between the modules. Hence, two objects can register themselves as publisher and subscriber without knowing each other.
CAB supports the loosely coupled events even on dependency injection level. The publishers have to define their event declaration with the EventPublication attribute. The subscribers define the event handler method with the EventSubscription attribute. The attributes use a string as identifier for the event topic. The event publisher and subscribers are wired together by the framework during the object creation.
Spring .NET has a built in support for loosely coupled events too. The objects have to register themselves via the IEventRegistry interface at a central registry as publisher or subscriber. It is possible to create own event registries but it is more common to use the central event registry which is provided by the IApplicationContext. A Dependency Injection style of event wiring is not supported. Nevertheless, the main drawback is that a subscriber can wire itself to a specific event only if a unique delegate type is used for the event. Otherwise, the subscriber has to handle all events that match with its methods signatures. The only filter that can be applied during registration of a subscriber is the publisher type. Though, this can be a problem because the subscriber needs a reference to the publisher for applying the filter.
SharpDevelop does not support loosely coupled events at all.