TAILORING RUP FOR LMS SELECTION: A CASE STUDY

Learning Management System (LMS) development has become a high priority project for educational institutions and organizations, as it provides the virtual environment for online education. Acquiring and deploying a LMS is a difficult task that involves r isks related with costs and time. The goal of this research in progress is to introduce an extension i n the Rational Unified Process (RUP) in order to integrate the activities of selection and evaluatio n of LMS into this process framework. The additions of these activities in the RUP, improve the quality o f the selection process, obtaining a feasible candi date solution that ensure the exposing mismatches and ne gotiating tradeoffs among the critical use cases an d non-functional requirements, architectural and desi gn constraints, project management constraints and risks. These activities are inserted in the Analysi s and Design Discipline during the initial stages o f the project. A case study is presented, implementing an d deploying a LMS in an organization.


INTRODUCTION
Information Technologies have become part of education and training in organizations in the last few years [33]. New elements have appeared to improve the level of the education and training, making them more accessible eliminating the space-temporary barriers that could previously exist. Universities and organizations around the world are using network-based education to distribute remote courses that lean in the use of the technologies supported by Internet through Virtual Learning Environments (VLE).
A VLE is a software system designed to facilitate teachers in the management of online educational courses for their students, especially by helping teachers and learners with course administration. The system can track the learners' progress, which can be monitored by both teachers and learners. Components of these systems usually include templates for content pages, discussion forums, chat, quizzes and exercises such as multiple-choice, true/false and one-word-answer. Teachers fill in these templates and then release them for learners to use. Services generally provided include access control, provision of e-learning content, communication tools, and administration of user groups. Such e-learning systems are also called Learning Management Systems (LMS), Course Management Systems (CMS), Managed Learning Environments (MLE), Learning Support Systems (LSS) or Learning Platforms (LP) [6,10,14,20,26,36]. The unique characteristics of these software introduce dynamics and specific constraints that must be accommodates. Projects that build learning solutions based on this software require dedicated guidance [3,4].
In the last few years, there is an increased interest in the process of selecting, implementing, integrating and deploying an LMS in an organization [13,27,28]. Recent studies [16] have shown that these processes have an impact in cost, time and customer satisfaction. In studies made by the Bersin & Associates Research Center [16] found that only 30% of the companies developed their own LMS. From the companies that bought the LMS, only 34% were large companies or institutions that took more than a year in implementing a LMS, whereas only a 2% took less than three months. Also they found that the costs of implementation of commercial LMS in large institutions were near USD 400.000. From these data it is possible to infer that it is very common that companies acquire LMS instead of developing it, and that once it is acquired, it turns out to be expensive to deploy it.
Organizations that decide to acquire, implement and deploy a LMS instead of developing their own, have the option to integrate a proprietary or an open source LMS in the existing Information Technologies of the organization [6]. A decision has to be taken, either to develop their own, to select and hire a proprietary software or to select an open source one. This decision depends on the economic, technical and human resources, available time and instructional needs of the institution [6,13]. The right choice will depend on the requirements and necessities of the institution. Some authors, like Fernandez [13], recommends the analysis and selection of a LMS that is already available in the market instead of developing a new LMS that match the needs of the institution.
RUP is a software engineering process. It provides a disciplined approach to assigning tasks and responsibilities within a development organization. Its goal is to ensure the production of high quality software that meets the needs of its end users within a predictable schedule and budget [24] RUP is also a process framework that can be adapted and extended to suit the needs of an adopting organization. Roles, artifacts, activities, guidelines, concepts and mentors are elements that you can add or replace to evolve or adapt the process to the organization's needs [24].
This research in progress examines the RUP extensions that can be added in order to obtain a framework process for LMS delivery.
This article is structured in the following way: in the first place, LMS definition, LMS selection and evaluation processes are presented. Evaluation methods are reviewed in the literature. Then, a tailoring is proposed for RUP, followed by its application on a case study in an organization. The results are presented and discussed, and finally, the conclusions and future works are presented.

LEARNING MANAGEMENT SYSTEMS (LMS) DEFINITION
A LMS is a software that automates the administration of training events and supports the management of learning in an organization [10,14,22]. All LMSs manage the log-in and registration of users, manage course catalogs, record data from learners, and provide reports to management [14]. According to Brockbank [6], an LMS ties six elearning components: (1) content, (2) collaboration, (3) testing and assessment, (4) skills and competency, (5) ecommerce and, (6) Internet video-based learning, in a framework that tracks, supports, manages and measure elearning activities. Kanahele [21] states that a LMS provides the infrastructure that centralizes several components associated which each phase of the learning cycle. These three phases and their components are: (1) Assessment phase: knowledge assessment, competency assessment and learning evaluation; (2) Preparation Phase: learning catalog, ecommerce and enrollment; and (3) Learning phase: learning activity, expert forum and community components.
WCET-Edutools [35] proposes two sets of tools that have to be present in a LMS: (1) Learner tools: communication tools, productivity tools and student involvement tools, (2) Support tools: administration tools, course delivery tools and curriculum design.

LMS SELECTION AND EVALUATION PROCESSES
There are dozens of LMS in the market to choose from [35]. From proprietary to open source ones. Therefore, it is not necessary to build one starting from the beginning, only in especial circumstances [33].
In the literature review [5,6,8,9,11,12,15,19,20,35], it has been identified three activities for selecting a LMS for an organization. These three steps are: 1. Perform an initial study for the organization 2. Preselect the LMS from the dozens in the market. 3. Evaluate the LMS preselected

Initial organization study
Before selecting the right LMS for the organization, Brockbank [6] proposes to consider the following criteria, so the correct decision can be made: Analyze the organization's current training and learning environment, commitment, technology and resources. Determine what needs must be met by an LMS. What existing IT training (tools, content, etc.) will need to be integrated into the LMS? What is the schedule for the deployment of the LMS? For Avgeriou et al. [2], the analysis of LMS must take into account the pedagogical context and the instructional design of the organization. As for Rosenberg [33], the key is to choose a LMS that is right for the organization size, budget and complexity.

Preselection activity
Some organizations and researchers propose a preselecting activity before the evaluation and final selection of the LMS, that would be used in the organization [5,8,11,12,15,20]. Some Preselection criteria have been considered by some authors: LMS must conform to the minimum definition of LMS [20]. Decide from open source or commercial [9]. The LMS has been used within the country [8,11]. The evaluation group had positive experiences with the LMS, or heard positive comments about it from others [11]. LMS support multiple languages [12]. LMS server runs on multiple operating systems [8,12].

Preselection criteria
The LMS has been used within the country 3 The evaluation group had positive experiences with the LMS or heard positive comments about it from others 4 LMS supports multiple languages 5 LMS server runs on multiple operating systems 6 LMS integrates homogeneously the learning environment 7 LMS has basic documentation available 8 LMS is compliant with elearning standards 9 The LMS organization has an active development with at least 2 full time developers (only for Open Source) 10 The LMS organization has an active support community (only for Open Source) For open source LMS, these additional preselection criteria have been proposed: The LMS organization has an active development, with at least 2 full time developers [12].
The LMS organization has an active support community [8,12].

LMS EVALUATION METHODS
In this section, two evaluation methods are presented for LMS evaluation. The criteria presented in these evaluation methods are different from the preselection criteria because they focus mainly on the LMS individual features, while the preselection criteria focus on ten general criteria that must be present in the LMS in order to be chosen out of the variety of software that claim to be a LMS.
This research proposes to use two evaluation methods to make sure the LMS selected is the one that gets the highest score in both evaluation activities. The two LMS evaluation methods used in this research are presented below. They were chosen among the available evaluation methods, due to their complete documentation and criterion for scoring the results.

LMS Evaluation Framework
The LMS Evaluation Framework [5] is based on the Software evaluation method by Hollander [15]. Following Beshears framework [5] the major criteria for selecting a LMS are: Known Requirements: Ability of the software to meet the university's current academic and administrative requirements, and future requirements that are currently known to exist. Unknown Future Requirements: Ability to modify the software to meet the university's new requirements as they become known. Implementability: Ability to implement the software easily. Supportability: Ability of the vendor to support both the software and the University in the future. Cost: Total cost to purchase and implement the software as well as ongoing maintenance and support costs. To help determine the relative importance of the major LMS selection criteria, the evaluators should allocate 100 points among the five criteria, with the most important receiving the highest rating and then, the criterion rating is used to weight the raw scores each LMS receives for that item.

LMS Evaluation Tool User Guide
This LMS evaluation method has been developed by the Commonwealth of Learning in 2004 [9], which follow four steps: completing LMS registry, evaluating the general criteria, rating the LMS functionalities and completing the results. It is possible to factor in, the differences in importance by specifying different weights for each criterion. The weight is a number between 1 and 5 (5 means the criterion is most important; 1, least important).
The general criteria are: Cost of Ownership, Maintainability, Usability and User documentation, User Adoption/ Vendor Profile, Openness, Standards Compliancy, Integration Capacity, Learning Object Metadata Integration, Reliability and Effectiveness, Scalability, Security, Hardware and Software Considerations and Multilingual Support. The set of feature-related evaluation criteria are: Administration, Security, Access, Integration with other systems, Course Design and Development, Course Monitoring, Assessment Design, Online Collaboration and Communications and Productivity Tools.
These two evaluation method all-together allow to evaluate a set of functional and non-functional requirements, becoming a valuable guidance to ensure that the final LMS selected will be the one with the best functional and non-functional requirements.

RATIONAL UNIFIED PROCESS
Following Krutchen [24] definition, RUP is a software engineering process. It provides a disciplined approach to assigning tasks and responsibilities within a development organization. Its goal is to ensure the production of high quality software that meets the needs of its end users within a predictable schedule and budget. RUP is also a process framework that can be adapted and extended to meet the needs of an adopting organization. Roles, artifacts, activities, guidelines, concepts and mentors are elements that you can add or replace to evolve or adapt the process to the organization's needs [24].
The decision to select RUP as a process framework in order to select, evaluate, integrate and deploy a LMS is based on the facility that this framework provides, so it can be tailored to suit the needs of the project.
Why to focus on LMS selection instead of developing one from scratch? Up to now, designing and deploying this kind of software is not an easy task, as this process is complex and involve a wide variety of organizational, instructional and technical components [13]. Even though, there are few efforts to lower time and cost in the design and development of LMS [4], there are also great risks involved in the designing of a LMS in an organization [16,32].
Examples of tailoring RUP for different kind of projects can be found in the literature [3,12,30,34]. Renaux et al. [31] show the advantage of using a component-based design as an approach to Model Driven Engineering (MDE), adapting RUP for this purpose. Similarly, Avgeriou et al. [2] adopt and customize great part of RUP to propose an architecture for LMS.
Other examples of tailoring RUP to be used as a process framework are: the case for modeled Internet applications [30] and the adaptation of RUP to integrate the human-computer interaction [34]. Bygstad et al. [7] have used RUP to try four patterns of information systems integration, since RUP handles technical integration and the external stakeholders. The patterns represent ideal types of how and when, the development of a project would have to include and to adapt the technology and stakeholders, and to stabilize the behavior of the network. One of the patterns is illustrated with the study of the integration of a LMS in a University Institution made during the period of 2001-2003 [7]; finding that, when integrating the LMS with the organization in the final stages of development, it produces the risk of resistance of the users and delays in the organizational deploy.

TAILORING RUP FOR LMS SELECTION
RUP is proposed as a process framework to select, evaluate, integrate and deploy a LMS that is already developed, but not customized and integrated into the organization. As it was mentioned before, there are dozens of LMS to choose from, so it is important to choose the right one for the organization requirements. As Fernández [13] pointed out, the selection of the LMS is a transcendental decision for the organization that has to be taken in the initial phase of the technological management for the distance education project.
It is therefore, than an extension is proposed at the level of disciplines in RUP. Disciplines are the containers used to organize activities in the process, so they are the place to start with, for introducing structural changes in RUP [24]. For this research in progress, the first step is to propose RUP additions to take into account the evaluation and selection activities in a LMS delivery in an organization.
To find out which disciplines will be used to start with, table 2, shows an analysis of activities related with LMS selection and the disciplines in the RUP.
As can be observed in table 2, the Analysis and Design Discipline is the one that is more related with the activities of preselection, evaluation and selection of a LMS, even though there are others disciplines that can be affected as well, as Test Discipline, Configuration and Change Management Discipline and Environment Discipline. For this research in progress, only the changes in the Analysis and Design discipline will be analyzed. The decision to select this discipline is based in these two points: Through this discipline, an architectural foundation analysis is performed as well as the collection of material to build and configure the software architecture. It allows defining the candidate architecture, refining the architecture and designing the system components that fit to the requirements previously established at the Requirements Discipline. The activities are added in two workflows: Perform a Architectural Synthesis and Define a Candidate Architecture. These activities are added in these workflows because the LMS is a technological element that is going to add constraints to the software architecture. Defining candidate solutions requires exposing mismatches and negotiating tradeoffs among the critical use cases and non-functional requirements, architectural and design constraints, project management constraints and risks.

Activities inserted in the workflow Perform a Architectural Synthesis
In the workflow Perform a Architectural Synthesis, one activity is inserted (Figure 1):

Search and collect an initial list of LMS:
Search for several LMS that fit the minimum LMS definition. This LMS list would help to demonstrate that there are feasible candidates solutions that can help to assemble an architectural proof-of-concept which can be examined in detailed in the Elaboration phase. This activity will produce a list of LMS with the main characteristics and would be performed by the software analyst and the instructional designer and can be done in the Inception phase. The following artifacts are used to identify the LMS for this initial list: Supplementary Specification, Use-Case Model (preliminary), Glossary, Software Requirements Specification, Business vision document and specialized sources for information.

RUP Discipline Justification Business Modeling
The purpose of the Business Modeling discipline is to: (1) understand the structure and the dynamics of the organization in which a system is to be deployed (the target organization), (2) understand current problems in the target organization and identify improvement potential, (3) ensure that customers, end users, and developers have a common understanding of the target organization, and (4) derive the system requirements needed to support the target organization. In this sense, the activities of preselection, evaluation and selection are not involved with the objective of this discipline. Nevertheless, the artifacts that are generated in this discipline give support to these activities, since devices such as Business Rules Document, Business Vision and Business Architecture Document are input artifacts in the preselection activity.

Requirements
The purpose of the Requirements discipline is to: (1) establish and maintain agreement with the customers and other stakeholders on what the system should do, (2) provide system developers with a better understanding of the system requirements, (3) define the boundaries of the system, (4) Provide a basis for planning the technical contents of iterations, and (5) Provide a basis for estimating the cost and time to develop the system. Even though the activities of evaluation and selection are not related to this discipline, the devices such as Vision document, supplementary specifications, software requirements, etc., that take place here are taken into account for the preselection and evaluation activities. The preselection activity could be related in some ways as it helps establishing the requirements of the system-to-be.

Analysis and design
Through this discipline an analysis of the candidate architecture is made to be used in the project. In this discipline the requirements are transformed into a design of the system-to-be and evolve a robust architecture for the system. In this sense an existing LMS already has an architecture, which must be compatible with the system-to-be with the purpose of obtaining the product in the smaller possible time. Therefore, the activities of preselection, evaluation and final selection, lead to the selection of the LMS that can be used for this aim. Implementation For this discipline, the architecture must be defined previously, so the activities of preselection, evaluation and selection had to already been done. This discipline would go into action for those requirements that must be developed, on the base of the architecture of the selected LMS, because the selected LMS does not contemplate them Test This discipline would be centered in proving the integration of additional features for the LMS. In addition, it would verify the satisfaction of the quality level for the LMS selected throughout the process. Deployment This discipline is very important to obtain the complete deployment of the LMS in the organization, since this includes the acceptance tests of the end product. Nevertheless, it is discipline does not have a direct relation with the selection of the LMS.

Configuration and Change Management
The purpose of the Configuration and Change Management discipline is to: (1) Identify configuration items . (2) Restrict changes to those items, (3) Audit changes made to those items and (4) Define and manage configurations of those items. In this discipline, the Configuration management plan document must include take into account the information Project management The purpose of the Project Management discipline is to: (1) Manage a software-intensive project, (2) Plan, staff, execute, and monitor a project and (3) Manage risk. Therefore the artifacts: Project Plan, Risk Management Plan and Risk List, must take into account the management of the risks involve in the process of acquiring the LMS and integrating them into the organization. Environment The purpose of the Environment discipline is to provide the software development organization with the software development environment-both processes and tools-that will support the development team. This includes configuring the process for a particular project, as well as developing guidelines in support of the project. Through some activities of this discipline it is possible "to configure" templates and guidelines that makes possible "to institutionalize" and therefore, to incorporate, the most important aspects to take into account at the time of making the selection of a LMS for a particular project. All this is reflected in the artifact "Project Specific Guidelines". This discipline is not directly related with the selection of the LMS; rather, it is the one that contributes to the selection of the LMS, as it does the discipline of requirements. Table 2. Analysis of activities related with LMS selection and the disciplines in the RUP [24].

Activities inserted in the workflow Define a Candidate Architecture:
In the workflow Define a Candidate Architecture, three activities are inserted ( Figure 2): 1. Search and select two LMS evaluation guides: Collect guides for LMS evaluation with available documentation and instructions for scoring the results and select two of them.

LMS Preselection:
From the initial list of LMSs that fit the LMS definition criterion, apply the preselection criteria and select a short list of four or three LMS that would support the architectural Proof-of-concept and make feasible the candidate solution.

Final LMS evaluation and selection:
Evaluate the few LMS preselected, apply two LMS evaluation methodologies and select the LMS with the higher scoring results for the final recommendation. These three activities will be performed by the software analyst in conjunction with the instructional designer or educational specialist. The following artifacts are needed to be reviewed for the proper completion of these activities: Literature sources specialized in LMS evaluation, Initial list of LMS, Preselection criteria, Business Rules Document, Business architecture document, Use-Case Model (preliminary), ADL SCORM certified products, Short list of preselected LMS, Use-Case Model, Tools, Software Requirements Specification document, Supplementary Specification document, Document describing the LMS evaluation guides.
Note that, every project must decide whether they are going to use commercial or open source LMS. For this decision made at the beginning of the search, will easy the process of searching and preselection the LMS. The list of LMS would be unique for every project as they depend strongly on the characteristics of each project, their requirements, budget, technological constrains, etc.  Table 3 summarizes the elements that are specific to the Analysis and Design Discipline in RUP for LMS selection. All these activities are new for RUP. Additionally, Table 4 shows a more complete table with activities, objectives, incoming artifacts, roles and output artifacts, to be added into this Discipline.   Table 4. Activities, objectives, roles and output artifacts, to be added into the Analysis and Design Discipline in RUP for LMS selection.

CASE STUDY: IESA
A pilot project was selected as a case study for testing the tailoring proposed in RUP for LMS evaluation, as it is proposed by DESMET [23]. DESMET is a methodology for evaluating methods or tools. It identifies nine evaluation methods and a set of criteria to help the evaluator choose the most appropriate one for his needs based on the different evaluation contexts the evaluating team may have to deal with [23].

Instituto de Estudios Superiores de Administración, IESA
IESA is an academic centre, independent of currents, economic, political, religious or governmental groups. It is dedicated to the education of the management, leaning in the investigation, as much in administration as in other disciplines, orienting his teaching towards the development of the management in publics and private organizations (www.iesa.edu.ve) [17].
The institution, nowadays, occupies a distinguished place between the institutions that compose the national educational system, as well as the specialized academic organizations in administration world wide. IESA is an institution that understands the management like all activity whose intention is to orient, to direct, to choose options, to assign resources, to supervise and to evaluate the actions that are undertaken and organized to reach certain objectives. When understanding the management this way, IESA is interested to know the reality and the managerial necessities for the public and private sector, for profit or non-profit organizations [18].

IESA LMS Requirements
IESA needed a LMS that allowed the centralized administration for the academia offer, the distribution of educational material and the control of the academic records for students in online courses.
The requirements of this LMS can be represented in the use case model depicted in Figure 3.

CASE STUDY ANALYSIS
In this analysis, only the results related to the selection and evaluation of the LMS are presented, because these are the activities that were added to RUP for LMS selection. The rest of the activities in each phase of the project and its artifacts were developed following the RUP and therefore they will not be shown all here. In the first proposed activity in the workflow Perform a Architectural Synthesis: Search and collect an initial list of LMS, a list of 13 LMS was developed.
The list was then reduced from 13 to 2 LMS, in the activity of LMS Preselection, when the preselected criteria were applied. These Preselection criteria are explained in a previous section in this article. The commercial LMS Microsoft Class Server 3.0 was included [21], because the institution had the license of this software and wanted to compare it with the other LMSs. This preselection process took into account the data collected in documents from vision, business model, requirements, use cases, business objectives and costs. The two preselected LMS were Sakai Project 1.0 and Moodle 1.4.2.

LMS evaluation and selection
Two methodologies were selected: LMS Evaluation Framework [5] and LMS Evaluation Tool User Guide [9], in the activity Search and select two LMS evaluation methodologies included in the workflow Perform a Architectural Synthesis. They were chosen between the available evaluation guides, due to their complete documentation and criterion for scoring the results.
On the basis of the LMS requirement for IESA, the document of vision, business model, use cases, business objectives and costs, the three LMS underwent and evaluation process based in the two selected evaluation methodologies. The results of this evaluation are presented next.

Evaluation with the LMS Evaluation Framework
The final results can be seen in Table 5. To each criterion, a weight was assigned according to its importance within the institute, assigning 0 (zero) for "Not used or not important", up to 40 (forty) for "extremely important". Each criterion was evaluated in a scale from 1 to 10. The percentage corresponds to the score with weight. It can be observed that Moodle 1.4.2 has the greater score with 66%, followed by Sakai Project 1,0 with 62%. The second evaluation is made to determine the final LMS.  Table 5. Results for the LMS evaluation based on the evaluation guidelines by Beshears [5].

Evaluation with the LMS Evaluation Tool
In Table 6 Table 6. Final results for the LMS evaluation base on the evaluation guidelines by the Commonwealth of Learning [9].
When applying these two LMS evaluation methodologies, the resulting LMS was Moodle 1.4.2. So, applying both methodologies aim at the same LMS. Moodle 1.4.2 is the LMS that is recommended to be used for this project.
It is important to emphasize that Moodle 1.4.2 passed three evaluations: firstly, in the LMS preselection activity, secondly, with the application of the first evaluation methodology and thirdly, with the application of the second evaluation methodology. The application of these activities in the early phases and in the discipline of Analysis and Design, allowed to recommend a LMS for the proper management of a virtual learning environment, which assures the quality of the final product.

LMS functional prototype
As a result of this project, using RUP for LMS selection, a product was obtained that is characterized for allowing: Administration of the academic offer, courses, students and educational resources. Control of the evaluation, allocation of tasks, assessment and testing. The asynchronous and synchronous communication through forums, mail and chat. The LMS interface customization according to the IESA website design. Customized access to students, professors and administrators. The creation of courses according to the instructional design of the IESA. Figure 4 shows the LMS interface for the administrator view. On the right side, it is possible to see the picture of the person who has just logged in. It also appears the available courses, the news, the calendar and the administration tools (see http://virtual.iesa.edu.ve/moodle/index.php). As it can be seen in the case of IESA, the selection of the LMS in early phases of the development process positively influence the following development phases, obtaining in this way, a successful solution that is adequate for the organization.

CONCLUSIONS AND FUTURE WORKS
This paper introduces an extension in the Rational Unified Process (RUP) in order to integrate the activities of selection and evaluation of LMS into this process framework. This RUP improvement takes into account the particular characteristics of LMS which present an intrinsic complexity by supporting virtual learning communities. This characteristic should be mainly considered in the selection, to guarantee a successful implementation and deployment.
During the incorporation of particular activities and artifacts into RUP, it was possible to reuse several artifacts proposed by this methodology. This result reveals an advantage of using a widely documented methodology. Two extra workflows are identified as required and incorporated into the complete development process. The two workflows are: Perform a Architectural Synthesis and Define a Candidate Architecture. Activities, roles and artifacts are added and defined. These two workflows belong to the Analysis and Design Discipline.
A case study was selected to try the RUP for LMS selection at IESA. The results show that Moodle 1.4.2 is selected over Sakai Project 1.0 and Microsoft Class Server 3.0. The LMS selected is a feasible candidate solution that ensures the exposing mismatches and negotiating tradeoffs among the critical use cases and non-functional requirements, architectural and design constraints, project management constraints and risks.
It is important to emphasize that the application in a real project proved the potentiality and confidence of the work performed in order to improve the quality of the selection process of a LMS to any organization.
As future works, the research will focus in tailoring the RUP for whole process of delivering a LMS in an organization.