To print a PDF copy of this article, click here.
The Defense Acquisition University (DAU) is an integral component in the career of every Defense Acquisition Workforce member, from the time they enroll in their first DAU course until they retire. One of the many keys to DAU’s success is its ability to measure the effectiveness of its training programs, monitor performance, and improve its curriculum. To this end, the authors conducted a data mining exercise within the training evaluation data to determine the key drivers of its success. This article explains the methodological approach used (structural equation modeling) as well as the results, recommended actions, and outcomes. Within the DAU learning enterprise, more than 326,000 training events were evaluated during 19 months between January 1, 2008, and July 30, 2009. Results indicate that DAU’s learning enterprise positively influences job impact and business results.
The Defense Acquisition University (DAU) is critical to ensuring the Defense Acquisition Workforce is well trained to meet our nation’s needs. As such, DAU is fully integrated in the careers of its workforce from the time they enroll in their first DAU course until they retire. One of the many keys to DAU’s success is its ability to measure the effectiveness of its training programs, monitor performance, and improve its curriculum (DAU, 2010). To this end, DAU conducted a data mining exercise with its training evaluation data to determine the key drivers of its success.
DAU measures and monitors its own performance by administering a state-of-the-art, end-of course survey instrument, which is a Web-based learning evaluation system with an extensive database of performance benchmarks collected from student survey data. DAU evaluates customer satisfaction based on the 4-level Kirkpatrick training assessment model and uses survey items on a 7-point Likert-type scale (Kirkpatrick, 1998). Students are provided a link to the survey at the end of each course, which includes questions on course content, quality of faculty, and job applicability. Ratings are reviewed regularly, and improvements are made in DAU’s learning products and services based on these evaluations.
This study focuses on the evaluation of key drivers for successful training events. An advanced statistical approach called causal modeling (i.e., structural equation modeling) was used to determine relationships among latent constructs and isolate likely paths of causation. The main objectives of this research study were fourfold.
- Evaluate the survey instrument DAU uses and determine whether or not it is a valuable instrument to provide information for decision making;
- Assess the relationship between job performance and impact as perceived by participants attributed to DAU training;
- Access the antecedents and outcomes of learning and provide a benchmarking analysis of DAU scores versus other organizations; and
- Provide recommendations for isolated points of intervention, which will yield the largest improvements for the learning enterprise.
Development of Hypotheses
Although the investment in learning by various organizations is far from consistent across industries (or even across departments within the same organization), few would argue as to its importance. The resultant training and development budget isolates this investment and is often referred to when senior leaders are questioned as to their commitment for human capital development. The two primary expenditures related to training are instruction and materials. High levels of instructor quality are synonymous with effective learning. This positivist relationship is at the foundation of why instructors are continually evaluated. The performance of a teacher as expressed by a student after a course is complete is a universally adopted founding tradition of every educational institution. For this reason, a common expectation is the existence of a positive relationship between instructor effectiveness and individual learning.
Hypothesis No. 1
There is a positive relationship between instructor effectiveness and individual learning. In addition to the quality of instruction, courseware quality is also an important antecedent to learning. Whether the materials are physical in the form of books and notes, or online, students grasp difficult concepts by reading them over and over again. While the instructor may reinforce the importance of the text, the explicit documents act as a permanent record of the content that a student is expected to master. As such, four additional important hypotheses regarding business results and impact were also tested.
Hypothesis No. 2
There is a positive relationship between courseware quality and individual learning. Given the assumption that an instructor is competent and that course materials are adequate, students often have pre-conceived notions with regards to the value of a course before it has been completed. To be accurate, the perception of a worthwhile investment is more than just the cost of the registration fee. In most cases, the opportunity cost of time while sitting through a course (and therefore, not doing the job) is often more valuable to the learner. Only when both these perceptions (i.e., the cost of registration and the opportunity cost of time) are deemed to be fair and adequate, can a student realize a satisfactory learning experience. Therefore, the perception of a worthwhile investment is also expected to have a positive relationship with individual learning.
Hypothesis No. 3
There is a positive relationship between worthwhile investment and individual learning. Sustainable high levels of organizational performance can be attributable in large part to a superior learning enterprise that transforms human capital development into actionable job impact and business results (Bontis & Fitz-enz, 2002). It follows then that an expected positive relationship between individual learning and job impact and business results should be realized. The issue here is one of temporal lag. How do learners know for sure if a course will impact their job later? One way to deal with this limitation is to assess the outcomes longitudinally. In other words, provide learners with an opportunity to predict an outcome immediately after the course is complete, and then again sometime into the future, a retrospective analysis. As such, the following important hypotheses regarding business results and impact were also tested:
Hypothesis No. 4
There is a positive relationship between individual learning and future job impact.
Hypothesis No. 5
There is a positive relationship between individual learning and future business results.
Hypothesis No. 6
There is a positive relationship between individual learning and actual job impact.
Hypothesis No. 7
There is a positive relationship between individual learning and actual business results.
When analyzing large sets of data, a variety of statistical techniques are available. One common approach used by researchers is null hypothesis testing with experimental and quasi-experimental designs (Shadish, Cook, & Campbell, 2002). An alternative approach is data modeling (Rodgers, 2010). Structural equation modeling (SEM) is one such approach and is useful for many reasons. Foremost is its ability to test multiple hypotheses simultaneously and produce a visual model of the causal relationships within a data set. While the benefits of SEM are plentiful, a significant level of interpretation of these models is necessary. In its simplest form, SEM is an advanced statistical technique that computes the mathematical relationships among multiple variables simultaneously in order to describe a chain of causation.
The measurement and structural models were estimated by using Partial Least Squares (PLS). PLS is a second generation SEM technique that has received positive recognition in the scientific community (Chin, 1998; Gefen, Straub, & Boudreau, 2000). PLS was developed by Wold (1975), and it focuses on maximizing the variance of the dependent variables explained by the independent ones. It is a rigorous SEM technique that requires only minimal assumptions about the distribution of the data. PLS has five main advantages over other covariance methods (e.g., LISREL,1 AMOS,2 etc.) for this research study: (a) it does not assume normally distributed raw data; (b) the presence of multicollinearity in the data is handled well; (c) it is better suited to explain complex exploratory relationships; (d) it allows variable weights to scale for indicators; and (e) it allows the use of noninterval scales. The raw data set that was to be analyzed fit well with the corresponding advantages of PLS. Furthermore, PLS is often used in exploratory research with the ultimate goal to maximize the explanatory power of the resultant model. PLS also benefits from considering all path coefficients simultaneously, allowing analysis of direct, indirect, and spurious relationships and the estimation of multiple individual item loadings in the context of a theoretically specified model rather than in isolation (enabling researchers to avoid biased and inconsistent parameter estimates for equations).
Whereas in traditional path analysis the calculation of reliability and validity statistics is independent of the model being tested, PLS generates a variety of reliability and validity statistics calculated in the context of the theoretical model under investigation. To validate the measurement model, the authors executed the following series of steps. First, construct reliability was assured by calculating Cronbach’s alpha values for each construct. Cronbach’s alpha is often used to confirm that respondents interpreted the meaning of survey items accurately, and would continue to do so in the future. In other words, survey items were understood by the respondents and tended to hang together in a cluster. A Cronbach’s alpha value exceeding 0.70 is considered the minimum threshold (Cronbach, 1951). Table 1A outlines that all items and their corresponding constructs used in this study exceeded 0.70 (in fact, most exceeded 0.90).
Table 1A. Measures of Reliability and Validity for Exogenous Constructs
|ID||Metric (Base, DAU)||Lamda|
|Instructor||Effectiveness (Alpha = 0.946, 0.931)||Base||DAU|
|1058P||The instructor was knowledgeable about the subject||0.895||0.910|
|1059P||The instructor was prepared and organized for the class||0.856||0.923|
|1269P||The instructor was responsive to participants’ needs and questions.||0.856||0.913|
|1270P||The instructors’ energy and enthusiasm kept the participants actively engaged||0.851||0.906|
|Courseware Quality (Alpha = 0.899, 0.798)|
|1065P||The examples presented helped me undertand the content||0.886||0.805|
|2726P||The scope of the material was appropriate to my needs||0.891||0.806|
|2730P||The participant materials (manual, handouts, etc.) will be useful on the job.||0.845||0.800|
|2924P||The material was organized logically||0.892||0.779|
|Worthwhile Investment (Alpha = 0.958, 0.971)|
|2743P||The training was a worthwhile investment in my career development||0.980||0.986|
|2744P||This training was a worthwhile investment for my employer||0.979||0.986|
|Individual Learning (Alpha = 1.000, 1.000)|
|919P||I learned new knowledge and skills from this training||1.000||1.000|
Loading values (lamdas) were used to measure the validity of items. This test examines whether or not the survey items used to evaluate the effectiveness of the instructor (four items in this case) load on to an overall construct about the instructor. The opposite would be if a survey item used to measure the quality of instructors actually did a better job of measuring courseware quality. Again, a measure of 0.70 or higher is desired, and this minimum threshold was exceeded in all cases (Nunnally, 1978). The results of Table 1B illustrate that both the survey instrument and DAU models used valid and reliable measurement instruments. In essence, the psychometric evaluation of the scales used in this study was successful and therefore adequate for model interpretation.
Table 1B. Measures of Reliability and Validity for Endogenous Constructs
|Instructor||Perceived Future Job Impact (Alpha = 0.832, 0.833)||Base||DAU|
|712P||I will be able to apply the knowledge and skills learned in this class to my job.||0.726||0.749|
|1279P||How critical is applying the content of this training to your job success? 0% – 100%||0.786||0.925|
|1423P||What percent of your total work time requires the knowledge or skills presented in this training? 0% – 100%||0.840||0.898|
|2788P||What percent of new knowledge and skills learned from this training do you estimate you will directly apply to your job? 0% – 100%||0.873||0.907|
|Perceived Future Business Results (Alpha = 0.775, 0.805)|
|2470P||Estimate how much you expect your job performance related to the course subject matter to improve in the next 12 months. 0% – 100%||0.909||0.928|
|2741P||Based on your response to the prior question, how much of the improvement will be a direct result of this training, as opposed to other factors?||0.898||0.901|
|Job Impact in 60 Days (Alpha = 0.892, 0.929)|
|1737F||What percent of your total work time have you spent on tasks that require the knowledge/skills presented in the training? Check only one. 0% – 100%||0.889||0.932|
|1738F||On a scale of 0% (not at all) too 100% (extremely critical), how critical was applying the content of the training to your job success? Check only one.||0.916||0.934|
|Business Results in 60 Days (Alpha = 0.708, 0.814)|
|2751P||Given all factors, including this training, estimate how much your job performance related to the course subject matter has improved since the training. 0% – 100%||0.906||0.931|
|2502F||Based on your response to the prior question, estimate how much of the improvement was a direct result of this training. 0% – 100%||0.845||0.905|
Survey Instrument vs. DAU Model Interpretation
In 2009, Dr. Nick Bontis created a predictive learning analytics model that describes the relationship between training and business performance (Bontis & KnowledgeAdvisors, 2009). The analysis proved successful and the resulting model is depicted in Figure 1. As one might expect, the model is complex. However, the model is relatively easy to decipher with one key piece of information. The chain of causation lies along the pathways with the highest beta values.
Figure 1. Predective Learning Analytics Model
Before examining the model, a description of the data source is essential. Data used in the analysis were extracted from the survey instrument system. Using Web-hosted surveys, learners provided feedback about their training experience immediately after training and 60 days after training. The immediate survey asked learners to rate the quality of their training experience as well as predict whether training would improve their job performance and, in turn, contribute to business results. In the model shown in Figure 1, the immediate survey results contribute to the following factors about training: Instructor Effectiveness, Courseware Quality, Worthwhile Investment, and Individual Learning.
The immediate responses also contributed estimates of future job performance and estimated impacts on business results: Perceived Future Job Impact and Perceived Future Business Results. The far right side of the model has two factors, Job Impact in 60 Days and Business Results in 60 Days, which represent retrospective input gathered from learners after they have had 60 days to apply their learning on the job. More than a million data points were used in the survey instrument base model with learners assessed from many well-known, globally recognized companies (e.g., Microsoft, HSBC, Caterpillar, and BAE).
The left side of the model represents the three most important antecedent aspects of training: Worthwhile Investment, Courseware Quality, and the learner’s perspective about Instructor Effectiveness. But how do these three factors contribute to individual learning? Effective instructors contribute by developing high-quality courseware (ß = 0.556) (e.g., materials, delivery format, learning environment, etc.). Both contribute to the learner’s perception that training was a worthwhile investment (ß = 0.625). While each factor also has a direct relationship with individual learning, the strongest relationship is through the last factor—worthwhile investment (ß = 0.483).
The second half (right side) of the model represents how individual learning influences job performance and business (organizational) outcomes. By examining the values, we see from the nondominant (lowest values) paths that individual learning does not have a strong direct effect on Job Impact and Business Results directly (both far right) 60 days after training. However, by following the dominant paths, we see that Individual Learning leads to Perceived Future Job Impact, actual Job Impact in 60 Days and Business Results in 60 Days. This path indicates that knowledge for the sake of knowledge (Newman, 1947) is not sufficient. Training must be perceived as relevant, practical, and applicable before learners will indicate it will (and does) have an impact on performance and eventually business results. The entire causal pathway for the model looks like a giant N, starting at the bottom left, rising to the top left, slanting diagonally to the bottom right, and then up to the top right.
Does the model effectively describe the relationship among variables? Yes, the model has a relatively high explanatory power for predicting Job Impact in 60 days (R2 = 40.0%) and Business Results in 60 days (R2 = 59.2%). A model that predicts the structure and relationships perfectly would have an R2 value of 100 percent, although this situation is virtually impossible to achieve in social science research. To put this measure in perspective, consider that it is an algorithm you could use at a casino. Given the value of the cards in your hand (e.g., Instructor Effectiveness, Courseware Quality, etc.), you would win 59.2 percent of the time.
This predictive learning analytics model (KnowledgeAdvisors, Inc.) is relevant to DAU because it serves as a base model (derived from a data set of over 1 million surveys industry-wide) for benchmarking purposes. As a result, the DAU can compare its model and results to the KnowledgeAdvisors’ predictive learning analytics model to determine where it can improve and where it is outperforming other organizations.
The DAU Model
Each year the DAU collects hundreds of thousands of evaluations after training events to determine whether the curriculum and its outcomes were effective. Immediate postcourse evaluations are deployed as well as 60-day follow up evaluations. In this way, learners provide feedback about the quality of the course and predict whether they will apply what they learned. On the 60-day follow up evaluation, learners indicate whether training contributed to improved job performance and business results. For this study more than 326,000 evaluations were collected during 19 months between January 1, 2008, and July 30, 2009.
Figure 2 shows the DAU model with KnowledgeAdvisors benchmark values above the DAU model values. The analysis reveals many important facts about the DAU’s curriculum.
Figure 2. Predictive Learning Analytics Model – DAU and Survey Instrument Model Benchmark
First, the causal chain depicted in the model explains the relationships among the data well. In fact, the model fits the DAU’s data better than the survey instrument benchmark data. (This is not unusual; the model will fit some data sets better or worse than the benchmark.) The model predicts 52.7 percent of the Job Impact in 60 Days and 67.9 percent of the Business Results in 60 Days. The values for the KnowledgeAdvisors Benchmark data are lower at 40.0 percent and 59.2 percent, respectively. These results indicate that the key aspects of training that drive job impact and business performance hold true for both the KnowledgeAdvisors Benchmark data and the DAU data. As a refresher, Instructor Effectiveness links to Courseware Quality, which links to Worthwhile Investment. All three in that order optimize Individual Learning. In turn Individual Learning leads to Perceived Future Job Impact, Job Impact in 60 Days, and then Business Results in 60 Days in that order. To improve the effectiveness of courses—at least in terms of increasing job performance and business impact—DAU should focus on Instructor Effectiveness, Courseware Quality, and ensure that learners perceive that training is a Worthwhile Investment. Improvement actions will be discussed later in this article.
Second, the strength of the causal relationships is somewhat stronger for the DAU compared to the KnowledgeAdvisors Benchmark for five relationships (arrows). These are designated with a superscripted a after the value. For only one relationship, the link between Courseware Quality and Individual Learning, the DAU value is lower than the survey instrument benchmark and is indicated by a superscripted b. When this relationship was examined in more detail, it was discovered that Courseware Quality was more important for younger learners and less so for older learners. Younger learners preferred e-learning, whereas older learners preferred traditional classrooms and effective instructors. Interestingly, younger learners indicated that training had a greater impact on job impact and business impact 60 days after training.
Third, DAU instructors have a strong influence on Individual Learning and eventually Job Impact and Business Results. In fact, when compared to the KnowledgeAdvisors benchmark (0.083), the relationship between Instructor Effectiveness and Individual Learning (ß = 0.163) is almost twice as large for DAU. A stronger relationship between Instructor Effectiveness and Courseware Quality still exists, but by comparing the magnitude of the relationship between the DAU and the benchmark, clearly, instructors hold more influence within DAU than at other organizations.
Fourth, guest speakers also impact learning. When guest speakers taught courses, higher levels of Individual Learning occurred. When guest speakers were not included, Job Impact and Business Results were generally lower than the survey instrument benchmark.
Fifth, application is a critical element to successful courses. High job application scores were linked to high learning scores, extremely high Job Impact scores, and Business Results scores.
Sixth, application is also strongly linked to whether learners recommend courses for future learners. When recommendation scores were low, the Business Results in 60 Days were also lower. This is a strong indicator that the course is not meeting individual needs and organizational needs and therefore should be revised or retired.
Seventh, Defense Acquisition Workforce Improvement Act (DAWIA) levels were investigated to test for their influence on training outcomes. DAWIA specifies three skill levels (Basic, Intermediate, and Advanced) associated with 13 career fields in the acquisition system. No consistent pattern of influence emerged across the model for the three DAWIA levels.
Eighth, the educational level of learners (e.g., high school, college, or graduate school) does influence outcomes of the model. Learners with some graduate education are more critical of instructors and appreciate good courseware. Learners with a high school education have the lowest perception that training is a Worthwhile Investment, but yield the highest response to Individual Learning. Interestingly, this group scored much lower than the benchmark regarding future results.
Lastly, the DAU’s course offerings and their influence on job performance and outcomes were evaluated for longitudinal improvement. Indeed, it was confirmed that scores improved from 2008 to 2009. In fact, scores for every category improved except for Instructor Effectiveness, which was already high.
An important and useful finding of this study indicates that the key aspects of training drive Job Impact and Business Results. This in itself is valuable, but such value quickly fades if insights cannot be turned into action to improve the curriculum. Table 2 provides a summary of the results of this study as well as recommended actions. If the DAU pursues these actions, the curriculum is likely to improve as evidenced by improved scores on the training evaluations.
Table 2. Recommended Actions
|Application is a critical element of training||As appropriate, DAU can improve its impact on job performance and business results by increasing the opportunities to apply what is learned during training.|
|Courseware Quality is more important for younger learners||To improve learning among younger learners, invest in self-study modules and quality courseware.|
|DAU instructors have a strong influence on older learners||For instructor-led courses, especially with older learners as the target audience, invest time and effort to find high-quality instructors who can effectively teach the materials regardless of the quality of the courseware.|
|Guest speakers also impact learning||When appropriate, use guest speakers to augment or lead instructor-led courses for older learners. Guest speakers tend to have more impact on learning than the standard cadre of instructors.|
|Learners recommend effective training||Use the question, “I would recommend this training to other learners” as a leading indicator of the quality of training and whether it will lead to job performance and business impact. If the rating for this question is low for a given course, it should be revised with a focus on improving the ability to apply what is learned during training.|
|DAWIA levels do not influence training impacts||When building DAU courses, it is not necessary to consider the DAWIA level of the audience. Other factors like age and education are more influential than DAWIA levels.|
|Education impacts learning and performance||To ensure that training leads to performance and future results, courses should be tailored to the educational level of the audience.|
In its evolution, DAU has broadly embraced adult learning designs in its formal courses and accepted the fact that adults learn best “by doing,” whether in the formal learning environment or on the job. With formal training, DAU attempts to “train as the workforce should work,” and prepares the workforce to “work as they are trained” by using the same training tools and learning assets at their individual places of work that they formerly used in the classroom. This study provides strong evidence that the key aspects of DAU’s approach to training drive Job Impact and Business Results. Having empirical evidence derived from relatively large data sets is very useful in rationalizing the cost of training regarding improving performance on the job. Additionally, confirming the reliability and validity of the survey instrument is important to any DAU curricula and recourse decisions based on survey instrument scores as well as other considerations.
Given that Job Impact results were based on self-reporting perceptions and not an independent external measure (not within the instrument), the outcome is still very strong in its implications, largely due to the size of the sample as well as previous relationship studies concerning the close relationships between measured perception and actual reality. Dess and Robinson (1984) indicate that such perceived measures of business results are reasonable surrogates for more tangible and objective measures of business outcomes (e.g., revenue growth, profits). Others (Geringer & Hébert, 1989; Hansen & Wernerfelt, 1989; Lyles & Salk, 1997; Venkatraman & Ramanujam, 1987) have demonstrated that such perceived measures also are positively correlated with objective financial performance metrics.
Finally, the recommendations discussed previously are now being incorporated within DAU course development and course update design strategies. Therein lies the power of: (a) benchmarking learning data across a very large set of comparative peer organizations; and (b) using structural equation modeling to ascertain specific points of intervention for evaluating and improving the learning enterprise, thereby assuring a healthy return on investment for training dollars.
To print a PDF copy of this article, click here.
Nick Bontis, Ph.D., is a 3M National Teaching Fellow and an award-winning professor at the DeGroote School of Business, McMaster University. He is recognized as the world’s leading expert on intellectual capital and its impact on performance. He was recently named as one of the Top 30 Management Gurus World-wide and is a leading management consultant and keynote speaker. His most recent book is entitled Information Bombardment: Rising Above the Digital Onslaught.
(E-mail address: email@example.com)
Chris Hardy, Ph.D., is the director of the Global Learning and Technology Center of the Defense Acquisition University (DAU). He is co-author of Leading a Learning Revolution: The Story Behind DAU’s Reinvention of Training (2007). Prior to joining the faculty at DAU, he served both in government and private industry, including a tenure as vice president, INNOLOG, Inc., a global supply chain logistics company headquartered in McLean, Virginia. Dr. Hardy holds a Ph.D. from Virginia Polytechnic Institute and State University.
(E-mail address: firstname.lastname@example.org)
John R. Mattox, II, Ph.D. is the Director of Research at KnowledgeAdvisors. Prior to joining KnowledgeAdvisors in March 2010, John led training evaluation teams at KPMG, PricewaterhouseCoopers and Arthur Andersen for nearly 12 years. Academically, he received his undergraduate degree in psychology from The Johns Hopkins University and his doctoral degree in Experimental Psychology (Quantitative and Industrial/Organizational) from The University of Memphis.
(E-mail address: email@example.com)
Bontis, N., & Fitz-enz, J. (2002). Intellectual capital ROI: A causal map of human capital antecedents and consequents. Retrieved from http://www.Bontis.com/ic/publications/JICBontisFitz-enz.pdf
Bontis, N., & KnowledgeAdvisors, Inc. (2009). The predictive learning impact model. Retrieved from http://www.NickBontis.com/BontisPredictiveLearningImpactModel.pdf
Chin, W. (1998). Issues and opinion on structural equation modeling. MIS Quarterly, 22(1), 7–16.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
Defense Acquisition University. (2010). Defense Acquisition University 2011–2016 strategic plan, FY11 organizational performance plan. Retrieved from http://www.dau.mil/aboutDAU/default.aspx
Dess, G. G., & Robinson, R. B. (1984). Measuring organizational performance in the absence of objective measures: The case of the privately held firm and conglomerate business unit. Strategic Management Journal, 5(3), 265–273.
Gefen, D., Straub, D. W., & Boudreau, M. (2000). Structural equation modeling and regression: Guidelines for research practice. Communications for the Association of Information Systems, 4(7).
Geringer, J., & Herbert, L. (1989). Control and performance of international joint ventures. Journal of International Business Studies, 20, 235–254.
Hansen, G., & Wernerfelt, B. (1989). Determinants of firm performance in relative importance of economic and organizational factors. Strategic Management Journal, 10(5), 399–411.
Kirkpatrick, D. L. (1998). Evaluating training programs: The four levels (2nd ed.). San Francisco, CA: Berrett-Koehler Publishers.
Lyles, M., & Salk, J. (1997). Knowledge acquisition from foreign partners in international joint ventures: An empirical examination in the Hungarian context. In P. Beamish & P. Killing (Eds), Cooperative Strategies: European Perspectives. San Francisco, CA: New Lexington Press.
Newman, J. H. (1947). The idea of the university. New York, NY: Longmans, Green, and Co.
Nunnally, J. C. (1978). Psychometric theory. New York, NY: McGraw-Hill.
Rodgers, J. L. (2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65(1), 1–12.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.
Venkatraman, N., & Ramanujam, V. (1987). Planning system success: A conceptualization and an operational model. Management Science, 33(6), 687–705.
Wold, H. (1975). Path models with latent variables: The NIPALS approach. In H. M. Blalock, A. Aganbegian, F. M. Borodkin, R. Boudon, & V. Capecchi (Eds.), Quantitative sociology: International perspectives on mathematical and statistical modeling (pp. 307-357). New York, NY: Academic.
1. LISREL, an acronym for linear structural relations, is a statistical software package used in structural equation modeling. LISREL was developed in the 1970s by Karl Jöreskog, then a scientist at Educational Testing Service in Princeton, NJ, and Dag Sörbom, later both professors of Uppsala University, Sweden.
2. AMOS, an acronym for analysis of moment structures, is designed primarily for structural equation modeling, path analysis, and covariance structure modeling, though it may be used to perform linear regression analysis. It features an intuitive graphical interface that allows the analyst to specify models by drawing them. It also has a built-in bootstrapping routine and superior handling of missing data. It reads data from a number of sources, including MS Excel® spreadsheets.