University of Ciego de Ávila Máximo Gómez Báez

ISSN: 2309-8333

RNPS: 2411

|13(1) |2025|

This is an Open Access article under the license CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/)

Estrategia y Gestión Universitaria EGU

Scientific and technological

research article

How to cite:

Guerra Bretaña, R. M.,

Pupo

Méndez, K., & Ramos Azcuy, F. J. (2025).

Improvement of an instrument to evaluate

the impact of postgraduate training in the

Chair of Quality, Metrology and

Standardization.

Estrategia y Gestión

Universitaria

, 13(1), e8827.

https://doi.org/10.5281/zenodo.15065643

Received: 26/02/2025

Accepted: 14/03/2025

Published: 31/03/2025

Corresponding author:

mayelin@biomat.uh.cu

Conflict of interest:

the authors declare

that they have no conflict of interest,

which may have influenced the results

obtained or the proposed interpretations

Improvement of an instrument to

evaluate the impact of postgraduate

training in the Chair of Quality,

Metrology and Standardization

Perfeccionamiento de un instrumento

para evaluar el impacto de la formación

de posgrado en la Cátedra de Calidad,

Metrología y Normalización

Aperfeiçoamento de instrumento para avaliação

do impacto da formação pós-graduada na Cátedra

de Qualidade, Metrologia e Normalização

Abstract

Introduction: evaluating the impact of training is a scientific

and academic challenge due to the need for continuous

improvement of program quality in terms of relevance and

academic excellence. Objective: to analyze the results of the

refinement of an instrument for evaluating the impact of

postgraduate academic training. Method: the instrument for

evaluating the impact of training, refined and implemented in

the 10th Edition of the Master's Program in Quality and

Environmental Management, is a modification of the

Kirkpatrick Model to include the program's societal impacts.

Results: the refinement of the instrument focuses on

specifying the strategic sectors where the impact occurs, as

well as the levels of introduction and potential generalization

of the achieved results. Its application demonstrates the

positive effects of the program on the performance of

graduates and their organizations, as well as the incorporation

of research results in seven strategic sectors for the country's

economic and social development. Conclusion: the developed

instrument for evaluating the impact of academic training

allows for understanding the positive effects of the evaluated

program in personal, organizational, and social domains.

Keywords: impact, evaluation, training, postgraduate

Resumen

Introducción: la evaluación del impacto de la formación es un

reto científico y académico ante la necesidad de mejorar

continuamente la calidad de los programas desde su

pertinencia y excelencia académica. Objetivo: analizar los

resultados del perfeccionamiento de un instrumento para la

evaluación del impacto de la formación académica de

posgrado.

Rosa Mayelín Guerra Bretaña

Universidad de La Habana

https://orcid.org/0000-0002-0561-6678

mayelin@biomat.uh.cu

Cuba

Karen Pupo Méndez

Universidad de La Habana

https://orcid.org/0000-0002-4628-9570

karen.pupo@biomat.uh.cu

Cuba

Fridel Julio Ramos Azcuy

Pontificia Universidad Católica del Ecuador

https://orcid.org/0000-0001-5945-446X

fjramosa@pucesm.edu.ec

Ecuador

Estrategia y Gestión Universitaria

ISSN

: 2309-8333

RNPS:

2411

13(1) | Enero-Junio |2025|

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

Método:

el instrumento para la evaluación del impacto de la formación,

perfeccionado e implementado en la Edición 10 del Programa de Maestría en

Gestión de la Calidad y Ambiental, es una modificación del Modelo de Kirkpatrick

para incluir los impactos del programa en la sociedad.

Resultados:

perfeccionamiento del instrumento radica en especificar los sectores estratégicos

en los que se produce el impacto, así como los niveles de introducción y potencial

generalización del resultado obtenido. En su aplicación se evidencian los efectos

positivos del programa en el desempeño de los egresados y sus organizaciones,

así como la introducción de los resultados de las investigaciones realizadas en

siete sectores estratégicos para el desarrollo económico y social del país.

Conclusión:

el instrumento desarrollado para la evaluación del impacto de la

formación académica, permite conocer los efectos positivos del programa

evaluado en los ámbitos personal, organizacional y social.

Palabras clave:

impacto, evaluación, formación, posgrado

Resumo

Introdução: a avaliação do impacto da formação é um desafio científico e

acadêmico devido à necessidade de melhoria contínua da qualidade dos programas

em termos de relevância e excelência acadêmica. Objetivo: analisar os resultados

do aperfeiçoamento de um instrumento para a avaliação do impacto da formação

acadêmica de pós-graduação. Método: o instrumento para avaliação do impacto

da formação, aperfeiçoado e implementado na 10ª Edição do Programa de

Mestrado em Gestão da Qualidade e Ambiental, é uma modificação do Modelo de

Kirkpatrick para incluir os impactos do programa na sociedade. Resultados: o

aperfeiçoamento do instrumento concentra-se na especificação dos setores

estratégicos onde o impacto ocorre, bem como nos níveis de introdução e

potencial generalização dos resultados alcançados. Sua aplicação evidencia os

efeitos positivos do programa no desempenho dos egressos e de suas organizações,

assim como a incorporação dos resultados das pesquisas realizadas em sete setores

estratégicos para o desenvolvimento econômico e social do país. Conclusão: O

instrumento desenvolvido para a avaliação do impacto da formação acadêmica

permite compreender os efeitos positivos do programa avaliado nos âmbitos

pessoal, organizacional e social.

Palavras-chave:

impacto, avaliação, formação, pós-graduação

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

Introduction

The training impact evaluation allows for understanding the effects

attributable to a training program in relation to objectives concerning the

development of individuals, institutions, and society (Díaz Leyva & Marrero Fornaris,

2021; Guerra Castillo, 2021; Parra Robledo & Ruiz Bueno, 2020). It also contributes

to improving the training process and ultimately identifying the qualitative and

quantitative benefits that training generates. Thus, impact evaluation is one of the

activities that help assess the quality of the training provided, both in terms of its

relevance and from the perspective of academic excellence. In Cuba, the Evaluation

and Accreditation System for Master’s Programs has included social relevance among

the characteristics of program quality since its early versions, necessitating the

systematic application of a tool capable of evaluating training impact.

An evaluation is a systematic process that involves the collection,

processing, analysis, and interpretation of data. “It is based on specific criteria

designed to determine the level of achievement of objectives and to provide

feedback on the successes and failures of the training process” (Ramos Azcuy &

Guerra Bretaña, 2023, p. 11). The training impact evaluation is considered to have

two fundamental moments: the evaluation of learning and the transformation that

occurs when the learned material is applied in the workplace, that is, when

knowledge is incorporated into organizational practices (Stable Rodríguez & Núñez

García, 2021; Gómez Miranda; 2023; Jiménez-Pitre et al., 2023). In evaluating

training impact, factors that influence the transfer of acquired knowledge to the

workplace and the role performed by program graduates must also be considered.

The characteristics of the work environment and the participants themselves

influence the achievement of the expected impacts of the training program (Guerra

Bretaña et al., 2017).

The Kirkpatrick model, initially published in 1959, has gained the most

popularity for evaluating training impact in organizations and for managerial training

(Nawaz et al., 2022). In this model, effects are evaluated at four levels: Reaction,

Learning, Transfer, and Results. The Phillips model, initially published in 1997, is

inspired by Kirkpatrick’s postulates but takes a more quantitative approach,

including a fifth level based on calculating return on investment (Guerra Castillo,

2021). Another adaptation of the Kirkpatrick model was made by Watkins in 1998,

contributing a fifth stage for evaluating social impact (Ramos Azcuy et al., 2016).

Cahapay (2021) has identified some limitations in how evaluators apply the

Kirkpatrick model in higher education, such as primarily using only its lower levels,

failing to consider all the contextual characteristics of the evaluated program, and

the scarcity of evidence regarding causal interconnections between the model’s

levels. This author also noted that evaluating impact solely through participants’

perceptions can be subjective and diminish the reliability of results. Direct

observation of changes in behavior, collection of verifiable data on results

attributable to training, and surveys of employers can contribute to a better

evaluation of impact.

Despite the aforementioned considerations, the Kirkpatrick model can be

adapted and enriched to understand the effects generated by provided training as a

means to improve the quality of programs in educational organizations (Alsalamah &

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

Callinan, 2021). The limitations raised in applying the model in higher education

should be viewed not as obstacles but as opportunities and challenges for evaluators.

The effects of training on graduates in terms of performance improvement, prestige,

and professional development are relatively easy to measure. However, measuring

impacts on organizations and society presents a significant research challenge.

Evaluating the profitability of training is particularly difficult, requiring quantitative

data on incurred expenses and the gains obtained as a result of the training provided.

For this reason, most training impact evaluations are confined to the first three

levels of the Kirkpatrick model (Reaction, Learning, and Transfer), without

considering organizational performance indicators or economic benefits.

Shewchuk et al. (2023) conducted a systematic review on how training

programs focused on knowledge application have been evaluated. They found that,

to gauge participants’ reactions, evaluators employed both quantitative methods

(surveys) and qualitative methods (interviews and focus groups), investigating

satisfaction levels with the program and its specific components, the organization of

the curriculum, the competencies of instructors, and the utility and relevance of the

training for participants’ job performance.

Bahl et al. (2024) studied the relationships among the levels of the

Kirkpatrick model, finding that positive participant reactions significantly influence

learning and that behavior, as measured through job performance, has a positive

impact on outcomes. These relationships were particularly strong in managerial

training. Other authors have also successfully utilized the Kirkpatrick model to

evaluate the effectiveness of training programs in healthcare (Firooznia et al., 2020)

and pharmaceutical services (Yi et al., 2020).

Stable Rodríguez & Núñez García (2021) established a methodology to

evaluate the impact of training in scientific and technological information

organizations, consisting of three phases: evaluation preparation, evaluation

implementation, and results analysis. The impact evaluation (the second phase) was

conducted at all four levels of the Kirkpatrick method.

In a similar approach, Guerra Castillo (2021) proposed a methodology for

evaluating the training impact in the companies of the Business Group of the Steel

Mechanical Industry in Cuba. The methodology was structured in two stages. The

first, prior to training, involved planning the evaluation process, its levels,

indicators, and instruments, as well as exploring participants’ expectations and

motivations. The second stage, following training, focused on conducting the impact

evaluation across five levels similar to the Phillips method. Finally, he suggested

assessing training effectiveness by systematizing the results obtained regarding its

efficacy and efficiency.

García González et al. (2021) reflected on the importance of evaluating the

impact of training for managers in Cuba but did not propose a specific instrument

for its implementation. Carrera Morales et al. (2022) suggested a methodology for

evaluating the impact of graduate education consisting of four stages: planning,

organization, impact measurement, and information analysis. For the evaluation of

the graduate program’s impact, they established three variables, primarily in the

personal domain and focused on knowledge management:

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

• Students – dimensions of intellectual and professional growth and

satisfaction level.

• Graduates – dimensions of professional performance and intellectual output.

• Instructors – dimensions of scientific production and professional prestige.

Fernández Medina and Ruiz Arnaud (2023) recognized that tracking graduates

is a fundamental method in evaluating the impacts of training programs, especially

at the graduate level. These authors established four variables with a total of 51

indicators to assess the impact of graduate programs. The variables are:

• Impact Evaluation (9 indicators).

• Training Outcomes (17 indicators).

• Satisfaction (15 indicators).

• Graduate Recognition (10 indicators).

Paredes-Dávila et al. (2022) emphasized the importance of understanding

the impact of the program on performance, not only from the graduates’ perspective

but also from the employers’ perspective. Other authors applied various

measurement instruments to evaluate the results and impacts of graduate programs

without establishing a general theoretical model (Nieto Acosta et al., 2022).

At the Chair of Quality, Metrology, and Standardization at the University of

Havana, an instrument was developed to assess student satisfaction with the courses

offered since the first edition of the Master’s Program in Quality and Environmental

Management (2006-2009). Another survey was administered to graduates and their

employers to evaluate the program’s impact. This instrument has undergone

modifications over time for continuous improvement (Ramos Azcuy et al., 2016) and

has been applied, with necessary adaptations, to the other two academic programs

offered by the Chair (Master’s in Metrology and Postgraduate Specialization in

Standardization).

A significant modification occurred when the programs transitioned to a

blended format, which included a new satisfaction survey for courses delivered in

the Virtual Teaching-Learning Environment (EVEA) of the University of Havana.

Additionally, to guide instructors in developing virtual courses, a procedure was

established that outlines the quality criteria to be considered for designing virtual

courses in graduate programs. These criteria include: objectives, instructional

design, interactivity, effective communication, evaluation and feedback on results,

accessibility, and course evaluation by students. Currently, the Chair has the

technical instruction IT 45 on the Evaluation of the Impact of Graduate Training,

which is part of the documented information of the Quality Management System of

the Center for Biomaterials, certified according to the NC-ISO 9001:2015 standard.

The objective of this study is to analyze the results of refining an instrument

designed to evaluate the impact of graduate academic training at the Chair of

Quality, Metrology, and Standardization at the University of Havana, aiming to

demonstrate its effects on individuals, organizations, and Cuban society.

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

Methods and materials

The instrument for impact evaluation, once the program editions were

completed, was theoretically grounded in the Kirkpatrick Model, with adaptations to

include the social impacts of training. In its current refinement, the instrument was

structured as shown in Table 1. In addition to the aspects reflected in the table, the

instrument included two general questions:

• About the student’s retention in the organization during the program and

after its completion (Question 1).

• Suggestions for program improvement.

Table 1

Dimensions of the Impact Evaluation of Graduate Academic Training (Survey for

Graduates)

Level Dimension Question

Number

of Items

Satisfaction

Quality attributes of

the program

Question 7

Question 8

Overall satisfaction

level

Question 9

Fulfillment of

expectations

Question 10

Would you

recommend?

Question 11

Learning

Evaluations conducted in all program curriculum activities.

Results

Objectives

Question 2. Fulfillment of

program objectives

Impacts

Personal domain

Question 3. Contribution to

knowledge management and

professional and research

competencies

Question 4. Impact on

personal performance

Organizational domain

Question 5

Social domain

Question 6

Source: Own elaboration.

The questions were designed in both closed and open formats, with some using a

five-point Likert scale for responses. An instrument was also applied to the

employers of the graduates to gather their perceptions about the program and its

impacts, which included the dimensions of satisfaction, results, and impacts (in the

organization), similar to what is presented in Table 1.

For the application of the instrument, a mixed-methods research approach was

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

conducted, consisting of both qualitative and quantitative elements, of an

exploratory cross-sectional type, as it investigated the perceptions of graduates from

the 10th Edition (10/2021-11/2024) of the Master’s Program in Quality and

Environmental Management. The responses were processed using SPSS v.27,

obtaining the median as a measure of central tendency and frequency distributions.

Bar charts were produced using Excel. Additionally, the non-parametric Spearman’s

Rho test was employed to ascertain potential relationships between the perceptions

of graduates and their employers, as well as between the Learning and Satisfaction

levels.

Results and discussion

The impact evaluation focused on how students have transferred the

acquired knowledge and skills to their work within the organization, how they have

utilized these skills, and how the training received has influenced their professional

performance, the organization, and society.

Student satisfaction with the courses was evaluated through two types of

surveys: one for in-person courses and another for those delivered in the Virtual

Teaching-Learning Environment. Learning evaluation was conducted directly by the

instructors through questions in classes, seminars, extracurricular assignments, and

exams, following the assessment methods planned for each course, seminar,

workshop, and final project defense, in accordance with the procedures established

in the document IT 43 Evaluations and Certifications in Graduate Programs (Center

for Biomaterials, 2023).

While academic training is a strategic investment, and immediate results are

often not achievable, the impact is evaluated in the Master’s programs in Quality

and Environmental Management, Metrology, and the Postgraduate Specialization in

Standardization during the self-evaluation process at the end of each edition.

Graduates are expected to acquire the necessary knowledge and skills to enhance

their performance and conduct research aimed at promoting improvements or

innovations that impact their organizations (Sosa Vera et al., 2024).

Upon completing the 10th Edition in November 2024, the satisfaction of

students with the program and its impact was assessed using questionnaires

distributed to graduates and their direct supervisors, as they are considered

beneficiaries of the academic training service provided.

In refining the instrument, new elements were included, such as:

• Strategic sectors in which the impact occurs (Ministry of Economy and

Planning, 2020).

• Levels of introduction and potential for generalization of the results

obtained.

• Levels of satisfaction with the competencies and experiences of instructors

and advisors, as well as with program management.

Table 2 presents the statistical results from the application of the instrument

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

to 12 graduates (92.3%) of the 10th Edition of the Master’s Program in Quality and

Environmental Management. Additionally, five employers (71.4%) of the seven

contacts provided by the graduates responded to the impact evaluation instrument

(Table 3).

Table 2

Quantitative Results of the Instrument Application (Graduates)

Items Median

Frequencies (%)

Question 2. Fulfillment of Objectives (4: Satisfactorily; 5: Fully)

2.1 Contribute to the Improvement of

Organizational Performance.

4 41,7 58,3

2.2 Argue and Disseminate the

Importance of Management Systems.

5 83,3 16,7

2.3 Conduct Research Related to

Management Systems.

4 41,7 58,3

2.4 Interpret the Legal, Regulatory, and

Normative Basis of Management

Systems.

5 66,7 33,3

Question 3. Contribution of Developed Activities to (4: Satisfactorily; 5:

Significantly)

3.1 Update and Deepen Knowledge

83,3

16,7

3.2 Obtain New Knowledge

75,0

25,0

3.3 Produce Knowledge

75,0

25,0

3.4 Develop Specific Professional Skills

58,3

41,7

3.5 Acquire the Methodology Required

for Scientific Research

5 66,7 33,3

Question 4. Effect on Graduates’ Social Functions (3: Sufficient; 4: Significant; 5:

Very significant)

4.1 Effect on Professional Performance

58,3

33,3

8,3

4.2 Effect on Intellectual Production

4,5

50,0

41,7

8,3

4.3 Effect on Professional Prestige

58,3

41,7

Question 5. Effect on Organizational Performance

Yes (1)

Partial

5.1 The academic training received has

influenced your organization’s

performance

1 83,3 16,7

5.2 There are conditions in your

organization for change

1 83,3 16,7

5.3 From your position, you can

implement changes in the organization

1 75,0 25,0

Question 6. The results of your research

have the potential to be implemented

1 100

Question 7. Evaluate (3- Good; 4- Very good; 5- Excellent.)

7.1 Pedagogical Competence of

Instructors and Advisors

5 75,0 25,0

7.2 Scientific Competence of

83,3

16,7

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

Items Median

Frequencies (%)

Instructors and Advisors

7.3 Experience of Instructors and

Advisors in the Knowledge Area

5 75,0 25,0

7.4 Availability of Updated Bibliography

66,7

33,3

7.5 Access to Networks and ICT

58,3

41,7

7.6 Equipment and Facilities

25,0

50,0

25,0

7.7 Access to Computing Resources

41,7

16,7

7.8 Organization, Execution, and

Control of the Program

4,5 50,0 41,7 8,3

7.9 Attention to Students’ Academic

Needs

5 66,7 25,0 8,3

Question 8. Evaluate (3- Good; 4- Very good; 5- Excellent.)

8.1 Requirement for Independent Study

75,0

25,0

8.2 Requirement for Research Work

83,3

16,7

Question 9. General Impression of the

Course

5 83,3 8,3

Question 10. Fulfillment of

Expectations

5 58,3 41,7

Question 11. Would you recommend?

(1: Yes)

100

Source: Own elaboration.

Table 3

Quantitative Results of the Instrument Application (Employers)

Items Median

Frequencies (%)

Question 2. Fulfillment of Objectives (4: Satisfactorily; 5: Fully)

2.1 Contribute to the Improvement of

Organizational Performance.

5 60 40

2.2 Argue and Disseminate the Importance of

Management Systems.

5 60 40

2.3 Conduct Research Related to Management

Systems.

4 40 60

2.4 Interpret the Legal, Regulatory, and

Normative Basis of Management Systems.

5 60 40

Question 4. Effect on Professional

Performance:

Yes (1)

1 100

Question 5. Effect on Organizational Performance:

Yes (1)

Partial

5.1 The academic training received has

influenced your organization’s performance

1 100

5.2 There are conditions in your organization

for change

1 100

5.3 From your position, you can implement

100

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

Items Median

Frequencies (%)

changes in the organization

Question 6. The results of your research have

the potential to be implemented

1 100

Question 9. General Impression of the Course

(5: Very satisfied; 4: Satisfied)

5 100

Question 10. Fulfillment of Expectations (5:

100%; 4: 90%)

5 100

Question 11. Would you recommend? (1: Yes)

100

Source: Own elaboration.

Note. The question numbers in this table do not follow the consecutive order of the

survey but correspond to the dimensions defined in Table 1.

Regarding the first question about the retention of graduates in the

organizations they belonged to at the start of the master’s program, 67% (8)

remained in their positions. Of the four graduates who changed jobs during the

program, one was promoted to work in the OSDE, two moved to the non-state sector,

and another left teaching to work in the tourism sector. The student who did not

respond to the survey broke her employment link immediately after completing her

master’s studies. These results reflect the labor mobility manifesting in the national

context.

Overall, the responses from the surveyed employers ranged from 4 to 5 as a

measure of central tendency (median). The objectives that received the highest

percentage of responses regarding their total fulfillment were related to:

• Being able to argue and disseminate the importance of quality and

environmental management for the sustainability of organizations.

• Interpreting the legal, regulatory, and normative basis for the

implementation of quality management systems, environmental systems,

and others.

All respondents indicated that the program generated positive effects on the

graduates and their organizations. The aspects of personal performance most

influenced by the master’s program, according to the perceptions of respondents

(Figure 1), included:

• Publications made and participation in events.

• Development of capacities as a teacher, researcher, consultant, auditor, or

other functions exercised by the graduate.

There is a correlation of 0.975 (Spearman’s Rho, p=0.005) between the

opinions of graduates and their employers regarding the perceived effects on

personal performance. These opinions are objectively supported by the scientific

production of the students from the 10th Edition during the program, which includes

19 publications and 30 presentations at events, as well as other activities related to

their scientific, technical, and professional work (teaching, consulting, and

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

auditing).

Figure 1

Responses on the Aspects of Personal Performance of Graduates Most Influenced by

the Master’s Program

Source: Own elaboration.

100% of the graduates considered that the program has a significant or

satisfactory contribution to knowledge management and professional and research

competencies in relation to the following elements:

• Updating and deepening knowledge.

• Obtaining new knowledge.

• Producing knowledge.

• Developing specific professional skills.

• Acquiring the methodology required for scientific research.

Additionally, both graduates and their employers noted that positive impacts

have been achieved in organizational performance, contingent upon the existence

of conditions for change within these organizations, and that graduates have the

ability to influence these changes from their positions.

Table 4 outlines the theses completed by graduates of the 10th Edition.

Regarding the strategic sectors of the National Economic and Social Development

Plan until 2030 to which the results of the conducted research contribute, these

include:

• Integrated Transport Logistics: 3

• Professional Technical Services: 3

0% 20% 40% 60% 80% 100%

Publications and participation in

events

Development of skills as a teacher,

researcher, consultant, auditor or…

Obtaining scientific, teaching or

technological categories

Participation as an expert in different

forums

Promotion to positions of greater

responsibility

Response percentage

Graduates Employers

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

• Electrical Energy: 2

• Integrated Logistics of Hydraulic and Sanitary Networks and Installations: 2

• Tourism: 1

• Pharmaceutical, Biotechnological, and Biomedical Productions: 1

• Food Production Sector: 1

Table 4

Theses Defended in the 10th Edition by Organizations and Strategic Sectors

Theses Defended

Organization

Sector

Implementation of the Integrated

Management System at the Technological

Management Information Center of

Cienfuegos

CIGET/

CITMA

Professional

Technical

Services

Digital Transformation of Documented

Information in MYCRON Laboratories MINFAR

Professional

Technical

Services

Procedure for Environmental Performance

Evaluation at the Cuban Air Navigation

Company

ECNA

Integrated

Transport

Logistics

Implementation of Knowledge

Management at the Base Business Unit:

“Service to Airlines”

ECASA

Integrated

Transport

Logistics

Proposal of Actions to Enhance Customer

Satisfaction with Photovoltaic Systems

Connected to the National Electric Grid

UNE

Electrical

Energy

Procedure for Identifying Customer Needs

and Expectations at Pagary Company SNE

Integrated

Transport

Logistics

Improvements to the Production Process

of Pre-Cooked Tostone in a Food

Processing Mini-industry

SNE Food Production

Implementation of NC-ISO 9001:2015 at

the National Drilling and Construction

Company

INRH

Hydraulic

Networks and

Installations

Procedure for Implementing

Organizational Surveillance and

Intelligence at the Geominsal Business

Group

GEOMINSAL

Professional

Technical

Services

10.

Implementation of Documentation

Management Procedures at the Base

Enterprise for the Production of Semi-

solids

BioCubaFarma

Pharmaceutical,

Biotechnological

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

Theses Defended

Organization

Sector

11.

Procedure for Identifying Environmental

Aspects and Impacts at the Maintenance

and Repair Company for Hydraulic Works

INRH

Hydraulic

Networks and

Installations

12.

Integral Risk and Opportunity

Management at the Santiago de Cuba

Electric Company

UNE

Electrical

Energy

13.

Implementation of an Action Plan to

Enhance Customer Satisfaction among

Yachters at the MARLIN Nautical Branch,

Marina Hemingway

OSDE Ecosol Tourism

Source: Own elaboration.

Graduates considered that their research has led to organizational

improvements or innovations in the following aspects:

• Organizational improvements/innovations.

• Implementation of management systems.

• Process optimization.

• Enhancements in environmental performance evaluation.

• Digital transformation in laboratory management.

• Improvements in documentation management.

• Comprehensive management of risks and opportunities.

• Improvements in customer satisfaction.

All results obtained by the graduates have been fully or partially

implemented, which is a requirement established in the program for the defense of

final projects, ensuring they do not remain as mere proposals for solutions.

Graduates believed that their results could be generalized to different levels (Figure

2).

Figure 2

Potential Level of Generalization of Solutions Provided by Graduates

16.7

33.3

50.0

National Sectorial Local

Responses, %

Possible level of generalization

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

Source: Own elaboration.

The conditions that favor the implementation of the proposed improvements

or innovations in the graduates’ organizations include:

• The existence of a formalized management system.

• The academic preparation and training of personnel.

• An organizational culture that fosters an environment valuing and promoting

innovation and creativity, encouraging workers to propose new ideas and

solutions.

• Cloud-based work and the allocation of computing resources.

• The willingness and leadership of management.

When planned changes are only partially achieved, it is often because

managers remain reluctant to change and are comfortable with their current

practices. Furthermore, for the organizational changes proposed by graduates to be

fully implemented, leadership and commitment from the organization’s

management are required. In this regard, it is important to highlight that good

human resource management contributes to employee commitment to achieving

strategic objectives and improving organizational performance (Mohamed Ali et al.,

2024). Training is a key factor influencing employee motivation and the development

of competencies, which ultimately enhances their organizational commitment (Tumi

et al., 2022).

Regarding the social needs (current and prospective) addressed by the

program, graduates identified the following:

• The program promotes teamwork, information searching, and the exchange

of updated knowledge at a global level to find the best solutions, based on

compliance with current norms and regulations, and the use of new

technologies, which leads to resource savings and environmental protection.

• Cuban society requires a significant change in the state business sector; any

activity that directly or indirectly supports this goal should be prioritized.

This training provides knowledge that positively impacts the implementation

of organizational innovations in these entities.

• The master’s program has provided key tools to help organizations comply

with current environmental regulations, both locally and globally. This is

crucial in a context where environmental regulations are becoming

increasingly stringent, and communities require effective solutions to

mitigate the effects of pollution and promote sustainable development.

• It improves professional performance, fosters critical thinking, problem-

solving, and research skills. In the future, it may contribute to access to

better-paying and more responsible jobs.

• It contributes to technological change and increases customer satisfaction.

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

• It provides the necessary knowledge and tools for innovation and

management that lead to the sustained and sustainable success of

organizations.

• It addresses needs in the fields of quality and environmental management

and other management systems, as well as the integrated management of

these systems, along with research methodology.

• It enhances professional competencies in management, primarily in quality

management.

• It influences the improvement of environmental management in

organizations.

• It allows for a comprehensive understanding of the use of legislation, current

norms, and the interconnection of quality, occupational health and safety,

and environmental systems within an organization, which would reduce

information duplication. It provides a broader perspective on processes.

In analyzing satisfaction regarding the quality attributes of the program

(Table 2), 100% of graduates expressed high satisfaction (levels 4 - Very Good and 5

- Excellent) with the following attributes:

• Pedagogical competence of instructors and advisors.

• Scientific competence of instructors and advisors.

• Experience of instructors and advisors in the knowledge area.

• Availability of updated bibliographic resources.

• Access to networks and other information and communication technologies.

It is important to highlight that, as the program is currently offered in a

blended format with all courses in the Virtual Teaching-Learning Environment

(EVEA), students are required to have internet access as a condition for enrollment.

Additionally, a preparatory course titled “Use of Information and Communication

Technologies for Training and Research” is provided. Another preparatory course,

“Introduction to Statistical Methods,” essential for the subsequent application of

statistical tools for management and research, is also taught in the EVEA. This should

contribute to students being familiar with the virtual environment by the time they

begin the academic activities of the curriculum.

The seminars and thesis defenses are conducted at the Biomaterials Center

of the University of Havana, which has a meeting room used for these purposes.

Although students are informed that, if necessary, they can access the internet or

the information available at the institution in both printed and digital formats, they

rarely express this need. However, the percentage of students with high satisfaction

was lower regarding:

• Equipment and facilities (75%).

• Access to computing resources (83%).

When responding about their overall satisfaction with the program, there

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

was a high response to expectations, high satisfaction, and an intention to

recommend the program to other students.

Regarding student learning, this was evaluated through curriculum activities

(courses, seminars, and thesis defenses). Out of an initial enrollment of 16 students,

three did not complete all activities preceding the thesis defense, resulting in an

81% retention rate. Of the 13 students who accumulated all prerequisite credits, 11

defended their theses within the timeframe of the edition, representing 84.6% of

the final enrollment, while the other two defended two months later, achieving a

100% effectiveness rate. The thesis defenses were conducted with quality and were

evaluated rigorously, with nine rated as Excellent (69%), three as Good (23%), and

one as Approved.

The calculation of the medians of the graduates’ grades yielded the results

shown in Table 5. A total of 61.5% had a median evaluation of Excellent (5). This

indicates that the courses offered in the EVEA facilitate effective student learning.

Additionally, it is interesting to note that students with better learning outcomes

also demonstrate higher satisfaction with the program. This positive correlation is

significant, with a Spearman’s Rho coefficient of 0.672 (p=0.017).

Table 5

Distribution of the Medians of Graduates’ Grades

Value

Simple Frequencies

Cumulative Frequencies

Absolute

Relative

Absolute

Relative

3,5

7,7 %

4,0

15,4 %

23,1 %

4,5

15,4 %

38,5 %

5,0

61,5 %

100,0 %

Total

100,0 %

Source: Own elaboration.

Students provided suggestions for the master’s program, primarily related to

the lack of in-person meetings for clarifying doubts in courses and insufficient

connectivity. Both aspects present significant challenges for the development of

blended programs, especially if transitioning to full virtuality is desired, considering

the technological and energy situation in the country, where synchronous online

interactions are quite limited.

In empirical studies, other authors (DiLoreto et al., 2022) have identified

that the presence of the academic advisor is the best predictor of both student

satisfaction and perceived learning in virtual training environments. Self-regulated

learning processes and student-academic advisor dialogue/interaction processes

have also been identified as mediating factors for the learning and satisfaction

perceived by students (Eom & Ashill, 2023).

On their part, Arslankara et al. (2024) have suggested that factors such as

computer literacy and students’ self-efficacy in using technology play an important

role in the effectiveness of virtual training implementation. Additionally, the quality

of the course’s instructional design and the technology environment, as well as the

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

assurance of student-to-student and teacher-to-student interactions, contribute to

student satisfaction, engagement, and motivation. Given the importance attributed

to interaction in the technology-mediated teaching and learning process, Hamtini

(2008) proposed this designation for the first level of evaluating the effectiveness of

virtual training as a replacement for the term “reaction.”

Similar findings were evidenced through the observations made in this study,

indicating that when students do not dedicate sufficient time or exhibit a lack of

autonomous learning capacity, or when there are gaps in student-academic advisor

interaction, perceived satisfaction is lower, and learning outcomes are negatively

affected.

Conclusions

The instrument developed for evaluating the impact of academic training

allows for an understanding of the positive effects of the evaluated program in

personal, organizational, and social spheres. The assessment of the effects of the

Master’s program in Quality and Environmental Management across four levels

(Satisfaction, Learning, Results, and Impacts) demonstrates the relevance and

quality of the program, as well as identifies opportunities for improvement. The

study identifies the contextual conditions of organizations that contribute to the

positive impact of student outcomes, aimed at enhancing personal performance and

organizational innovation, based on the research developed in a master’s program

in quality and environmental management. Factors such as students’ self-study and

learning capabilities, as well as the necessary interaction with their academic

advisors, influence their perceptions of satisfaction and the results of their academic

training.

References

Alsalamah, A., & Callinan, C. (2021). Adaptation of Kirkpatrick’s Four-Level Model

of Training Criteria to Evaluate Training Programmes for Head Teachers.

Education Sciences, 11(3), 116. https://doi.org/10.3390/educsci11030116

Arslankara, V. B., Arslankara, E., Asan, İ., Külekçi, M., & Usta, E. (2024). Assessment

of an In-Service Training Activity Transformed into an E-Learning

Environment Using the Kirkpatrick Model. Journal of Teacher Education and

Lifelong Learning, 6(1), 207-221.

https://dergipark.org.tr/en/download/article-file/3748232

Bahl, K., Kiran, R., & Sharma, A. E. (2024). Evaluating the effectiveness of training

of managerial and non-managerial bank employees using Kirkpatrick’s model

for evaluation of training. Humanities Social Sciences Communications, 11,

508. https://doi.org/10.1057/s41599-024-02973-y

Cahapay, M. B. (2021). Kirkpatrick Model: Its Limitations as Used in Higher Education

Evaluation. International Journal of Assessment Tools in Education, 8(1),

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

135-144. https://doi.org/10.21449/ijate.856143

Carrera Morales, M. A., Mesa Carpio, N., & Padilla Cuellar, Y. (2022). Metodología

para evaluar el impacto de la educación de posgrado. Transformación, 18(1),

53-69. http://scielo.sld.cu/pdf/trf/v18n1/2077-2955-trf-18-01-53.pdf

Díaz Leyva, C. A., & Marrero Fornaris, C. (2021). La evaluación del impacto de la

capacitación: retos y beneficios para las organizaciones actuales. Revista

Universidad y Sociedad, 13(6), 28-38.

http://scielo.sld.cu/pdf/rus/v13n6/2218-3620-rus-13-06-28.pdf

DiLoreto, M., Gray, J. A., & Schutts, J. (2022). Student Satisfaction and Perceived

Learning in Online Learning Environments: An Instrument Development and

Validation Study. Education Leadership Review, 23(1), 115-134.

https://files.eric.ed.gov/fulltext/EJ1380107.pdf

Eom, S., & Ashill, N. J. (2023). Learning Outcomes and Learner Satisfaction: The

Mediating Roles of Self-regulated Learning and Dialogues. Journal of

International Technology and Information Management, 32(1), 1-31.

https://doi.org/10.58729/1941-6679.1557

Fernández Medina, C., & Ruiz Arnaud, J. (2023). Instrumento para la evaluación del

impacto de programas de posgrado a partir de la dimensión egresados desde

el Centro de Estudios de la Educación Superior Agropecuaria. ReHuSo, 8(2),

137-156. https://doi.org/10.33936/rehuso.v8i2.5715

Firooznia, M., Hamta, A., & Shakerian, S. (2020). The effectiveness of in-service

training “pharmacopeia home health” based on Kirkpatrick's model: A quasi-

experimental study. Journal of Education and Health Promotion, 9(1), 218.

https://doi.org/10.4103/jehp.jehp_170_20

García González, M., García Rodríguez, A., & Ortiz Cárdenas, T. (2021). Análisis

desde la evaluación de impacto en la capacitación a directivos. Avances,

23(3), 1-16. https://www.redalyc.org/articulo.oa?id=637869395002

Gómez Miranda, O. M. (2023). Factores institucionales que impactan en la actividad

emprendedora de los estudiantes universitarios. Región Científica, 2(1),

202327. https://doi.org/10.58763/rc202327

Guerra Bretaña, R. M., Meizoso Valdés, M. C., Ramírez García, J. R., & Iglesias

Morell, A. (2017). Los programas de Maestría como canal de transferencia

del conocimiento universidad-industria en Cuba. Revista COFIN Habana,

11(1), 1-22. http://www.cofinhab.uh.cu/index.php/RCCF/article/view/205

Guerra Castillo, S. (2021). Metodología para evaluar el impacto de la capacitación

en las empresas cubanas. Revista Universidad y Sociedad, 13(6), 237-249.

http://scielo.sld.cu/pdf/rus/v13n6/2218-3620-rus-13-06-237.pdf

Hamtini, T. M. (2008). Evaluating E-learning Programs: An Adaptation of Kirkpatrick's

Model to Accommodate E-learning Environments. Journal of Computer

Science, 4(8), 693-698. https://doi.org/10.3844/jcssp.2008.693.698

Jiménez-Pitre, I., Molina-Bolívar, G., & Gámez Pitre, R. (2023). Visión sistémica del

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

contexto educativo tecnológico en Latinoamérica. Región Científica, 2(1),

202358. https://doi.org/10.58763/rc202358

Ministerio de Economía y Planificación. (2020). Plan Nacional de Desarrollo

Económico y Social hasta 2030.

https://www.mep.gob.cu/es/noticia/folleto-sobre-plan-nacional-de-

desarrollo-economico-y-social-hasta-2030-agenda-2030

Mohamed Ali, A. A., Usman, I., & Kurniawati, M. (2024). The role of human resource

management practices in enhancing organizational commitment: Systematic

literature review. World Journal of Advanced Research and Reviews, 24(03),

2164-2176. https://doi.org/10.30574/wjarr.2024.24.3.3783

Nawaz, F., Ahmad, W., & Khushnood, M. (2022). Kirkpatrick Model and Training

Effectiveness: A Meta-Analysis 1982 To 2021. Business & Economic Review,

14(2), 35-55. https://doi.org/10.22547/BER/14.2.2

Nieto Acosta, O. M., González Blanco, I., & Grimón Cebreiro, M. Y. (2022).

Evaluación del impacto de la maestría en Farmacología del Instituto de

Farmacia y Alimentos. Revista Cubana De Educación Superior, 41(Especial

2), 351–367. https://revistas.uh.cu/rces/article/view/194

Paredes-Dávila, H., Padrón-Estrada, & Elena, M. (2022). Seguimiento a egresados:

impacto de un posgrado de formación docente en el desempeño laboral.

Psicología Educativa, 10(1), 57-64.

https://revistapsicologiaeducativa.unam.mx/index.php/psicologiaeducativ

a/article/view/47/40

Parra Robledo, R., Ruiz Bueno, C. (2020). Evaluación de impacto de los programas

formativos: aspectos fundamentales, modelos y perspectivas actuales.

Revista Educación, 44(2), 512-524.

https://doi.org/10.15517/revedu.v44i2.40281

Ramos Azcuy, F. J., & Guerra Bretaña, R. M. (2023). Instrumento para la

autoevaluación de programas virtuales de maestrías. RECIE. Revista

Caribeña De Investigación Educativa, 7(1), 7–29.

https://revistas.isfodosu.edu.do/index.php/recie/article/view/406

Ramos Azcuy, F. J., Meizoso Valdés, M. C., & Guerra Bretaña, R. M. (2016).

Instrumento para la evaluación del impacto de la formación académica.

Revista Universidad y Sociedad, 8(2), 114-124.

http://scielo.sld.cu/pdf/rus/v8n2/rus13216.pdf

Shewchuk, S., Wallace, J., & Seibold, M. (2023). Evaluations of training programs to

improve capacity in K*: a systematic scoping review of methods applied and

outcomes assessed. Humanities and Social Sciences Communications, 10,

887. https://doi.org/10.1057/s41599-023-02403-5

Sosa Vera, R., Ramírez García, J. R., & Guerra Bretaña, R. M. (2024). Aportes y retos

de la educación de posgrado de la Universidad de La Habana para el

fortalecimiento de la infraestructura nacional de calidad. Rimarina, 8(1),

91-101. https://doi.org/10.61236/rima.v8i1.614

Stable Rodríguez, Y., & Núñez García, L. C. (2021). Metodología para la evaluación

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

e8827

del impacto de la capacitación en organizaciones de información científica

tecnológica. Revista Cubana de Información en Ciencias de la Salud, 3(2),

1606. http://scielo.sld.cu/pdf/ics/v32n2/2307-2113-ics-32-02-e1606.pdf

Tumi, N. S., Hasan, A. N., & Khalid, J. (2022). Impact of Compensation, Job

Enrichment and Enlargement, and Training on Employee Motivation. Business

Perspectives and Research, 10(1), 121-139.

https://doi.org/10.1177/2278533721995353

Yi, Z. M., Zhou, L. Y., Yang, L., L, Y., Liu, W., Zhao, R. S., & Zhai, S. D. (2020).

Effect of the international pharmacy education programs: a pilot evaluation

based on Kirkpatrick’s model. Medicine, 99(27), e20945.

https://doi.org/10.1097/MD.000000000002

| Rosa Mayelín Guerra Bretaña | Karen Pupo Méndez | Fridel Julio Ramos Azcuy |

About the main author

Rosa Mayelín Guerra Bretaña:

es Dra. C. Químicas por el Centro Nacional de

Investigaciones Científicas (1991), Cuba, Licenciada en Física (1982) por la

Universidad Estatal de San Petersburgo, Rusia. Es presidente de la Cátedra

Honorífica de Calidad, Metrología y Normalización de la Universidad de La Habana.

Declaration of author responsibility

Rosa Mayelín Guerra Bretaña

Conceptualization, Data Curation, Formal Analysis,

Research, Methodology, Resources, Software, Supervision, Validation/Verification,

Visualization, Writing/original draft and Writing, review and editing.

Karen Pupo Méndez

2: Research, Methodology, Validat

ion/Verification,

Visualization, Writing/original draft, and Writing, review and editing

Fridel Julio Ramos Azcuy

Methodology, Supervision, Writing/original draft, and

Writing, review and editing

Financing:

This research was carried out using our own resources.

Special Acknowledgments: