System for the generation, management and evaluation of in-basket tests for public competitions and operation method thereof

Abstract

The present invention relates to a system (S) for the generation, management and evaluation of at least one In-Basket test for public competitions and selections for at least one candidate (C), comprising at least one management device (1), that can be used by an operator, to activate the procedure for the generation, management and evaluation of said at least one In-Basket test in an anonymous manner; at least one data transfer unit (2), operatively connected to said management device (1), that is able to transfer data in said system (S) for the generation, management and evaluation of said at least one In-Basket test; a plurality of multimedia devices (3), operatively connected to said at least one data transfer unit (2), that can be used by said at least one candidate (C) to perform said at least one In-Basket test, based on the data received from said at least one data transfer unit (2); at least one Artificial Intelligence module (4), operatively connected to said at least one data transfer unit (2) that is able to generate said at least one In-Basket test based on the data received from said at least one data transfer unit (2) and to evaluate said at least one In-Basket test performed by said at least one candidate (C). The present invention also relates to a method for operating the system.

Description

[0001] Title: System for the generation, management and evaluation of In-Basket tests for public competitions and operation method thereof

[0002] DESCRIPTION

[0003] Field of application

[0004] The present invention relates to a system for the generation, management and evaluation, even anonymously, of In-Basket tests for staff recruitment through public competitions and selections, in particular in the Public Administration and in the Special Agencies of Local Authorities, as well as in Universities.

[0005] The present invention also relates to the method for operating the system.

[0006] More in detail, the invention concerns a system of the above type, studied and created in particular for the generation, administration and evaluation of In-Basket tests that can be used in selecting the staff of the Public Administration and the Special Agencies of Local Authorities, i.e. selection processes also known as public competitions.

[0007] The description will be directed hereinafter to a system for the generation, management and evaluation of In-Basket tests for staff selection through public competitions, by employing proprietary software, multimedia devices such as smartphones, virtual reality viewers, laptops, PCs, tablets, web browsers, speakers, headphones, microphone, workstations and the like and specially trained Artificial Intelligence modules, but it is very evident that the same should not be considered as limited to this specific use.

[0008] Prior art

[0009] It is currently known that in a public competition or selection, the In- Basket test is an evaluation tool used to determine how efficiently the candidates for the selection perform certain tasks, thus testing their behavioural and organizational skills. Known In-Basket tests are tools designed and used by occupational psychologists or professionals in the staff selection field, based on specific scientific methodologies.

[0010] In the tests, the candidate for the selection must take a role in a company or organization and, in a limited time period, must make decisions in relation to contingent problems found on the desk in the form of incoming mail, information to be memorized, messages and the like. In particular, the candidate must organize its own work, answer the mail, decide on the actions to be implemented firsthand and plan those to be delegated to collaborators, by using the available business structures and resources. The In-Basket test allows recruiters to note the presence of cognitive, management and behavioural skills of the candidate.

[0011] Each In-Basket test is implemented by groups of recruiters specialized in different fields, who must prepare the documents, materials and tests in a balanced manner and complying in turn with the specificities related to the specific selection field.

[0012] It is thus evident that the preparation of current In-Basket tests is expensive in terms both of employed staff and of time needed for the preparation of the different tests and the subsequent evaluation thereof.

[0013] A further disadvantage of known methodologies is represented by the difficulty of exactly reproducing the working conditions in which a candidate would have to perform the job for which he is participating to the competition.

[0014] Moreover, since in current In-Basket tests the candidate usually mainly uses paper means, it is impossible for recruiters to interact with the candidate himself when executing the test.

[0015] A disadvantage of the mode for preparing the current In-Basket tests consists in the difficulty in creating always balanced tests for each sector in which the competition is performed. Moreover, the correction of In-Basket tests is sometimes determined by the subjectivity of the recruiting staff, since it is often impossible to lay down objective criteria for each single test.

[0016] In the light of the drawbacks and unsolved problems of the prior art, an object of the present invention is to provide a system for the generation, administration and evaluation of In-Basket tests for public competitions and selections that is able to overcome the problems and drawbacks shown by prior art systems.

[0017] In particular, an object of the present invention is to provide a system that is able to optimize and automate the test preparation and evaluation times.

[0018] Another object of the present invention is to provide a system through which a candidate can perform the test in conditions that are more realistic and similar to the actual conditions in which he would perform his job after passing the test.

[0019] A further object of the present invention is to provide a system through which the candidate can interact with recruiters during the test, by either speaking or listening.

[0020] Still an object of the present invention is to provide a system that allows to prepare standard and balanced tests for each sector in which the competition is performed.

[0021] A further object of the present invention is to provide a system that is able to perform an automatic, objective and real-time correction of the test attended by the candidate.

[0022] Finally, an object of the present invention is to provide the method for operating the system.

[0023] Summary of the invention The idea underlying the present invention is to provide a system for the generation, management and evaluation of In-Basket tests for staff selection through public competitions, in particular in the Public Administration and in the Special Agencies of Local Authorities, based on the use of Artificial Intelligence modules.

[0024] An object of the present invention is thus a system for the generation, management and evaluation of at least one In-Basket test for public competitions and selections for at least one candidate, comprising at least one management device, that can be used by an operator, to activate the procedure for the generation, management and evaluation of said at least one In-Basket test, at least one data transfer unit, operatively connected to said management device, that is able to transfer data in said system for the generation, management and evaluation of said at least one In- Basket test, a plurality of multimedia devices, operatively connected to said at least one data transfer unit, that can be used by said at least one candidate to perform said at least one In-Basket test, based on the data received from said at least one data transfer unit, at least one Artificial Intelligence module, operatively connected to said at least one data transfer unit that is able to generate said at least one In-Basket test based on the data received from said at least one data transfer unit and to evaluate said at least one In-Basket test performed by said at least one candidate.

[0025] Furthermore according to the invention, said system can comprise at least one database, operatively connected to said at least one data transfer unit, comprising the information needed for the configuration of said at least one In-Basket test to be generated and the operating modes of said plurality of multimedia devices.

[0026] Still according to the invention, said plurality of multimedia devices comprise smartphones, virtual reality viewers, laptops, PCs, tablets, web browsers, speakers, headphones and microphone. Always according to the invention, said at least one data transfer unit communicates with said Artificial Intelligence module via application programming interfaces - API, supported by an Internet network.

[0027] A further object of the present invention is a method for operating the system for the generation, management and evaluation of at least one In- Basket test for public competitions and selections for at least one candidate, comprising the following steps:

[0028] 110. by means of said at least one management device, sending a request to said at least one data transfer unit, for the generation of said at least one In-Basket test;

[0029] 120. by means of said at least one data transfer unit, sending the request to said Artificial Intelligence module;

[0030] 130. by means of said Artificial Intelligence module, generating said at least one In-Basket test and sending it to said data transfer unit;

[0031] 140. by means of said at least one data transfer unit, sending the first contents of said at least one In-Basket test to said plurality of multimedia devices;

[0032] 150. by means of said at least one data transfer unit, sending again a request to said Artificial Intelligence module that provides new contents to said at least one data transfer unit;

[0033] 160. once the execution of said at least one In-Basket test by said at least one candidate is finished, by means of said plurality of multimedia devices, providing the test result and / or contents to said at least one data transfer unit that simultaneously sends an evaluation request to said Artificial Intelligence module;

[0034] 170. by means of said Artificial Intelligence module, performing the evaluation of said at least one In-Basket test and sending the outcome to said at least one data transfer unit that sends the outcome to said at least one management device.

[0035] Furthermore according to the invention, in said step 120., said at least one data transfer unit sends the request to said Artificial Intelligence module via an application or application programming interface - API, supported by an Internet network.

[0036] Still according to the invention, after said step 140., said plurality of multimedia devices send new content requests, together with new contents, to said at least one data transfer unit.

[0037] Always according to the invention, the interaction between said at least one data transfer unit, said plurality of multimedia devices and said at least one Artificial Intelligence module is continuous, up to the end of said at least one In-Basket test.

[0038] Preferably according to the invention, in said step 170., based on the configurations set in said at least one database, said at least one data transfer unit sends the results of the evaluation of said at least one In- Basket test to said plurality of multimedia devices.

[0039] Furthermore according to the invention, said Artificial Intelligence module is trained according to a training method, comprising the following steps:

[0040] 210. providing a training dataset comprising data selected and pre- processed by occupational psychologists and other specialized staff, a skill statement, evaluation and score documents, a set of rules and score schemes that define how an In-Basket test should be evaluated;

[0041] 220. performing a fine-tuning on pre-trained models and further optimizing the pre-trained model by fme-tuning based on the projectspecific datasets;

[0042] 230. activating an iterative feedback cycle. Brief description of the drawings

[0043] The present invention will be now described by non-limiting examples, according to a preferred embodiment, with particular reference to the attached figures, in which: figure 1 shows a schematic representation of the components of the system for the generation, management and evaluation of In-Basket tests, according to the present invention; figure 2 shows a conceptual scheme of the operation of an Artificial Intelligence module comprised in the system shown in figure 1 ; and figure 3 shows a block diagram of the operation of the method, which is the subject matter of the present invention.

[0044] Detailed description

[0045] Referring to the attached figures, the system S for the generation, management and evaluation of In-Basket tests in public competitions to which one or more candidates C participate, substantially comprises a management device or PC manager 1 , that can be used by an operator, a data transfer unit or Web Server 2, operatively connected to said management device 1, a plurality of multimedia devices 3 such as smartphones, virtual reality viewers, laptops, PCs, tablets, web browsers, speakers, headphones, microphone and the like, operatively connected to said data transfer unit 2, each multimedia device being usable by one or more candidates C when performing the test, an Artificial Intelligence - Al module 4 and being operatively connected to said data transfer unit 2 and a database 5, operatively connected to said data transfer unit 2.

[0046] Said management device 1 is a device of the workstation type, that can be used by an operator, that is enabled to start the process for the generation and evaluation of the In-Basket test. In particular, it is recognized by an identification code or Personal Identification Number - PIN that is previously stored in said database 5. The PIN, previously loaded into the database 5 and already assigned to the operator role, is recognized by the system S that, after verifying the existence and role thereof, allows to use the management functions in the system S. The PIN insertion by the operator can occur by direct typing through the keyboard, by optic reading of a QR-CODE or the like or possibly also by reading NFC tags. Said management device is operatively connected to said data transfer unit 2 by a network of the Internet type and is configured to send a request for the generation and start of the In-Basket test.

[0047] Said data transfer unit 2 is a central node for sorting data and information between the different components of the system S. In fact, it is able to process the requests coming from said management device 1 and from the plurality of multimedia devices 3 and directly communicates with said Al module 4 and database 5. In particular, after receiving the activation request from said management device 1, it is able to send a request for the generation of an In-Basket test to said Al module 4, to receive it and then to address it to said plurality of multimedia devices 3 in a continuous exchange for the whole duration of test execution, until the exchange finishes by sending the test evaluation processed by said Al module 4 to said management device 1 , as it will be described in detail below. The communication between said data transfer unit 2 and plurality of multimedia devices 3 occurs via a connection of the Internet type. The communication between said data transfer unit 2 and Al module 4 occurs via application programming interfaces - API, supported by an Internet network.

[0048] Said plurality of multimedia devices 3, that can be used by one or more candidates C during the test, can be multiple, such as smartphones, virtual reality headsets, laptops, PCs, tablets, web browsers, speakers, headphones, microphone, workstations and the like, and they are operatively connected to said data transfer unit 2 via connections of the Internet or LAN type. Said Artificial Intelligence or Al module 4 is available via the Internet since it is installed on a cloud infrastructure. The Al module 4 can be proprietaiy, and thus accessible via a local network or local area network - LAN, or provided by third parties, in both cases it is previously trained for the generation and evaluation of In-Basket tests, according to a method 200. This is able to generate and evaluate specific In-Basket tests, complying with a specific scientific methodology, it is able to generate and understand texts in a natural language, by the use of Large Language Model - LLM, both in the written form, but also in the spoken one, by the use of appropriate functions, and it is therefore able to evaluate both the content but also the tones of the pronunciation and the emotions of candidates C.

[0049] In a preliminaiy training step 210., a training dataset A is provided, comprising data specially selected and pre-processed by occupational psychologists and other specialized staff, in particular technical documents.

[0050] The training dataset A comprises In-Basket test documents written by skilled persons, including examples of In-Basket tests. Each test represents a simulation of complex business situations that require a series of decisions by the participant in the selection. These texts are used as a basis to train the Al in understanding and generating similar scenarios, ensuring that the model is able to replicate the complexity and depth of real situations.

[0051] The training dataset A further comprises a skill statement that provides a detailed description of the skills and abilities that the Al should be able to evaluate. The statements include specific definitions of skills. The Al module 4 has converted these skills into labels or metadata to be used as guides in identifying and evaluating the participants in the selection during In-Basket tests.

[0052] The training dataset A also comprises evaluation and score documents, in particular, a set of rules and score schemes that define how an In- Basket test should be evaluated. These documents include guidelines for assigning scores, both qualitative and quantitative ones, to the answers of the participants in the selection. These rules, directly integrated into the commands or prompts, have been used by the Al module 4 as evaluation criteria to standardize the results, ensuring consistency and accuracy in skill evaluation.

[0053] Said Al module 4 outputs output data of the In-Basket test and a corresponding evaluation.

[0054] In particular, In-Basket tests, generated by using advanced language models, are provided through external APIs. The scenarios are designed to be realistic and adaptable, creating situations that can be used to test the skills of the participants in the selection in a simulated business context.

[0055] The skill evaluation is performed by evaluating the answers of the candidates, comparing them with the skill statements. By using evaluation algorithms based on classification models, the Al module 4 assigns scores and qualitative indications which accurately reflect the abilities demonstrated by the participants in the selection.

[0056] The Al module 4 was subjected to a tuning, through a combination of automatic techniques and manual interventions.

[0057] In a first step 220., a fine-tuning was performed on pre-trained models, by using the external APIs, and the pre-trained model was further optimized by fine-tuning, based on the project-specific datasets. This process has involved the adjustment of key model parameters, to maximize the precision and consistency of the generations.

[0058] In a second step 230., an iterative feedback cycle was activated. After each iteraction for test generation, the scenarios produced by the Al were analysed by specialists in the sector. The questions and / or texts that needed improvements were manually modified and reinserted into the dataset for further training sessions. This process has allowed to refine the model abilities in generating high-quality contents.

[0059] Each training cycle was followed by a test and validation step with results compared with predetermined benchmarks. It was thereby possible to identify the incremental improvements and bring targeted corrections to the model through the specific analysis of datasets and training parameters, used in turn for each fine-tuning step, and of the possible modifications brought during the process.

[0060] Said database 5 comprises the detailed configuration information of the In-Basket test to be generated and of the interaction mode of the plurality of multimedia devices 3 during the test. It further comprises the questions in particular, the number, the type whether open or closed, static, conditional i.e. generated depending on the previous received answers, it comprises the enabled PINs i.e. the identification codes, which are anonymous for the candidates, with which the multimedia devices 3 and the management device 1 introduce to the data transfer unit 2. It further comprises the type of interaction, such as the requests of web resources and communications via API on port 80, used for HTTP (HyperText Transfer Protocol) traffic and port 443, used for HTTPS (HyperText Transfer Protocol Secure) traffic, i.e. the mode with which said plurality of multimedia devices 3 interact in the process and it comprises multimedia data such as textual contents, audios, images, videos and / or virtual reality.

[0061] The method 100 for operating the system S for the management of the procedure of a public competition comprises the following steps.

[0062] In an activation step 110., said management device 1 sends a request to said data transfer unit 2 for the generation of an In-Basket test. As a consequence, this request also activates the interaction between said plurality of multimedia devices 3 and said data transfer unit 2. In a first transfer step 120., said data transfer unit 2 sends the request to said Al module 4, via API.

[0063] In a generation step 130., the Al module 4 performs the generation of the In-Basket test and sends it to said data transfer unit 2. In this step, the In-Basket test is not yet completely generated since some contents are dynamic and will be thus generated later, ad hoc, based on the answers sent by said plurality of multimedia devices 3 to said data transfer unit 2.

[0064] In a second transfer step 140., said data transfer unit 2 sends the first contents of the In-Basket test to said plurality of multimedia devices 3.

[0065] In a request step, said plurality of multimedia devices 3 sends new requests, caused by the direct actions taken on multimedia devices 3 by the candidate C, of contents, together with new contents, to said data transfer unit 2. The new contents can be audios, videos, images, texts, and are dynamically generated and customized based on the received request.

[0066] In a third transfer step 150., said data transfer unit 2 sends again a request to said Al module 4 that provides new contents to said data transfer unit 2.

[0067] The interaction between said data transfer unit 2, said plurality of multimedia devices 3 and said Al module 4 is continuous, up to the end of the In-Basket test and the new requests arise from said plurality of multimedia devices 3.

[0068] In a final step 160., which corresponds to the end of the In-Basket test, said plurality of multimedia devices provide the test result and / or contents to said data transfer unit 2, that simultaneously sends an evaluation request to said Al module 4. In an evaluation step 170., said Al module 4 performs the evaluation of the In-Basket test and sends the outcome to said data transfer unit 2, that sends in turn the outcome to said management device 1.

[0069] Optionally, in said step 170., based on the configurations set in said database 5, said data transfer unit 2 can send test results to said plurality of multimedia devices 3 as well.

[0070] As it is evident from the above-mentioned description, the advantage of the system and method that are the subject matters of the present invention is to employ dedicated software, Artificial Intelligence modules and multimedia devices, so as to automate the execution of In-Basket tests, thus allowing the specialists in the sector to focus on training the Artificial Intelligence module, rather than on implementing the single In- Basket tests.

[0071] Thereby, it is thus possible to automatically generate In-Basket tests that comply with the scientific methodologies used in the staff selection field.

[0072] Moreover, by using an automated and computerized process dedicated to the generation, management, administration and evaluation of In-Basket tests, it is possible to reduce times and costs for managing the whole process. Moreover, it is possible to perform a more realistic simulation, due to the aid of audios, videos and virtual reality viewers, that it, it is possible to perform the In-Basket test in a manner that is much more similar to reality.

[0073] Moreover, due to the use of a microphone and speakers / headphones it is possible to interact, during the simulation, in a more immediate and above all natural manner, that is, even with no need to write, but rather by speaking and listening. This type of interaction simulates in an even more complete manner the real interaction with co-workers.

[0074] It is also possible to generate In-Basket tests that are balanced in terms of difficulty and complexity level, by the certain application of the same scientific rules. Finally, it is possible to ensure a greater transparency and objectivity, since the outcome evaluation, being it performed in a computerized manner, is based on scientific rules, and being it automatically performed, is free from contingent factors and unaware biases that would alter the evaluation.

[0075] Obviously, in order to meet contingent and specific requirements, a person skilled in the art will be allowed to bring several modifications and alternatives to the above-described support, all however falling within the scope of protection of the invention as defined by the following claims.

Claims

CLAIMS1. A system (S) for the generation, management and evaluation of at least one In-Basket test for public competitions and selections for at least one candidate (C), comprising at least one management device (1), that can be used by an operator, to activate the procedure for the generation, management and evaluation of said at least one In-Basket test; at least one data transfer unit (2), operatively connected to said management device (1), that is able to transfer data in said system (S) for the generation, management and evaluation of said at least one In-Basket test; a plurality of multimedia devices (3), operatively connected to said at least one data transfer unit (2), that can be used by said at least one candidate (C) to perform said at least one In-Basket test, based on the data received from said at least one data transfer unit (2); at least one Artificial Intelligence module (4), operatively connected to said at least one data transfer unit (2) that is able to generate said at least one In-Basket test based on the data received from said at least one data transfer unit (2) and to evaluate said at least one In-Basket test performed by said at least one candidate (C) .

2. The system (S) according to the previous claim, characterized in that it comprises at least one database (5), operatively connected to said at least one data transfer unit (2), comprising the information needed for the configuration of said at least one In-Basket test to be generated and the operating modes of said plurality of multimedia devices (3).

3. The system (S) according to any one of the previous claims, characterized in that said plurality of multimedia devices (3) comprises smartphones, virtual reality viewers, laptops, PCs, tablets, web browsers, speakers, headphones and microphone.

4. The system (S) according to any one of the previous claims, characterized in that said at least one data transfer unit (2) communicates with said Artificial Intelligence module (4) via application programming interfaces - API, supported by an Internet network.

5. A method (100) for operating the system (S) for the generation, management and evaluation of at least one In-Basket test for public competitions and selections for at least one candidate (C), according to any one of the previous claims 1-4, characterized in that it comprises the following steps:

110. by means of said at least one management device (1), sending a request to said at least one data transfer unit (2) for the generation of said at least one In-Basket test;120. by means of said at least one data transfer unit (2), sending said request to said Artificial Intelligence module (4);130. by means of said Artificial Intelligence module (4), generating said at least one In-Basket test and sending it to said data transfer unit (2);140. by means of said at least one data transfer unit (2), sending the first contents of said at least one In-Basket test to said plurality of multimedia devices (3);150. by means of said at least one data transfer unit (2), sending again a request to said Artificial Intelligence module (4) that provides new contents to said at least one data transfer unit (2);160. once the execution of said at least one In-Basket test by said at least one candidate (C) is finished, by means of said plurality of multimedia devices (3), providing the test result and / or contents to said at least one data transfer unit (2) that simultaneously sends an evaluation request to said Artificial Intelligence module (4);170. by means of said Artificial Intelligence module (4), performing theevaluation of said at least one In-Basket test and sending the outcome to said at least one data transfer unit (2) that sends the outcome to said at least one management device (1).

6. The method (100) according to the previous claim, characterized in that in said step 120., said at least one data transfer unit (2) sends the request to said Artificial Intelligence module (4) via application or application programming interface - API, supported by an Internet network.

7. The method (100) according to any one of the previous claims 5 or 6, characterized in that after said step 140., said plurality of multimedia devices (3) send new content requests, together with new contents to said at least one data transfer unit (2).

8. The method (100) according to any one of the previous claims 5-7, characterized in that the interaction between said at least one data transfer unit (2), said plurality of multimedia devices (3) and said at least one Artificial Intelligence module (4) is continuous, up to the end of said at least one In-Basket test.

9. The method (100) according to any one of the previous claims 5-8, characterized in that, in said step 170., based on the configurations set in said at least one database (5), said at least one data transfer unit (2) sends the results of the evaluation of said at least one In-Basket test to said plurality of multimedia devices (3).

10. The method (100) according to any one of the previous claims 5-9, characterized in that said Artificial Intelligence module (4) is trained according to a training method (200), comprising the following steps:

210. providing a training dataset (A) comprising data selected and pre- processed by occupational psychologists and other specialized staff, a skill statement, evaluation and score documents, a set of rules and score schemes that define how an In-Basket test should be evaluated;220. performing a fine-tuning on pre-trained models and further optimizing the pre-trained model by fine-tuning based on the projectspecific datasets;230. activating an iterative feedback cycle.

Images