Trojan malware detection using formal verification

Abstract

A method of detecting malware in a hardware design for an electronic device. In a processor: receiving (205), in a test environment, a digital circuit portion under test, the digital circuit portion under test having at least one input and at least one output; receiving (210) at least one input; receiving (215) one or more assertions relating to the integrity of the digital circuit portion under test; allocating (220) a first input as watched data in dependence on a free variable; using (225) one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data; detecting (230) the possible presence of malware in the digital circuit portion under test in dependence on the verification; and in response to detecting the possible presence of malware, providing (235) a test environment output indicative of the possible presence of detected malware.

Description

[0001] TROJAN MALWARE DETECTION USING FORMAL VERIFICATION

[0002] Field of the invention

[0003] The present invention relates to a method and system of detecting malware in a hardware design for an electronic device. to the invention

[0004] It is important to verify that hardware designs (e.g. system-on-chip (SoC) designs) meet their required specification both before the design is implemented and whilst the design is in use. It is known to verify hardware designs using simulation verification or formal verification.

[0005] As part of the verification process, it is investigated whether a malware is present in the hardware design. Malware is software or hardware with malicious intent which causes an electronic circuit or computer to operate in a way other than what is intended. The present invention relates to the detection of Trojan malware which has been integrated into an electronic circuit design and designed to breach the integrity and / or availability of data within the electronic circuit. The Trojan malware causes the circuit to operate in a way other than what is intended, with the effect of breaching the integrity and / or availability of data, for example it may modify, delete, replace, encrypt or decrypt data, or cause the circuit to output data which should not be output. Delivery of the functional modification is initiated by a trigger, which is built into the Trojan malware. The trigger can be dependent on a variety of signals in the hardware design and is likely to be hidden and difficult to detect within the hardware design.

[0006] It is in this context that the present inventions have been devised. of the invention

[0007] In accordance with an aspect of the present invention, there is provided a method to detect malware in a hardware design for an electronic device. The method may be performed in a processor. The method may comprise receiving (e.g. in a test environment) a digital circuit portion under test representing hardware for the electronic device. The digital circuit portion under test may have at least one input and at least one output. The method may comprise receiving at least one input. The method may comprise receiving one or more assertions relating to the integrity and / or availability of the digital circuit portion under test. The method may comprise allocating a first input as watched data in dependence on a free variable. The method may comprise using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data. The method may comprise detecting the possible presence of malware in the hardware represented by the digital circuit portion under test in dependence on the verification. The method may comprise, in response to detecting the possible presence of malware in the hardware represented by the digital circuit portion under test, providing a test environment output indicative of the possible presence of malware detected in the hardware represented by the digital circuit portion under test.

[0008] In accordance with a further aspect of the invention, there is provided a system to detect malware in a hardware design for an electronic device. It may be that the system comprises a digital circuit portion under test representing hardware for the electronic device. It may be that the digital circuit portion under test has at least one input and at least one output. It may be that the system comprises a test bench. It may be that the system comprises one or more state machines used to verify one or more assertions relating to the integrity and / or availability of the digital circuit portion under test. It may be that the system comprises an assertion unit in communication with the one or more state machines. It may be that the assertion unit is configured to receive the one or more assertions relating to the integrity and / or availability of the digital circuit portion under test. It may be that the test bench is configured to receive at least one input. It may be that the test bench is configured to allocate a first input as watched data in dependence on a free variable. It may be that the test bench is configured to verify the one or more assertions in dependence on the behaviour of the output of the watched data. It may be that the test bench is configured to detect the possible presence of malware in the digital circuit portion under test in dependence on the verification. It may be that the test bench is configured to, in response to detecting the possible presence of malware in the digital circuit portion under test, providing a test environment output indicative of the possible presence of malware detected in the digital circuit portion under test.

[0009] Advantageously, the allocation of watched data in dependence on a free variable provides an arbitrary selection of the watched data. This creates an additional element of non-determinism in the abstract model so that the watched data is not set on the very first appearance of the watched data on the input of the digital circuit portion under test but also on later occurrences of the watched data on the input of the digital circuit portion under test.

[0010] It may be that a SAT solver is used to control the free variable. It may be that the free variable is implemented in an equation to be solved in a SAT solver. It may be that the SAT solver repeatedly arbitrarily selects a value for the free variable. It may be that if the SAT solver can solve the equation for the arbitrarily selected value of the free variable, then the assertion fails. Therefore, the arbitrarily selected value for which the assertion fails may be output as a counter-example.

[0011] Simulation verification techniques can be defeated by Trojans which have very specific triggers, because simulation verification may be unlikely to ever provide the trigger stimulus. However, formal verification can determine that errors in outputs are possible without determining the specific triggers. The scope of the verification is greatly increased by the allocation of watched data in dependence on a free variable. Both the watched data - and the timing of the watched data - can be abstracted allowing detection of Trojans with hard to find triggers.

[0012] Advantageously, the method and system are able to verify the digital circuit portion under test using multiple occurrences of the watched data being entered in the input. This is particularly beneficial because using multiple occurrences of the watched data allows for completeness of the verification in that all possible input data values on the input and all possible inter-leavings of time can be observed. This means that it is possible to determine whether there is malware present in the digital circuit portion under test at any state at any time.

[0013] It may be that the method comprises tracking a single symbolic data chosen by an arbitrary point of sampling in time in dependence on the free variable. It may be that the test bench is configured to track a single symbolic data chosen by an arbitrary point of sampling in time in dependence on the free variable. Therefore, using temporal abstraction (using a non-deterministic time point), data abstraction is achieved (e.g. by choosing an arbitrary data value in dependence on the free variable to track all data values).

[0014] Typically, the assertions relate to the progress of the watched data through the digital circuit portion under test (for example, output of the watched data from the digital circuit portion under test) and refer to the watched data.

[0015] Advantageously, malware is detected when watched data does not progress through the digital circuit portion under test as expected. This is because the assertion fails the verification because the watched data does not appear on an output of the digital circuit portion under test at the time, or in the way that it should do.

[0016] It may be that the malware comprises Trojan malware in the hardware represented by the digital circuit portion under test.

[0017] Typically, Trojan malware is malware which is integrated into the hardware design disguised as genuine circuitry. The Trojan malware typically modifies, deletes or replaces data, or transmits data, or changes circuit function. The functional modification of the Trojan malware is initiated by a trigger. It may be that the method is to detect Trojan malware in a hardware design for an electronic device. It may be that the system is a system to detect malware in a hardware design for an electronic device.

[0018] Advantageously, the use of arbitrary selection by allocating watched data in dependence on a free variable is particularly suited to detecting Trojan malware because the free variable allows Trojan malware that is embedded deep into the hardware design to be detected. By deep Trojan malware, it is intended to refer to Trojan malware with a trigger that is not triggered at the first instance of the trigger in the hardware design. The Trojan malware is typically carefully concealed within the hardware design so as to not immediately be detected by conventional Trojan malware detection methods.

[0019] For example, formal verification may be used to verify whether the assertion that when the symbolic value ‘B’ is input into the hardware design, it appears on the correct output, at the correct time and in the correct form. The digital circuit portion under test may include a Trojan malware. The trigger of a Trojan malware may be the 100thappearance of the symbolic value ‘B’ on an input of the hardware design. Taking the first appearance of ‘B’ on the input of the hardware design, no Trojan malware would be detected from the input ‘B’ for the first time because the trigger of the 100thappearance of ‘B’ on the input has not occurred. Formal verification without the methods and systems of the claimed invention would then pass the assertion because the first appearance of ‘B’ on the input appears on the correct output, at the correct time and in the correct form. However, with the arbitrary selection by allocating watched data in dependence on a free variable as defined in the present invention, the 100thappearance of ‘B’ on the input is a reachable state of the digital circuit portion under test. Since the 100thappearance of ‘B’ on the input is the trigger for the functional modification of the Trojan malware, the formal verification will not pass the assertion. In this way, Trojan malware embedded deep into the digital circuit portion under test can be detected.

[0020] The trigger of the Trojan malware may be one or more of: (i) a data signal, (ii) a clock signal, (iii) a control signal. Typically, a data signal is a signal for the transport of data from one point to another point in the digital circuit portion under test. Typically, a clock signal is an internal clock cycle in the digital circuit portion under test in the test environment. Typically, a control signal is a signal carrying instructions for the digital circuit portion under test.

[0021] It may be that the digital circuit portion under test is a high-level language description of a portion of a System on Chip (SoC). Advantageously, the method and system described herein are particularly helpful for detecting malware in SoCs, where it is essential that malware is not present in the hardware design.

[0022] Typically, the digital circuit portion under test is a component of an SoC. It may be that the digital circuit portion under test is a high-level language description of a component of a System on Chip.

[0023] Typically, a SoC comprises one or more computing elements on a single chip connected by a network-on-chip (i.e. point-to-point connections between any processing element (e.g., CPU, or GPU) and memory (RAM), or peripherals such as USB, HDMI, ethernet etc.). The SoC may comprise one or more computing elements on a single chip connected by a fabric, one or more computing elements on a single chip connected by a bus. Typically, the one or more computing elements comprise at least one of: a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), one or more multimedia encoder / decoders, direct memory access (DMA), network interface card (NIC) and one or more data storage modules. It will be appreciated that this list is not exhaustive and other computer elements will be envisaged.

[0024] It may be that the digital circuit portion under test is a component of a pipelined process.

[0025] Advantageously, the application of the system and method to a component of a pipelined process is particularly useful because malware in such a component affects the subsequent components of the pipelined process. Therefore, it is beneficial to be able to determine whether there is malware in the digital circuit portion under test.

[0026] Typically, a pipelined process comprises a number of components connected in series.

[0027] Typically, the test environment is a test bench which forms the verification environment for the digital circuit portion under test.

[0028] Typically, at least one input (or at least one input port) is configured to receive input data and transfer the data to the digital circuit portion under test. It may be that the process of the input receiving input data and transferring the data to the digital circuit portion under test is a transaction. It may be that at least one input comprises an input bus. It may be that the at least one input port is connected to an input bus. It may be that at least one input port is part of an input bus. At least one input comprises a plurality of wires. It may be that at least one input port is connected to wire. At least one input may comprise one or more wires at any place or places within an SoC. The data may be a data record.

[0029] Typically, the watched data is data that is tracked through the digital circuit portion under test by the test bench (or in the test environment). It may be that the watched data is selected (e.g. at random) by the test bench (or in the test environment). It may be that the watched data is selected (e.g. at random) by the verification tool. It may be that the watched data is selected (e.g. at random) by one or more components representing a free variable. It may be that the watched data is data that is internal to the hardware design.

[0030] It may be that the watched data is symbolic data. It may be that the watched data is semi-symbolic. That is, the watched data may comprise data that is partially indicative of one or more bits used in the hardware design. The watched data may be abstracted data in that it represents a place holder for multibit data processed in the digital circuit portion under test. The watched data may be abstracted in time.

[0031] Typically, the watched data is allocated depending on the value of the free variable. The free variable is not a predetermined variable and is instead set by the test bench (or test environment). The first input does not refer to the first appearance of a data value on the input, and instead is used to refer to the data value on the input that is chosen (by the free variable) as the watched data. Typically, the free variable is used to drive the allocation of the watched data. The free variable may be controlled by the test bench to be ‘high’ or ‘low’. When the free variable is ‘high’ and a data value enters the input, that data value may be allocated as watched data. When the free variable is ‘low’ and a data value enters the input, that data value may not be allocated as watched data. In this way, the free variable drives the allocation of watched data.

[0032] It may be that the free variable comprises an undriven wire. Advantageously, using an undriven wire is an effective way of implementing a free variable for arbitrary selection for allocating the watched data in SystemVerilog for formal verification.

[0033] Typically, the undriven wire is never driven with specific values and the model checker (in the method and system) assigns Os and 1s to capture the non-determinism across all states.

[0034] In other examples, the free variable may comprise a register.

[0035] Typically, an assertion is an output property that needs to be verified (e.g. by the formal checker tool). The one or more assertions typically comprise a number of statements relating to properties of the hardware design. Typically, an assertion comprises a statement of something that must be proven to hold for all reachable states in the hardware design. It may be that the one or more assertions are provided in an assertion computer programming language, such as SystemVerilog Assertions (SV A). However, other types of assertion computer programming language will be envisaged, such as, but not limited to, Property Specification Language (PSL) and Open Verification Language (OVL).

[0036] The one or more assertions may relate to the integrity of the digital circuit portion under test. Typically, integrity of the digital circuit portion under test means that the data transported through the digital circuit portion under test is accurate. An example of an assertion relating to the integrity of the digital circuit portion may be an assertion to test the property of: “If a particular data is inspected at the entrance of the path, it has to appear with the correct value and at the correct time at the end of the path, at the correct output”.

[0037] The one or more assertions may relate to the availability of the digital circuit portion under test. Typically, availability of the digital circuit portion under test means that access to the data is restricted. An example of an assertion relating to the availability of the digital circuit portion may be an assertion to test the property of: “No outside elements were able to change, delay or reroute the data, and the data is inspected at correct place at the right time”. Typically, the one or more state machines may be one or more finite state machines (FSM). They may, for example, be implemented by an electronic circuit, such as a microprocessor in communication with a solid-state memory and executing a program. Typically, the one or more state machines each comprises a model that represents possible states of a hardware design. The one or more state machines can transition in and out of states. When the one or more state machines do not transition into the correct state for the watched data in the digital circuit portion under test, malware may be present in the digital circuit portion under test. In this way, the one or more state machines verify whether the one or more assertions hold for every reachable state in the digital circuit portion. If malware is present in the one or more state machines, the assertion fails. If malware is not present in the one or more state machines, the assertion passes.

[0038] The method may comprise monitoring the behaviour of the watched data as it appears on the output of the digital circuit portion under test. The test bench may be configured to monitor the behaviour of the watched data as it appears on the output of the digital circuit portion under test. The method may comprise tracking progress of the watched data through the digital circuit portion under test. The test bench may be configured to track progress of the watched data through the digital circuit portion under test.

[0039] It may be that verification of the one or more assertions takes into account the behaviour of the watched data (e.g. on the output). The test bench may be configured to verify the one or more assertions by taking into account the behaviour of the watched data (e.g. on the output).

[0040] Typically, the behaviour of the watched data refers to how the watched data progresses through the digital circuit portion under test.

[0041] Typically, the behaviour of the watched data refers to an output of the digital circuit portion under test in relation to the watched data. Typically, the behaviour of the watched data refers to a correlation between the watched data on the input and the watched data on the output. Typically, the behaviour of the watched data refers to the expected output of the digital circuit portion under test of the watched data (e.g. compared to the actual input of the digital circuit portion under test of the watched data). Typically, the behaviour of the watched data may comprise a timing of a transaction relating to the watched data, or output of the watched data (e.g. relative to a timing of the watched data on the input). Typically, the behaviour of the watched data may comprise a value of the watched data on the output (e.g. relative to a value of the watched data on the input). Typically, the behaviour of the watched data may comprise a location of the watched data on the output.

[0042] It may be that the digital circuit portion under test comprises an output. It may be that the output comprises an output bus. It may be that the output comprises a plurality of wires. It may be that the at least one output port is connected to an output bus. It may be that the at least one output port is connected to a wire.

[0043] Typically, based on the behaviour of the watched data, the one or more assertions are verified. Typically, if the behaviour of the watched data indicates that there exists a reachable state in the digital circuit portion under test that does not fulfil the requirements of an assertion, the verification of that assertion fails. Typically, if the behaviour of the watched data indicates that there is no state that exists in the digital circuit portion under test that fails to fulfil the requirements of an assertion, the verification of that assertion passes.

[0044] It may be that the assertion unit is distributed within the system between the test bench and outside of the test bench. It may be that the assertion unit is a component of the test bench (e.g. entirely within the test bench). It may be that the assertion unit is a component of the system outside of the test bench (e.g. entirely outside of the test bench).

[0045] Typically, depending on the result of the verification of the one or more assertions are verified (i.e. fail or pass), (e.g. Trojan) malware may be detected in the digital circuit portion under test. Typically, if the verification of an assertion fails, the method may comprise determining whether malware is (e.g. possibly) present in the digital circuit portion under test, and hence in the hardware represented by the digital circuit portion under test. Typically, if the verification of an assertion fails, the test bench may be configured to determine whether malware is (e.g. possibly) present in the digital circuit portion under test. It may be that the output indicative of the possible presence of malware detected in the hardware represented by the digital circuit portion under test comprises a visual indication of the possible presence of malware detected in the hardware represented by the digital circuit portion under test. It may be that the visual indication of the possible presence of malware in digital circuit portion under test comprises a symbol and / or message.

[0046] It may be that the method is a method for testing a hardware design prior to manufacture to detect the possible presence of malware in the hardware design for an electronic device.

[0047] It may be that the system is a system for testing a hardware design prior to manufacture to detect the possible presence of malware in the hardware design for an electronic device.

[0048] Advantageously, the method and system can be used to detect the possible presence of malware in a hardware design for an electronic device before the design is manufactured. This means that the possible presence of malware can be detected before the design is built in the pre-manufacturing stage.

[0049] It may be that the method comprises inputting the watched data into a buffer in dependence on the free variable.

[0050] It may be that the test bench is configured to input the watched data into a buffer in dependence on the free variable.

[0051] Advantageously, the buffer keeps the watched data constant through the digital circuit portion under test because the watched data is stored in the buffer. The buffer can be used in the test bench (or test environment) because the buffer is variable in all reachable states, but the watched data is protected inside the buffer and remains constant.

[0052] Typically, the buffer is a data buffer. Typically, the buffer stores inputted data as it progresses through the digital circuit portion under test. The digital circuit portion under test may comprise the buffer. The test bench may comprise the buffer. The buffer is typically connected to the at least one input such that data entering the input, to be inserted into the digital circuit portion under test, may be inputted into (e.g. added to or inserted into) the buffer.

[0053] Typically, symbolic data enters the input and is inputted into the buffer and remains as symbolic data in the buffer.

[0054] Typically, the free variable is used to drive the input of the watched data into the buffer. The free variable may be controlled by the test bench to be ‘high’ or ‘low’. When the free variable is ‘high’ and the watched data enters the input, the watched data may be inputted into the buffer. When the free variable is ‘low’ and the watched data enters the input, the watched data may not be inputted into the buffer. In this way, the free variable drives the input of the watched data into the buffer.

[0055] The method may comprise connecting the buffer to the output. It may be that the test bench is configured to connect the buffer to the output.

[0056] It may be that the method comprises providing an output indicative of a source (e.g. in the digital circuit portion under test) of the possible presence of malware in the digital circuit portion under test. It may be that the test bench is configured to provide an output indicative of a source, in the digital circuit portion under test, of the possible malware present in the digital circuit portion under test.

[0057] Advantageously, the visual output indicative of a source of the possible malware in the digital circuit portion under test allows the user to quickly identify the source of possible malware in the digital circuit portion under test.

[0058] It may be that the visual indication comprises a portion of a wave form presented in a different way to the rest of the waveform, e.g. shown in a different colour, outlined or marked differently. The user is typically able to identify said visual indication and investigate the source of the possible malware in the digital circuit portion under test. The user can therefore modify the hardware design to remove malware in the design in a much quick manner than if the user were required to individually work through each transaction or cycle in the design to identify the source of the possible malware. It may be that using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data comprises updating one or more registers of the test environment.

[0059] It may be that the test bench comprises one or more registers. It may be that the test bench is configured to update the one or more registers.

[0060] Advantageously, the one or more registers can be used to store data relating to the transactions of the digital circuit portion under test. This is beneficial because the behaviour of the watched data can be monitored using the registers.

[0061] Typically, the registers are configured to perform a read to the register in dependence on a transaction occurring in the digital circuit portion under test. Typically, the registers may be used to store data for monitoring the behaviour of the watched data. It may be that the one or more registers comprises a first register configured to register that the watched data entered the digital circuit portion under test (e.g. register that the watched data appeared on the input and optionally is inputted into the buffer). It may be that the one or more registers comprises a second register configured to register that the watched data exited the digital circuit portion under test (e.g. register that the watched data appeared on the output).

[0062] It may be that at least one of the one or more registers functions as a counter.

[0063] Advantageously, a register, when implemented as a counter can keep track of expected behaviour of the watched data. In particular, the counter can be used to determine when the watched data should exit the digital circuit portion under test.

[0064] Typically, a register functions as a counter when it increments and / or decrements based on a transaction occurring in the digital circuit portion under test. It may be that the counter is configured to increment when the first register has a particular value. It may be that the counter is configured to decrement when the second register has a particular value.

[0065] It may be that using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data comprises determining expected behaviour of the output of the watched data. It may be that using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data comprises detecting actual behaviour of the output of the watched data. It may be that using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data comprises comparing an expected behaviour of the output of the watched data to actual behaviour of the output of the watched data. It may be that using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data comprises verifying the one or more assertions in dependence on the comparison of behaviour of the output of the watched data.

[0066] It may be that the system comprises a comparison unit in communication with the one or more state machines. It may be that the comparison unit compares expected behaviour of the output of the watched data and detected behaviour of the output of the watched data. It may be that the test bench is configured to determine the expected behaviour of the output of the watched data. It may be that the test bench is configured to detect the actual behaviour of the output of the watched data. It may be that the test bench is configured to compare the expected behaviour of the output of the watched data to the actual behaviour of the output of the watched data. It may be that the test bench is configured to verify the one or more assertions in dependence on the comparison of behaviour of the output of the watched data.

[0067] Advantageously, the comparison of expected and actual behaviour of the watched data provides an efficient way of verifying the one or more assertions. This is because the watched data (in the digital circuit portion under test) is expected to have certain behaviour for a given assertion. If the actual behaviour of the watched data (in the digital circuit portion under test) is the same as the expected behaviour of the watched data (in the digital circuit portion under test), then the assertion holds for all reachable states in the hardware design of the digital circuit portion under test. However, if the actual behaviour of the watched data (in the digital circuit portion under test) is different to the expected behaviour of the watched data (in the digital circuit portion under test), then the assertion does not hold for all reachable states in the hardware design of the digital circuit portion under test.

[0068] It may be that the comparison unit is distributed within the system between the test bench and outside of the test bench. It may be that the comparison unit is a component of the test bench (e.g. entirely within the test bench). It may be that the comparison unit is a component of the system outside of the test bench (e.g. entirely outside of the test bench).

[0069] Typically, the expected behaviour of the watched data (in the digital circuit portion under test) refers to how the watched data progresses through the digital circuit portion under test for a given assertion. Typically, the expected behaviour of the watched data (in the digital circuit portion under test) refers to the expected output of the digital circuit portion under test for a given input of watched data. Typically, the expected behaviour of the watched data (in the digital circuit portion under test) may comprise an expected timing of a transaction relating to the watched data (in the digital circuit portion under test) (e.g. relative to a timing of the watched data being inputted on the input). Typically, the expected behaviour of the watched data (in the digital circuit portion under test) may comprise an expected value of the watched data (in the digital circuit portion under test) (e.g. relative to a value of the watched data on the input). Typically, the expected behaviour of the watched data (in the digital circuit portion under test) may comprise a location of the watched data on the output (e.g. relative to a location of the watched data on the input).

[0070] Typically, the actual behaviour of the watched data (in the digital circuit portion under test) refers to the observed output of the digital circuit portion under test for a given input to the digital circuit portion under test. Typically, the actual behaviour of the watched data (in the digital circuit portion under test) may comprise an observed timing of a transaction relating to the watched data (in the digital circuit portion under test) (e.g. relative to a timing of the watched data being inputted on the input). Typically, the actual behaviour of the watched data (in the digital circuit portion under test) may comprise an observed value of the watched data (in the digital circuit portion under test) (e.g. relative to a value of the watched data on the input). Typically, the actual behaviour of the watched data (in the digital circuit portion under test) may comprise a location of the watched data on the output (e.g. relative to a location of the watched data on the input).

[0071] Typically, the comparison between the expected and actual behaviour of the watched data comprises determining whether the actual and expected behaviour of the watched data (e.g. in the digital circuit portion under test) is the same. If the expected and the actual behaviour of the watched data (e.g. in the digital circuit portion under test) are the same, the assertion may pass verification. If the expected and the actual behaviour of the watched data (e.g. in the digital circuit portion under test) are different, the assertion may fail verification.

[0072] It may be that the method comprises receiving one or more assumptions relating to the digital circuit portion under test. It may be that the method comprises applying the one or more assumptions to the one or more state machines.

[0073] It may be that the system comprises an assumption unit in communication with the one or more state machines. It may be that the assumption unit is configured to receive one or more assumptions relating to the digital circuit portion under test. It may be that the test bench is configured to apply the one or more assumptions to the one or more state machines.

[0074] Advantageously, assumptions can be used to restrict the digital circuit portion under test to comply with specifications of the hardware design. That is, the assumptions can be used to place limitations on the digital circuit portion under test without requiring that the limitations are verified (i.e. tested to hold in all reachable states of the hardware design). In addition, the assumptions can be used to reduce the number of reachable states available to the one or more state machines when performing verification. This is particularly advantageous as reducing the number of states across which the assertion must be proven to hold reduces the amount of time and computing power necessary to perform the verification and ultimately detect whether it is possible that malware is present in the hardware design.

[0075] It may be that the assumption unit is distributed within the system between the test bench and outside of the test bench. It may be that the assumption unit is a component of the test bench (e.g. entirely within the test bench). It may be that the assumption unit is a component of the system outside of the test bench (e.g. entirely outside of the test bench).

[0076] It may be that the one or more assumptions are part of the digital circuit portion under test, which is in turn part of a hardware design. Typically, an assumption is a property of the hardware design that is assumed to hold for all reachable states of the design. An assumption is not a property that is tested whether it holds true for all reachable states of the hardware design. The one or more assumptions typically comprise a number of statements relating to properties of the hardware design. Typically, an assumption comprises a statement of something that holds for all reachable states of the design, without requiring proof (i.e. it is assumed to be true). It may be that the one or more assumptions are provided in an assertion computer programming language, such as SystemVerilog Assertions (SVA). However, other types of assertion computer programming language will be envisaged.

[0077] Typically, the term “unit” when used in relation to at least one of the assertion unit, assumption unit and comparison unit may refer to an electronic circuit, or at least a portion of an electronic circuit.

[0078] It may be that the electronic circuit is formed of circuitry, for example transistors and / or other hardware elements. The circuitry may be implemented to form one or more of: registers, flip flops, latches, logical operators, and mathematical operators.

[0079] It may be that the method comprises carrying out formal verification of the one or more assertions using a formal verification tool to search an entire reachable state space of the digital circuit portion under test to determine whether there is a state in which any of the one or more assertions relating to the integrity and / or availability of the digital circuit portion under test is not valid.

[0080] It may be that the system is configured to perform formal verification of the one or more assertions such that the test bench is configured to use a formal verification tool to search an entire reachable state space of the digital circuit portion under test to determine whether there is a state in which any of the one or more assertions relating to the integrity and / or availability of the digital circuit portion under test is not valid.

[0081] Advantageously, the method and system can be used for formal verification, which is quicker than simulation verification because it does not involve providing an individual input vector for each state.

[0082] Typically, formal verification uses a formal verification tool, such as Jasper (Jasper is a trade mark of Cadence Design Systems, Inc.), JasperGold (JasperGold is a trade mark of Cadence Design Systems, Inc.), VC Formal (VC Formal is a trade mark from Synopsys, Inc.), Questa PropCheck (Questa PropCheck is a trade mark from Siemens AG), Questa OneSpin Static Formal (Questa OneSpin Static Formal is a trade mark from Siemens AG) and SymbiYosys (SymbiYosys from YosysHQ GmbH). Typically, the formal verification tool comprises one or more SAT solvers, which are Boolean satisfiability problem solvers. Typically, formal verification is implemented by tracking the order of symbolic transactions, which are encoded using Boolean symbols rather than using individual Os and 1s to track every individual state of the transaction. Before the model checking is started conceptually, the one or more assertions may be logically negated before they are combined with register translation level (RTL), constraints and the model in the testbench to result in a Boolean formula. The Boolean formulas are unfolded for every time point during model checking and so they are sent to SAT solvers to compute if there is a satisfying assignment of Boolean values to variables (signals in the RTL, test bench and constraints) in this formula. Since the assertion was negated, when the SAT solver finds a satisfying assignment, this is actually indicative of malware possibly being present in the hardware design.

[0083] According to an aspect of the present invention, there is provided a computer readable storage medium having encoded thereon computer readable program code configured to perform the method described above.

[0084] Although the embodiments of the invention described with reference to the drawings comprise methods performed by a system, such as a computing device, the invention also extends to program instructions, particularly program instructions on or in a carrier, adapted for carrying out the processes of the invention or for causing a computer to perform as a computer apparatus of the invention. Programs may be in the form of source code, object code, a code intermediate source, such as in partially compiled form, or any other form suitable for use in the implementation of the processes according to the invention. The carrier may be any entity or device capable of carrying the program instructions.

[0085] For example, the carrier may comprise a storage medium, such as a ROM, for example a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example a floppy disc or hard disc. Further, the carrier may be a transmissible carrier such as an electrical or optical signal which may be conveyed via electrical or optical cable or by radio or other means. When a program is embodied in a signal which may be conveyed directly by cable, the carrier may be constituted by such cable or other device or means.

[0086] An example embodiment of the present invention will now be illustrated with reference to the following Figures in which:

[0087] Figure 1 is a schematic of a system according to an embodiment of the invention;

[0088] Figures 2 to 7 are flowcharts of a method according to an embodiment of the invention;

[0089] Figure 8 is a schematic of a computing device according to an embodiment of the invention; and

[0090] Figure 9 is an example waveform of a verification tool using a method according to an embodiment of the invention, where the x-axis depicts clock cycles.

[0091] Detailed Description of an Example Embodiment

[0092] Figure 1 is a schematic of a system 100 according to an embodiment of the invention used in formal verification techniques. The system 100 comprises a digital circuit portion under test 110, described in a hardware description language, and including an input bus 112, functioning as the input. In other examples, the input may not be limited to an input bus but may comprise wires from a number of different places within the SoC. In this example, the digital circuit portion under test 110 is a circuit portion of reduced instruction set computer-5 (RISC-V) processor, which forms part of a pipelined process and is a component of a System on Chip. The system 100 also comprises a test bench 120, also described in a hardware description language, which may include a buffer 122 (shown in dashed lines as this is optional). The buffer 122 is configured to store data received on the input bus 112 of the digital circuit portion under test. Data is transferred between the input bus 112 and the test bench (e.g. the buffer 122) by communication bus 105. The system 100 also includes a plurality of state machines 130 which are used to verify assertions across all of the reachable states of the digital circuit portion under test. The assertions are received by the assertion unit 140. The assertions relate to the availability and integrity of the digital circuit portion under test 110 and are statements of properties that are to be tested to hold true for all states of the design (at a particular clock cycle or eventually at some clock cycle). The assertion unit 140 is shown as outside of the test bench 120 in Figure 1. However, it will be appreciated that the assertion unit 140 may be part of the test bench 120 or distributed between the test bench 120 and the outside of the test bench 120.

[0093] The communication bus 105 inputs watched data that appears on the input bus 112 to the test bench 120. The test bench 120 verifies the assertions depending on the behaviour of the watched data. Based on the verification, the test bench 120 detects the possible presence of malware in the digital circuit portion under test 110 and provides an output to an external computing device 190 which indicates that the possible presence of malware has been detected in the digital circuit portion under test 110 and can also provide an output of the source of the possible malware. The external computing device 190 communicates with the communication bus 105 of the system but in some examples, the system 100 may have a communication module to communicate with the external computing device 190.

[0094] The test bench 120 also includes a first register 124, a second register 126 and a third register 128 which functions as a counter. The registers 124, 126 and the counter 128 are updated as watched data enters and exits the circuit portion under test.

[0095] The system 100 includes a comparison unit 150 which is in communication with the state machines 130 and the test bench 120 through communication bus 105. The comparison unit 150 compares expected behaviour of the watched data to actual behaviour of the watched data as will be discussed later. The comparison module 150 verifies the assertions depending on the outcome of the comparison. The comparison unit 150 is shown as part of the system 100 outside of the test bench 120 in this example, however it will be appreciated that the comparison unit 150 could be entirely within the test bench 120 or distributed between the test bench 120 and outside of the test bench 120.

[0096] The system 100 includes an assumption unit 160 which is in communication with the state machines 130 and the test bench 120 through communication bus 105. The assumption unit 160 receives assumptions relating to the digital circuit portion under test 110. The assumption unit 160 provides the assumptions to the test bench 120 and the assumptions are applied to the circuit portion under test 110. The assumption unit 160 is shown as part of the system 100 outside of the test bench 120 in this example, however it will be appreciated that the assumption unit 160 could be entirely within the test bench 120 or distributed between the test bench 120 and outside of the test bench 120.

[0097] The system 100 can be used to perform formal verification of the assertions using a formal verification tool to search an entire reachable state space of the digital circuit portion under test 110.

[0098] Figure 2 is a flowchart of a method 200 according to an embodiment of the invention. The method 200 will be exemplified by application of the method 200 to a First-Input- First-Output (FIFO) design. As will be understood, it is a requirement of a FIFO that data is output from the FIFO in the same order that it entered the FIFO.

[0099] In block 205, the digital circuit portion under test is loaded into the test environment. The digital circuit portion under test may be preconfigured and stored in a memory accessible to the test environment.

[0100] In block 210, the input bus ‘dataj’ receives an input of data. In this method, the input of the digital circuit portion under is connected to the buffer ‘wd’ of the test bench.

[0101] In block 215, the assertion to be verified is received. This assertion states that when the ready_to_sample_out register is ‘high’, the data on the output bus, data_o in the next clock cycle will be the watched data in the buffer. assert property (@(posedge elk) ready_to_sample_out |=> (data_o == wd));

[0102] It will be appreciated that this example method uses the positive edge of the clock signal (“@(posedge elk)”), however other examples could use the negative edge of the clock signal (“@(negedge_clk)”), or any other event in the design or testbench. The method also includes, in this example, method 600. Figure 6 is a flowchart of a method 600 according to an embodiment of the invention. In particular, the method 600 comprises receiving 605 assumptions and applying 610 the assumptions to the state machines. The two assertions used in this example are:

[0103] 1) assume property (@(posedge elk) empty_o |-> !pop_i); 2) assume property (@(posedge elk) full_o |-> IpushJ || popj);

[0104] Assumption 1 means that if FIFO is empty then there must not be a pop in the same clock cycle, where a pop is a read out of the FIFO. Assumption 2 means that if the FIFO is full then either there is no push, or if you do have a push, then there is also a pop in the same cycle.

[0105] The method also includes, in this example, method 400. Figure 4 is a flowchart of a method 400 according to an embodiment of the invention. In method 400 two registers are set up 405 in the test bench and are updated as will be discussed below. reg sampledjn; reg sampled_out;

[0106] There are also conditions for setting the registers which are defined as wire ready_to_sample_in; wire ready_to_sample_out;

[0107] Another register is provided as a counter as part of method 400 and set up to increment and decrement using the following code: assign incr = pushj && Isampledjn; assign deer = popj && !sampled_out; reg [DEPTH_BITS:0] data_count; sampledjn indicates that the watched data has entered in the FIFO. sampled_out indicates that watched data has exited the FIFO. data_count is used to count values ahead of watched data in the FIFO.

[0108] Therefore, when there is a push and the sampledjn register is low (i.e. the watched data has not been sampled in), there is an increment because the watched data is not input into the buffer yet. Once the watched data has entered the FIFO (via the buffer 122), the counter stops incrementing. When there is a pop and the sampled_out register is low (i.e. the watched data has not been sampled out), there is a decrement in the counter. The register data_count is the counter to determine how many items of data are ahead of the watched data in the FIFO. Once the watched data has exited the FIFO, the counter reads zero.

[0109] In block 220, the watched data is inputted into the FIFO in dependence on an undriven wire, which function as the free variable, for example using the code below: logic [DATA_WIDTH-1 :0] wd; always @(posedge elk) begin if (ready_to_sample_in) wd <= dataj; end

[0110] The above code inputs 305 watched data into the buffer, wd, in dependence on the free variable. This block 220 from the method 200 includes the method 300 shown in the flowchart of Figure 3 which includes method step 305 of inputting watched data into a buffer in dependence on the free variable

[0111] The following line sets an undriven wire using “sampling_point” which allows random selection of data in the input interface as the watched data. logic sampling_point;

[0112] Therefore, when: i) the sampling_point is high; ii) the increment is high (meaning that there is a push); and iii) the sampledjn register is low (meaning the chosen watched data has not already entered the FIFO), then ready_to_sample_in is set as high as the data in the input bus will be inserted into the buffer as watched data in the next clock cycle.

[0113] The watched data is written into the Fl FO when the undriven wire, functioning as a free variable, sampling_point is high.

[0114] The block 225 involves using the state machines to verify the assertion.

[0115] When: i) the data_count is 1 (meaning that all data ahead of the watched data has exited the FIFO and the next data in the order of input into the FIFO is the watched data); ii) the sampledjn register is high (meaning that the watched data entered the FIFO); iii) the decrement is high (meaning that there is a pop); and iv) the sampled_out register is low (meaning that the watched data has not already exited the FIFO) then the ready_to_sample_out is set as high and the watched data is output in the next clock cycle.

[0116] This is implemented in this example using the following code: assign ready_to_sample_in = sampling_point && incr && Isampledjn; assign ready_to_sample_out = (data_count==‘d1) && sampledjn && deer && !sampled_out always @(posedge elk or negedge resetn) if (Iresetn) begin sampledjn <= 1’bO; sampled_out <= 1’bO; end else begin sampledjn <= sampledjn || ready o_sampleJn; sampled_out <= sampled_out || ready o_sample_out; end

[0117] In this example, the assertion is verified by the method 500 depicted in Figure 5. In particular, the method 500 comprises determining 505 expected behaviour of the watched data and detecting 510 actual behaviour of the watched data. The expected and actual behaviours of the watched data are compared in block 515. Finally, the assertion is verified 520 depending on the comparison.

[0118] For example, the expected behaviour may be that the watched data in the buffer wd will appear on data_o at a given clock cycle ‘n’ as the counter reaches 1 in the clock cycle ‘n-T and the ready o_sample_out register is ‘high’. However, the actual behaviour may be that the incorrect data appears on data_o at a time when the watched data would be expected to appear on data_o. Therefore, the assertion that, for every reachable state in the design, when the ready_to_sample_out register is ‘high’, the data on the output bus, data_o in the next clock cycle is the watched data in the buffer, does not hold true.

[0119] Therefore, in block 230, the possible presence of malware is detected because the incorrect data is read out of the FIFO.

[0120] Subsequently, an output is provided 235 which indicates to the user that there is possible malware in the design.

[0121] The method 200 may be employed in the method 700 which is shown in the flowchart of Figure 7. The method 800 comprises carrying out 705 formal verification using 710 a formal verification tool to search every reachable state space. By searching every reachable state space, it is determined 715 whether there is a state in the assertion relating to the digital circuit portion under test is not valid.

[0122] If all verification steps are successfully complete, the design may be synthesized and used to create a semiconductor topology for manufacture. However, if malware is found, the design is instead reviewed, revised and verified again in the same way.

[0123] Figure 8 is a schematic of an exemplary computing-based device 800, and in which embodiments of the systems and methods described above for detecting malware in a hardware design for an electronic device.

[0124] The computing device 800 includes one or more processors 802 and a storage media 808. The components of the computing device 800, such as the one or more processors 802 and the storage media 808 communicate with one another via a system bus 803. The one or more processors 802 (e.g. central processing units (CPUs), graphics processing units (GFU5 or VPU5), digital signal processors (DSPs), microprocessors) execute computer executable instructions to control the operation of the computing device 800. The storage media 808 comprises a non-volatile storage medium, such as read only memory (ROM) 804 and a volatile storage medium, such as a random access memory (RAM) 806. The computing device 800 may be operable as part of a network connection to one or more remote computers, such as a remote computer, by a local area network (LAN) or wide area network (WAN). The computing device may communicate through the network via communications module 810.

[0125] The computing-based device 800 also includes an I / O controller which controls input device 814 and output device 816. The output device 816 is a display screen which is connected to the system bus 803. The display screen 816 provides the output indicative of the possible presence of malware. The input device 814 receives a user input indicative of commands and information entered into the computing device 800, for example a keyboard and / or mouse. In some cases the output device 816 and the input device 814 maybe a single component, e.g. a touchscreen.

[0126] The computing device 800 components may be distributed across a network in some examples.

[0127] Figure 9 is an example waveform of a verification tool using a method according to an embodiment of the invention, where the x-axis depicts clock cycles. This waveform represents a transaction inside the RISC-V processor.

[0128] In this example, the input data is shown on instructionj. This input is the operation code, which indicates what action needs to be taken with data. The design takes this instructionj as input, and stores it after changing it in accordance to the opcode, until it is its turn to appear on the output, instruction^.

[0129] For the purpose of demonstration, the Trojan malware is illustrated in line ‘trojan_activated’. In this example, the Trojan will only activate when the opcode in instructionj is ‘b110,1111 (JAL, jump and link) and after a certain number of JALs have already gone through the circuit portion without fault. Once the trojan is activated (activation is shown in clock cycle 5 on the ‘activatejrojan’ line), instead of the design storing the jaljarget_offset of the instructionj as it normally would, and making it appear on the output in due time, it stores the intended value plus 2, changing the intended target / instruction. In this example, ‘activate rojan’ is a signal that goes high when the Trojan is activated. In clock cycle 5, the Trojan is activated as shown in ‘activate rojan’ line. The Trojan can be seen in clock cycles 6 to 7 in the line ‘trojan_activated’ following its activation in clock cycle 5. The free variable, which is implemented by an undriven wire and allows for arbitrary selection of the watched data is shown in ‘sampling_point’ which goes high in clock cycle 5 which is when the ‘activate_Trojan’ goes high. In clock cycle 5, the ready_to_sample_in register is ‘high’. Therefore, in clock cycle 6, the watched data ‘hfffc 0002’ is inserted into the buffer, wd, as it appears in clock cycle 5 on jal_target_offset when the ready_to_sample_in register went ‘high’. In clock cycle 7, the data has been modified by the Trojan as ‘hfffc 0004’ appears on instructions, rather than ‘hfffc 0002’ as would be expected because that is the watched data inserted into the buffer. In practice, a SAT solver will vary the free variable and watched data to determine whether it can find any values of these inputs in which the output is not as expected and so one or more assertions fails. These inputs are then given as a counterexample.

[0130] Throughout the description and claims of this specification, the words “comprise” and “contain” and variations of them mean “including but not limited to”, and they are not intended to and do not exclude other components, integers, or steps. Throughout the description and claims of this specification, the singular encompasses the plural unless the context otherwise requires. In particular, where the indefinite article is used, the specification is to be understood as contemplating plurality as well as singularity, unless the context requires otherwise.

[0131] Features, integers, characteristics, or groups described in conjunction with a particular aspect, embodiment, or example of the invention are to be understood to be applicable to any other aspect, embodiment or example described herein unless incompatible therewith. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and / or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. The invention is not restricted to the details of any foregoing embodiments. The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed.

Claims

Claims1. A method to detect malware in a hardware design for an electronic device, the method comprising, in a processor: receiving, in a test environment, a digital circuit portion under test representing hardware for the electronic device, the digital circuit portion under test having at least one input and at least one output; receiving at least one input; receiving one or more assertions relating to the integrity and / or availability of the digital circuit portion under test; allocating a first input as watched data in dependence on a free variable; using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data; detecting the possible presence of malware in the hardware represented by the digital circuit portion under test in dependence on the verification; and in response to detecting the possible presence of malware in the hardware represented by the digital circuit portion under test, providing a test environment output indicative of the possible presence of malware detected in the hardware represented by the digital circuit portion under test.

2. The method of claim 1 , wherein the malware comprises Trojan malware in the hardware represented by the digital circuit portion under test.

3. The method of claim 1 or claim 2, wherein the free variable comprises an undriven wire.

4. The method of any preceding claim, wherein the method is a method for testing a hardware design prior to manufacture to detect the possible presence of malware in the hardware design for an electronic device.

5. The method of any preceding claim, comprising inputting the watched data into a buffer in dependence on the free variable.

6. The method of any preceding claim, wherein using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data comprises updating one or more registers of the test environment.

7. The method of any preceding claim, wherein using one or more state machines to verify the one or more assertions in dependence on the behaviour of the output of the watched data comprises: determining expected behaviour of the output of the watched data; detecting actual behaviour of the output of the watched data; comparing an expected behaviour of the output of the watched data to actual behaviour of the output of the watched data; and verifying the one or more assertions in dependence on the comparison of behaviour of the output of the watched data.

8. The method of any of preceding claim, comprising receiving one or more assumptions relating to the digital circuit portion under test and applying the one or more assumptions to the one or more state machines.

9. The method of any preceding claim, wherein the digital circuit portion under test is a high level language description of a portion of System on Chip.

10. The method of any preceding claim, comprising carrying out formal verification of the one or more assertions using a formal verification tool to search an entire reachable state space of the digital circuit portion under test to determine whether there is a state in which any of the one or more assertions relating to the integrity and / or availability of the digital circuit portion under test is not valid.

11. A computer readable storage medium having encoded thereon computer readable program code configured to perform the method of any of claims 1 to 10.

12. A system to detect malware in a hardware design for an electronic device, the system comprising:a digital circuit portion under test representing hardware for the electronic device, the digital circuit portion under test having at least one input and at least one output; a test bench; one or more state machines used to verify one or more assertions relating to the integrity and / or availability of the digital circuit portion under test; and an assertion unit in communication with the one or more state machines, the assertion unit configured to receive the one or more assertions relating to the integrity and / or availability of the digital circuit portion under test, wherein the test bench is configured to: receive at least one input, allocate a first input as watched data in dependence on a free variable, verify the one or more assertions in dependence on the behaviour of the output of the watched data, detect the possible presence of malware in the digital circuit portion under test in dependence on the verification, and in response to detecting the possible presence of malware in the digital circuit portion under test, providing a test environment output indicative of the possible presence of malware detected in the digital circuit portion under test.

13. The system of claim 12, wherein the malware comprises Trojan malware in the hardware represented by the digital circuit portion under test.

14. The system of claim 12 or claim 13, wherein the free variable comprises an undriven wire.

15. The system of any of claims 12 to 14, wherein the system is a system for testing a hardware design prior to manufacture to detect the possible presence of malware in the hardware design for an electronic device.

16. The system of any of claims 12 to 15, wherein the test bench is configured to input the watched data into a buffer in dependence on the free variable.

17. The system of any of claims 12 to 16, wherein the test bench comprises one or more registers and the test bench is configured to update the one or more registers.

18. The system of any of claims 12 to 17, comprising a comparison unit in communication with the one or more state machines, the comparison unit to compare expected behaviour of the output of the watched data and detected behaviour of the output of the watched data, wherein the test bench is configured to: determine the expected behaviour of the output of the watched data; detect the actual behaviour of the output of the watched data; compare the expected behaviour of the output of the watched data to the actual behaviour of the output of the watched data; and verify the one or more assertions in dependence on the comparison of behaviour of the output of the watched data.

19. The system of any of claims 12 to 18, comprising an assumption unit in communication with the one or more state machines, the assumption unit configured to receive one or more assumptions relating to the digital circuit portion under test, and wherein the test bench configured to apply the one or more assumptions to the one or more state machines.

20. The system of any of claims 12 to 19, wherein the digital circuit portion under test is a high level language description of a portion of a System on Chip.

21. The system of any of claims 12 to 20, wherein the system is configured to perform formal verification of the one or more assertions such that the test bench is configured to use a formal verification tool to search an entire reachable state space of the digital circuit portion under test to determine whether there is a state in which any of the one or more assertions relating to the integrity and / or availability of the digital circuit portion under test is not valid.

Images