This blog talks about the Vortex project and the work that we did during our remote internship under Dr. Hyesoon Kim. We started out by first understanding the architecture design of the Vortex project. For this, there were a set of assignments given to us that helped us in becoming familiar with the codebase. Further, we worked on extending the RTL design to include support for the RISC-V 64-bit ISA, as the current project is only suited for the 32-bit ISA.
Understanding the existing codebase
In order to understand the existing codebase, we were given a set of six assignments with increasing levels of complexity. The first two assignments were pretty straightforward, involving adding certain counters to check the performance of the GPU or to check the number of active requests made by all the threads in an active warp to the same memory address.
Assignment 1 — counting active threads
In the first assignment, the goal was to find out the number of active threads per cycle during program execution. Through this assignment, we were introduced to interfaces in SystemVerilog and how they are used in a large project. Any interface has two modules within it — a slave module and a master module — and using either of these two, an interface can be used to connect various modules together. We had to add the performance counter at various stages of the architecture. The actual counting of the active threads was done in VX_issue.sv, the issue stage where all the instructions are scheduled.

In order to print the average number of active threads per cycle, the total count of active threads from the RTL had to be passed to the VX_utils.cpp file. This was done by storing the active threads in a memory location and retrieving the same in the C++ file.
Assignment 2 — duplicate memory requests
For the second assignment, we had to calculate the number of times there was a warp that had all the active threads requesting the same memory address. VX_lsu_unit.sv had a signal called req_is_dup which was set high when all the active threads requested the same memory address as thread 0. Using this and checking if the request is valid, the counter value was incremented at every clock cycle.

Assignment 3 — hardware prefetcher
The later assignments were a bit more involved, as the number of files to change increased and testing of the code was explained. In assignment 3, the goal was to add a hardware prefetcher in the memory system for Vortex. The SimX implementation of the same is explained in a previous blog post. The load-store unit in Vortex takes a vector of addresses, each having valid and read status bits for each thread concatenated with the address. The valid bit is set low for inactive threads. The memory address is calculated by adding an offset to the base address. The prefetch address is computed every time a load address is computed; the prefetcher calculates the next line cache address and sends it to the data cache. Prefetch requests are only handled if there are no existing prefetch requests in the pipeline.

Assignment 4 — debugging
Assignment 4 involved debugging assignment 3 and showed in general how to debug the codebase. In order to run the processor in debug mode, a --debug flag has to be given to the run command, which generates a run.log file containing information about all the instructions executed by the processor and the corresponding trace flags printed by each module. Here, we can see the status of the flags and check if they are properly set after each instruction. Along with the log file, it also generates a VCD file that can be used to visualize all the signal waveforms using suitable waveform viewing applications like GTKWave or ModelSim.

Assignment 5 — adding a new instruction
Assignment 5 was very interesting, as we had to add a new instruction to the existing codebase. The aim was to add software prefetch instructions in RTL as well as in the SimX simulator. Software prefetch is an instruction that is currently not supported by the RISC-V ISA, so the corresponding assembly code had to be created first. The opcode for non-RISC-V instructions is 7'b1101011 (or 0x6b) and the identifier 5 is used for this instruction. Correspondingly, in the decode stage of the RTL, a case for identifier 5 had to be added and the prefetch flag had to be set. This tag was used later in VX_instr_demux.sv, which sends the instruction to the LSU unit. In this assignment we were also introduced to testing and adding tests for the new instruction. When the corresponding test is run, the kernel.dump file shows the opcode for the software prefetch instruction.
Assignment 6 — prefetcher performance counters
Finally, assignment 6 was the culmination of all the previous assignments, as we had to add code for calculating the performance of the software prefetcher. Using it, we had to find out the number of unique requests to main memory, unused prefetch blocks, and the number of late prefetches. The first part of the assignment was to extend the metadata tag to include information about prefetch requests. This tag bit was then used to calculate the number of unused prefetch blocks. These counter values were displayed to the terminal using vx_utils.cpp, where the value of the counter was taken from the required memory address.

In order to find the number of unique requests to main memory, a very simple logic is used. As we have seen earlier, multiple prefetch requests were generated to the same address, so the first request generated for the prefetch will always be a cache miss. All the subsequent requests will hit, as the corresponding block would be added to the cache. So we just need to find out the number of times a cache miss occurs inside a bank.

Inside VX_cache.sv, all such requests are added together per cycle to generate the total prefetch requests. This information is then used by the C++ file to display the total number of prefetch requests. Similar logic is used to calculate the total number of unused prefetch blocks and the number of late prefetches. Using the same prefetch test created in assignment 5, the codebase was tested.

Overall, these assignments are adequate for understanding the flow of instructions through the pipeline and how all modules are connected with each other through interfaces. We strongly recommend everyone first try out all the assignments in order to get a better understanding of the Vortex codebase.
Extending Vortex to the RISC-V 64-bit ISA
In order to extend the existing architecture, the first step was to parameterize all the signals in the Vortex codebase so that they can be changed from a single place. Earlier, address widths and data widths were all hard-coded to 32 bits (or [31:0]), which made it a fixed-length architecture. Now all such signals — instruction addresses, data addresses, ALU operands, ALU results, and so on — are parameterized with a variable called ADDR_WIDTH. The value of ADDR_WIDTH can be modified in a single place at hw/rtl/VX_define.vh.




After parameterizing the bus width, the next step is to add opcodes and func values for the 64-bit variants of instructions. This is first added to VX_define.vh, following the conventions in the official RISC-V manual.


Next, we changed the VX_decode.sv stage so that it can decode 64-bit instructions based on the opcode and func field values from the instruction. Even for 64-bit instructions the opcode and all the values remain the same, as only the lower 32 bits are used, so extending the instruction width from 32 bits to 64 bits was an easy task. Even if the opcodes for the 64-bit variants of instructions like ADD, SUBTRACT, and MULTIPLY are different, the execute units — the ALU, the multiply/divide unit, and so on — are still the same, as they are also parameterized and can support 64-bit operations. So the op_type for these instructions is also the same as their 32-bit variants.

As stated earlier, the remaining units in the execute and LSU stages remain the same, as they are already parameterized and can be changed to 64-bit variants as required. Since the 64-bit RISC-V ISA is compatible with all the 32-bit variants, it is easy to extend the same in the Vortex codebase.
Testing of these modules was done and it was found that all the existing tests in the OpenCL and regression folders were working properly. However, there were certain tests that were giving incorrect results and they are yet to be looked at. Future work involves correctly running the remaining tests in the runtime folder.