Description In this project, you will create a simple MIPS simulator. Your simulator will read a binary file containing a MIPS program and execute that program. This will occur in two steps. First, your program will generate the assembly code for the given MIPS program (disassembler). Second, your program will create an instruction-by-instruction simulation of the MIPS program. This simulation will execute instructions sequentially (non-pipelined) and output the contents of all registers and memory (the state of the processor and memory) after each instruction. You will not have to implement exception/interrupt handling You will be given the expected output for each input program. Your output must exactly match the expected output. The program will be graded simply by using the diff program. The diff program simply lists the differences between two files. Your output should have no differences with the expected output. We will suppress white space differences, so if you have a space instead of a tab, that is OK. The command to run diff and suppress white space differences is the following: diff -w Your disassembler can be reused for the next project, so try to write it in a way such that you can separate it from your code to execute the program. In fact, it is easier to write and debug the disassembler first, then write the emulation portion of the project. Implementation You may implement this project in any programming language of your choosing. You MUST include instructions in a README file that indicate how to compile (if necessary) and run your program. You MUST include a makefile to compile the code if it is in a language that requires compilation. Examples are linked on the course web site. Your code will be graded on linux on the home server. The only reason it will not be graded there is if you use such an obscure language that home does not have the correct compilers/interpreters for it. C/C++, and Java program must be able to run on home. You may develop locally, then test on home at the end. This program should require NO system libraries, except I/O, so there should be no trouble. 1 Details Refer to the MIPS instruction set architecture PDF that is posted along with the course notes on the course website. It provides the details for all MIPS instructions. NOTE that we are making the following changes to the instruction set architecture: Instead of a 6 bit opcode, we will use a 5 bit opcode that is preceded by a valid bit. The valid bit will be set to 1 if the instruction is valid and should be executed. If the valid bit is set to 0, then the instruction has no effect (it is effectively a NOP). The opcodes will be the same as those in the MIPS instruction set, just ignore the most significant bit (the first bit). We will not change the functionality of any instruction, simply we use this convention for the opcode. The table below illustrates this change: Bit 31 Bits 30 – 26 Bits 25 – 0 Valid bit Opcode The rest of the instruction A suggestion to make coding the program easier. First, read the entire file and simply print if each instruction is valid or not. This will force you to get the general structure of the disassembler in place and debugged before you begin the more complicated step of disassembly. You will be given an input file containing a sequence of 32 bit instruction words. Assume that the first instruction is at memory address 96. The final instruction in an instruction sequence is ALWAYS a “BREAK” instruction. Following the break instruction is a sequence of 32 bit 2’s compliment signed integers for the program data. These continue until the end of file. Your simulator/disassembler must support the following MIPS instructions. Check the MIPS manual for details on instruction representation and operation for each instruction. J, JR, BEQ, BLTZ ADD, ADDI, SUB SW, LW SLL, SRL MUL, AND, OR, MOVZ NOP Input Your program must accept command line arguments for execution. The following arguments must be supported (Executable named “mipssim”): mipssim –i INPUTFILENAME –o OUTPUTFILENAME Your program will produce 2 output files. One named OUTPUTFILENAME_sim.txt, which contains the simulation output, and one named OUTPUTFILENAME_dis.txt, which contains the disassembled program code for the input MIPS program. 2 Your program will be graded both with the sample input and output provided to you, and with input and output that is not provided to you. It is recommended you construct your own input programs for testing. Output Your program will produce 2 output files. ∙ One named OUTPUTFILENAME_sim.txt, which contains the simulation output ∙ One named OUTPUTFILENAME_dis.txt, which contains the disassembled program code for the input MIPS program. The disassembled output file should contain one line per word in the input file. It should be separated into 4 columns, each separated by tab character. The columns contain the following information: 1. The binary representation of the instruction word. If the word is an instruction (as opposed to memory data after the BREAK instruction), the instruction should be split into seven groups of digits: the valid bit, the opcode bits, four groups of 5 bits, and a final group of 6 bits. 2. The address of the memory location (in decimal) 3. The disassembled instruction opcode 4. If it is an instruction, print the operation, followed by a tab character, then print each argument separated by a comma and a space ( “, “). The simulation file must have the following format: ∙ 20 equal signs and a newline ∙ cycle: [cycle number] [tab] [instruction address] [tab] [instruction string (same as step 4 above)] ∙ [blank lane] ∙ registers: ∙ r00: [tab] [integer value of R00][tab] [integer value of R01][tab] . . . [integer value of R07] ∙ r08: [tab] [integer value of R08][tab] [integer value of R09][tab] . . . [integer value of R15] ∙ r16: [tab] [integer value of R16][tab] [integer value of R17][tab] . . . [integer value of R23] ∙ r24: [tab] [integer value of R24][tab] [integer value of R25][tab] . . . [integer value of R31] ∙ [blank line] 3 ∙ [data address]: [tab] [show 8 data words, with tabs in between] ∙ … [continue until last data word] Hint: Spend some time creating an output function or class or module or whatever fits your language of choice that handles printing out the various data in this format. Spending a little time to do it right in the beginning will save you a lot of time later. Another hint, the register file and memory can simply be implemented as arrays. Go ahead and make them global variables so you don’t have to pass them around to a bunch of functions. This just makes the code a little more simple to write. Instructions and arguments should be in capital letters. All integer values should be in decimal. Immediate values should be preceded by a # sign. Be careful and consider which instructions take signed values and which take unsigned values. Be sure to use the correct format depending on the context. You output will be graded with the diff command. Test your output against the provided sample outputs! Any differences reported by the diff command are assumed to be incorrect output! Sample files will be provided with the following extensions: ∙ .c – C code ∙ .mips – the compiled version of the C code ∙ .bin – the binary version of the .mips file ∙ sample output files ∙ a file named similar to sample_bin.txt which is a text version of the .bin file for your reference. Note that your program must accept the .bin file, not the text version. What to Turn In 1. Your source files, in ZIP or TAR format. 2. A README file in PLAIN TEXT FORMAT that contains the names and email addresses of your group members and instructions for compiling and running your program. The README file should NOT have a file extension (e.g., .txt). 3. A MAKEFILE that will compile your program using the default make target (all:). You may use any programming language you like. If your language of choice is NOT available on the home server, you must demo it to me in my office. The executable must be named “mipssim” once the program is compiled. If your programming language uses an interpreter to execute the program, indicate that in the README file. DO NOT turn in any sample input files or any previously generated output files. 4 Grading A valid attempt at the project that compiles and produces some output is worth 70 points. Each produced disassembly and simulation file is worth 5 points. There are 3 bin files from which 3 disassembly files are produced and 3 simulation files. You will receive 5 points for a produced file if it matches the provided file EXACTLY (with the exception of white space differences). Programs that do not compile or that do not produce any output will get 0 points. If you do not follow the directions in regards to command line arguments or expected behavior, the penalty is at the discretion of the grader. Input and Expected Output The remainder of the document contains PDF versions of the expected output. The input files and text files of the expected output will be available on the course webpage. The final two pages of this document contain programs to convert text files containing binary strings to binary files, and vice versa. Use those programs to develop your own MIPS executables for debugging and testing. Note that your program MUST USE A BINARY FILE AS INPUT. The test1 example contains a text representation of the input file as well as the c version of the program. These are there for informational purposes only. Your program only needs to read the binary file and produce the disassembled file and simulation file. 5 EX_readBinaryFile.cpp Page 1 of 1 #include #include #include #include using namespace std; int main() { char buffer[4]; int i; char * iPtr; iPtr = (char*)(void*) &i; int FD = open(“test2.bin”, O_RDONLY); int amt = 4; while( amt != 0 ) { amt = read(FD, buffer, 4); if( amt == 4) { iPtr[0] = buffer[3]; iPtr[1] = buffer[2]; iPtr[2] = buffer[1]; iPtr[3] = buffer[0]; cout 1 ) fprintf(stderr,”Usage: ./a.out < input_txt > output_bin ”); char word[32]; unsigned char vals[4]; unsigned char w, div, b; unsigned char tot ; w = 0; while( scanf( “%c”, &tot) != EOF ) { div = 128; for( b=0; b= div) { printf( “1” ); tot -= div; } else printf( “0”); div = div/2; } w ++; if ( w == 4 ) { printf(“ ”); w = 0; } } } 9 t2b.c Page 1 of 1 /******************************************************* This program converts a text file containing text strings of 1’s and 0’s to a binary file. for example, the text string: 00000001000000001111111110101010 would result in the following binary sequence (written as hex here) in the output file: 0100FFAA *******************************************************/ // gcc -o t2b t2b.c // Usage ./a.out < inputfile > outputfile #include main( int argc , char **argv ) { if( argc > 1 ) fprintf(stderr,”Usage: ./a.out < input_txt > output_bin ”); char word[32]; unsigned char vals[4]; int w, mul, b; unsigned char tot ; while( scanf(” %s”, word ) > 0 ) { for( w=0; w= 0) A[i] = B[i] – C; else A[i] = B[i] + C; } } 12 test1_dis.txt Page 1 of 1 0 01000 00000 00001 00000 00000 001010 96 Invalid Instruction 1 01000 00000 00001 00000 00000 001010 100 ADDI R1, R0, #10 1 01011 00000 00001 00000 00100 001000 104 SW R1, 264(R0) 0 00010 10000 00000 00000 00000 000000 108 Invalid Instruction 1 00011 00000 00001 00000 00100 001000 112 LW R1, 264(R0) 1 00001 00001 00000 00000 00000 001100 116 BLTZ R1, #48 1 00000 00000 00001 01010 00010 000000 120 SLL R10, R1, #2 1 00011 01010 00011 00000 00010 101100 124 LW R3, 172(R10) 1 00011 01010 00100 00000 00011 011000 128 LW R4, 216(R10) 1 00011 00000 00101 00000 00100 000100 132 LW R5, 260(R0) 1 00001 00011 00000 00000 00000 000010 136 BLTZ R3, #8 1 00000 00100 00101 00110 00000 100010 140 SUB R6, R4, R5 1 00010 00000 00000 00000 00000 100110 144 J #152 1 00000 00100 00101 00110 00000 100000 148 ADD R6, R4, R5 1 01011 01010 00110 00000 00010 101100 152 SW R6, 172(R10) 1 01000 00001 00001 11111 11111 111111 156 ADDI R1, R1, #-1 1 01011 00000 00001 00000 00100 001000 160 SW R1, 264(R0) 1 00010 00000 00000 00000 00000 011100 164 J #112 1 00000 00000 00000 00000 00000 001101 168 BREAK 11111111111111111111111111111111 172 -1 11111111111111111111111111111110 176 -2 11111111111111111111111111111101 180 -3 00000000000000000000000000000001 184 1 00000000000000000000000000000010 188 2 00000000000000000000000000000011 192 3 00000000000000000000000000000000 196 0 00000000000000000000000000000000 200 0 00000000000000000000000000000101 204 5 11111111111111111111111111111011 208 -5 00000000000000000000000000000110 212 6 00000000000000000000000000000000 216 0 00000000000000000000000000000000 220 0 00000000000000000000000000000000 224 0 00000000000000000000000000000000 228 0 00000000000000000000000000000000 232 0 00000000000000000000000000000000 236 0 00000000000000000000000000000000 240 0 00000000000000000000000000000000 244 0 00000000000000000000000000000000 248 0 00000000000000000000000000000000 252 0 00000000000000000000000000000000 256 0 00000000000000000000000000000001 260 1 00000000000000000000000000000000 264 0 13 test1_mips.txt Page 1 of 1 ; Initially PC is set to 100 ; Data section is right after the code section .text 100 .global _main _main: ADDI R1, R0, #10 ; init i SW R1, VAR_i(R0) ; store i FOR_0: LW R1, VAR_i(R0) BLTZ R1, END_FOR_0 ; i >= 0? SLL R10, R1, #2 ; get correct word boundary LW R3, A(R10) ; read A[i] LW R4, B(R10) ; read B[i] LW R5, C(R0) ; read C BLTZ R3, ELSE_0 ; A[i] >= 0 ? SUB R6, R4, R5 ; B[i] – C J TAIL_0 ELSE_0: ADD R6, R4, R5 ; B[i] + C TAIL_0: SW R6, A(R10) ; rewrite A[i] ADDI R1, R1, #-1 ; i– SW R1, VAR_i(R0) J FOR_0 END_FOR_0: BREAK A: .word -1, -2, -3, 1, 2, 3, 0, 0, 5, -5, 6 B: .word 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 C: ; let C be 1 .word 1 VAR_i: ; for var i .word 0 14 test1_sim.txt Page 1 of 32 ==================== cycle:1 100 ADDI R1, R0, #10 registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:2 104 SW R1, 264(R0) registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:3 112 LW R1, 264(R0) registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:4 116 BLTZ R1, #48 registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:5 120 SLL R10, R1, #2 registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 15 test1_sim.txt Page 2 of 32 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:6 124 LW R3, 172(R10) registers: r00: 0 10 0 6 0 0 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:7 128 LW R4, 216(R10) registers: r00: 0 10 0 6 0 0 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:8 132 LW R5, 260(R0) registers: r00: 0 10 0 6 0 1 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:9 136 BLTZ R3, #8 registers: r00: 0 10 0 6 0 1 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:10 140 SUB R6, R4, R5 registers: 16 test1_sim.txt Page 3 of 32 r00: 0 10 0 6 0 1 -1 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:11 144 J #152 registers: r00: 0 10 0 6 0 1 -1 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:12 152 SW R6, 172(R10) registers: r00: 0 10 0 6 0 1 -1 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:13 156 ADDI R1, R1, #-1 registers: r00: 0 9 0 6 0 1 -1 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 10 ==================== cycle:14 160 SW R1, 264(R0) registers: r00: 0 9 0 6 0 1 -1 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 17 test1_sim.txt Page 4 of 32 ==================== cycle:15 164 J #112 registers: r00: 0 9 0 6 0 1 -1 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:16 112 LW R1, 264(R0) registers: r00: 0 9 0 6 0 1 -1 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:17 116 BLTZ R1, #48 registers: r00: 0 9 0 6 0 1 -1 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:18 120 SLL R10, R1, #2 registers: r00: 0 9 0 6 0 1 -1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:19 124 LW R3, 172(R10) registers: r00: 0 9 0 -5 0 1 -1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 18 test1_sim.txt Page 5 of 32 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:20 128 LW R4, 216(R10) registers: r00: 0 9 0 -5 0 1 -1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:21 132 LW R5, 260(R0) registers: r00: 0 9 0 -5 0 1 -1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:22 136 BLTZ R3, #8 registers: r00: 0 9 0 -5 0 1 -1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:23 148 ADD R6, R4, R5 registers: r00: 0 9 0 -5 0 1 1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 -5 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:24 152 SW R6, 172(R10) 19 test1_sim.txt Page 6 of 32 registers: r00: 0 9 0 -5 0 1 1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:25 156 ADDI R1, R1, #-1 registers: r00: 0 8 0 -5 0 1 1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 9 ==================== cycle:26 160 SW R1, 264(R0) registers: r00: 0 8 0 -5 0 1 1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:27 164 J #112 registers: r00: 0 8 0 -5 0 1 1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:28 112 LW R1, 264(R0) registers: r00: 0 8 0 -5 0 1 1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 20 test1_sim.txt Page 7 of 32 236: 0 0 0 0 0 0 1 8 ==================== cycle:29 116 BLTZ R1, #48 registers: r00: 0 8 0 -5 0 1 1 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:30 120 SLL R10, R1, #2 registers: r00: 0 8 0 -5 0 1 1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:31 124 LW R3, 172(R10) registers: r00: 0 8 0 5 0 1 1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:32 128 LW R4, 216(R10) registers: r00: 0 8 0 5 0 1 1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:33 132 LW R5, 260(R0) registers: r00: 0 8 0 5 0 1 1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 21 test1_sim.txt Page 8 of 32 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:34 136 BLTZ R3, #8 registers: r00: 0 8 0 5 0 1 1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:35 140 SUB R6, R4, R5 registers: r00: 0 8 0 5 0 1 -1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:36 144 J #152 registers: r00: 0 8 0 5 0 1 -1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: 5 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:37 152 SW R6, 172(R10) registers: r00: 0 8 0 5 0 1 -1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:38 156 ADDI R1, R1, #-1 22 test1_sim.txt Page 9 of 32 registers: r00: 0 7 0 5 0 1 -1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 8 ==================== cycle:39 160 SW R1, 264(R0) registers: r00: 0 7 0 5 0 1 -1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:40 164 J #112 registers: r00: 0 7 0 5 0 1 -1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:41 112 LW R1, 264(R0) registers: r00: 0 7 0 5 0 1 -1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:42 116 BLTZ R1, #48 registers: r00: 0 7 0 5 0 1 -1 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 23 test1_sim.txt Page 10 of 32 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:43 120 SLL R10, R1, #2 registers: r00: 0 7 0 5 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:44 124 LW R3, 172(R10) registers: r00: 0 7 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:45 128 LW R4, 216(R10) registers: r00: 0 7 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:46 132 LW R5, 260(R0) registers: r00: 0 7 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:47 136 BLTZ R3, #8 registers: r00: 0 7 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 24 test1_sim.txt Page 11 of 32 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:48 140 SUB R6, R4, R5 registers: r00: 0 7 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:49 144 J #152 registers: r00: 0 7 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 0 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:50 152 SW R6, 172(R10) registers: r00: 0 7 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== cycle:51 156 ADDI R1, R1, #-1 registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 7 ==================== 25 test1_sim.txt Page 12 of 32 cycle:52 160 SW R1, 264(R0) registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:53 164 J #112 registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:54 112 LW R1, 264(R0) registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:55 116 BLTZ R1, #48 registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:56 120 SLL R10, R1, #2 registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 26 test1_sim.txt Page 13 of 32 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:57 124 LW R3, 172(R10) registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:58 128 LW R4, 216(R10) registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:59 132 LW R5, 260(R0) registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:60 136 BLTZ R3, #8 registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:61 140 SUB R6, R4, R5 registers: r00: 0 6 0 0 0 1 -1 0 27 test1_sim.txt Page 14 of 32 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:62 144 J #152 registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 0 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:63 152 SW R6, 172(R10) registers: r00: 0 6 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:64 156 ADDI R1, R1, #-1 registers: r00: 0 5 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 6 ==================== cycle:65 160 SW R1, 264(R0) registers: r00: 0 5 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 28 test1_sim.txt Page 15 of 32 ==================== cycle:66 164 J #112 registers: r00: 0 5 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:67 112 LW R1, 264(R0) registers: r00: 0 5 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:68 116 BLTZ R1, #48 registers: r00: 0 5 0 0 0 1 -1 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:69 120 SLL R10, R1, #2 registers: r00: 0 5 0 0 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:70 124 LW R3, 172(R10) registers: r00: 0 5 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 29 test1_sim.txt Page 16 of 32 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:71 128 LW R4, 216(R10) registers: r00: 0 5 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:72 132 LW R5, 260(R0) registers: r00: 0 5 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:73 136 BLTZ R3, #8 registers: r00: 0 5 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:74 140 SUB R6, R4, R5 registers: r00: 0 5 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:75 144 J #152 registers: 30 test1_sim.txt Page 17 of 32 r00: 0 5 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 3 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:76 152 SW R6, 172(R10) registers: r00: 0 5 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:77 156 ADDI R1, R1, #-1 registers: r00: 0 4 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 5 ==================== cycle:78 160 SW R1, 264(R0) registers: r00: 0 4 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:79 164 J #112 registers: r00: 0 4 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 31 test1_sim.txt Page 18 of 32 ==================== cycle:80 112 LW R1, 264(R0) registers: r00: 0 4 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:81 116 BLTZ R1, #48 registers: r00: 0 4 0 3 0 1 -1 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:82 120 SLL R10, R1, #2 registers: r00: 0 4 0 3 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:83 124 LW R3, 172(R10) registers: r00: 0 4 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:84 128 LW R4, 216(R10) registers: r00: 0 4 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 32 test1_sim.txt Page 19 of 32 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:85 132 LW R5, 260(R0) registers: r00: 0 4 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:86 136 BLTZ R3, #8 registers: r00: 0 4 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:87 140 SUB R6, R4, R5 registers: r00: 0 4 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:88 144 J #152 registers: r00: 0 4 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 2 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:89 152 SW R6, 172(R10) 33 test1_sim.txt Page 20 of 32 registers: r00: 0 4 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:90 156 ADDI R1, R1, #-1 registers: r00: 0 3 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 4 ==================== cycle:91 160 SW R1, 264(R0) registers: r00: 0 3 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:92 164 J #112 registers: r00: 0 3 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:93 112 LW R1, 264(R0) registers: r00: 0 3 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 34 test1_sim.txt Page 21 of 32 236: 0 0 0 0 0 0 1 3 ==================== cycle:94 116 BLTZ R1, #48 registers: r00: 0 3 0 2 0 1 -1 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:95 120 SLL R10, R1, #2 registers: r00: 0 3 0 2 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:96 124 LW R3, 172(R10) registers: r00: 0 3 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:97 128 LW R4, 216(R10) registers: r00: 0 3 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:98 132 LW R5, 260(R0) registers: r00: 0 3 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 35 test1_sim.txt Page 22 of 32 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:99 136 BLTZ R3, #8 registers: r00: 0 3 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:100 140 SUB R6, R4, R5 registers: r00: 0 3 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:101 144 J #152 registers: r00: 0 3 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:102 152 SW R6, 172(R10) registers: r00: 0 3 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:103 156 ADDI R1, R1, #-1 36 test1_sim.txt Page 23 of 32 registers: r00: 0 2 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 3 ==================== cycle:104 160 SW R1, 264(R0) registers: r00: 0 2 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:105 164 J #112 registers: r00: 0 2 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:106 112 LW R1, 264(R0) registers: r00: 0 2 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:107 116 BLTZ R1, #48 registers: r00: 0 2 0 1 0 1 -1 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 37 test1_sim.txt Page 24 of 32 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:108 120 SLL R10, R1, #2 registers: r00: 0 2 0 1 0 1 -1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:109 124 LW R3, 172(R10) registers: r00: 0 2 0 -3 0 1 -1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:110 128 LW R4, 216(R10) registers: r00: 0 2 0 -3 0 1 -1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:111 132 LW R5, 260(R0) registers: r00: 0 2 0 -3 0 1 -1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:112 136 BLTZ R3, #8 registers: r00: 0 2 0 -3 0 1 -1 0 r08: 0 0 8 0 0 0 0 0 38 test1_sim.txt Page 25 of 32 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:113 148 ADD R6, R4, R5 registers: r00: 0 2 0 -3 0 1 1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 -3 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:114 152 SW R6, 172(R10) registers: r00: 0 2 0 -3 0 1 1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:115 156 ADDI R1, R1, #-1 registers: r00: 0 1 0 -3 0 1 1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 2 ==================== cycle:116 160 SW R1, 264(R0) registers: r00: 0 1 0 -3 0 1 1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== 39 test1_sim.txt Page 26 of 32 cycle:117 164 J #112 registers: r00: 0 1 0 -3 0 1 1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:118 112 LW R1, 264(R0) registers: r00: 0 1 0 -3 0 1 1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:119 116 BLTZ R1, #48 registers: r00: 0 1 0 -3 0 1 1 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:120 120 SLL R10, R1, #2 registers: r00: 0 1 0 -3 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:121 124 LW R3, 172(R10) registers: r00: 0 1 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 40 test1_sim.txt Page 27 of 32 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:122 128 LW R4, 216(R10) registers: r00: 0 1 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:123 132 LW R5, 260(R0) registers: r00: 0 1 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:124 136 BLTZ R3, #8 registers: r00: 0 1 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:125 148 ADD R6, R4, R5 registers: r00: 0 1 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 -2 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:126 152 SW R6, 172(R10) registers: r00: 0 1 0 -2 0 1 1 0 41 test1_sim.txt Page 28 of 32 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:127 156 ADDI R1, R1, #-1 registers: r00: 0 0 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 1 ==================== cycle:128 160 SW R1, 264(R0) registers: r00: 0 0 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:129 164 J #112 registers: r00: 0 0 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:130 112 LW R1, 264(R0) registers: r00: 0 0 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 42 test1_sim.txt Page 29 of 32 ==================== cycle:131 116 BLTZ R1, #48 registers: r00: 0 0 0 -2 0 1 1 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:132 120 SLL R10, R1, #2 registers: r00: 0 0 0 -2 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:133 124 LW R3, 172(R10) registers: r00: 0 0 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:134 128 LW R4, 216(R10) registers: r00: 0 0 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:135 132 LW R5, 260(R0) registers: r00: 0 0 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 43 test1_sim.txt Page 30 of 32 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:136 136 BLTZ R3, #8 registers: r00: 0 0 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:137 148 ADD R6, R4, R5 registers: r00: 0 0 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:138 152 SW R6, 172(R10) registers: r00: 0 0 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:139 156 ADDI R1, R1, #-1 registers: r00: 0 -1 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 0 ==================== cycle:140 160 SW R1, 264(R0) registers: 44 test1_sim.txt Page 31 of 32 r00: 0 -1 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 -1 ==================== cycle:141 164 J #112 registers: r00: 0 -1 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 -1 ==================== cycle:142 112 LW R1, 264(R0) registers: r00: 0 -1 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 -1 ==================== cycle:143 116 BLTZ R1, #48 registers: r00: 0 -1 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 -1 ==================== cycle:144 168 BREAK registers: r00: 0 -1 0 -1 0 1 1 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 1 1 -1 -1 -1 -1 -1 204: -1 1 -1 0 0 0 0 0 236: 0 0 0 0 0 0 1 -1 45 test1_sim.txt Page 32 of 32 46 test2_dis.txt Page 1 of 1 1 00010 00000 00000 00000 00000 011010 96 J #104 0 01000 00000 00010 00000 00000 000010 100 Invalid Instruction 1 01000 00000 00001 00000 00001 100100 104 ADDI R1, R0, #100 1 01000 00000 00010 00000 00000 011000 108 ADDI R2, R0, #24 1 00000 00001 00010 00011 00000 100000 112 ADD R3, R1, R2 1 00000 00011 00000 00000 00000 001000 116 JR R3 0 01000 00000 10000 00000 00000 000001 120 Invalid Instruction 1 00000 00001 00010 00100 00000 100010 124 SUB R4, R1, R2 1 00000 00000 00010 00101 00001 000000 128 SLL R5, R2, #1 1 00000 00000 00101 00110 00001 000010 132 SRL R6, R5, #1 1 11100 00010 00110 00111 00000 000010 136 MUL R7, R2, R6 1 01000 00000 01000 00000 00000 000000 140 ADDI R8, R0, #0 1 00000 00100 01000 01001 00000 001010 144 MOVZ R9, R4, R8 1 00000 00000 00000 00000 00000 000000 148 NOP 1 00000 00000 00000 00000 00000 001101 152 BREAK 00000000000000000000000000000001 156 1 00000000000000000000000000000010 160 2 00000000000000000000000000000011 164 3 00000000000000000000000000000100 168 4 47 test2_sim.txt Page 1 of 3 ==================== cycle:1 96 J #104 registers: r00: 0 0 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:2 104 ADDI R1, R0, #100 registers: r00: 0 100 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:3 108 ADDI R2, R0, #24 registers: r00: 0 100 24 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:4 112 ADD R3, R1, R2 registers: r00: 0 100 24 124 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:5 116 JR R3 registers: r00: 0 100 24 124 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:6 124 SUB R4, R1, R2 registers: r00: 0 100 24 124 76 0 0 0 48 test2_sim.txt Page 2 of 3 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:7 128 SLL R5, R2, #1 registers: r00: 0 100 24 124 76 48 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:8 132 SRL R6, R5, #1 registers: r00: 0 100 24 124 76 48 24 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:9 136 MUL R7, R2, R6 registers: r00: 0 100 24 124 76 48 24 576 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:10 140 ADDI R8, R0, #0 registers: r00: 0 100 24 124 76 48 24 576 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:11 144 MOVZ R9, R4, R8 registers: r00: 0 100 24 124 76 48 24 576 r08: 0 76 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 49 test2_sim.txt Page 3 of 3 156: 1 2 3 4 ==================== cycle:12 148 NOP registers: r00: 0 100 24 124 76 48 24 576 r08: 0 76 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 ==================== cycle:13 152 BREAK registers: r00: 0 100 24 124 76 48 24 576 r08: 0 76 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 156: 1 2 3 4 50 test3_dis.txt Page 1 of 1 0 01000 00000 00001 00000 00000 001010 96 Invalid Instruction 1 01000 00000 00001 00000 00000 001010 100 ADDI R1, R0, #10 1 01011 00000 00001 00000 00100 001000 104 SW R1, 264(R0) 0 00010 10000 00000 00000 00000 000000 108 Invalid Instruction 1 00011 00000 00001 00000 00100 001000 112 LW R1, 264(R0) 1 00001 00001 00000 00000 00000 001100 116 BLTZ R1, #48 1 00000 00000 00001 01010 00010 000000 120 SLL R10, R1, #2 1 00011 01010 00011 00000 00010 101100 124 LW R3, 172(R10) 1 00011 01010 00100 00000 00011 011000 128 LW R4, 216(R10) 1 00011 00000 00101 00000 00100 000100 132 LW R5, 260(R0) 1 00001 00011 00000 00000 00000 000010 136 BLTZ R3, #8 1 00000 00100 00101 00110 00000 100010 140 SUB R6, R4, R5 1 00010 00000 00000 00000 00000 100110 144 J #152 1 00000 00100 00101 00110 00000 100000 148 ADD R6, R4, R5 1 01011 01010 00110 00000 00010 101100 152 SW R6, 172(R10) 1 01000 00001 00001 11111 11111 111111 156 ADDI R1, R1, #-1 1 01011 00000 00001 00000 00100 001000 160 SW R1, 264(R0) 1 00010 00000 00000 00000 00000 011100 164 J #112 1 00000 00000 00000 00000 00000 001101 168 BREAK 00000000000000000000000000000001 172 1 00000000000000000000000000000011 176 3 11111111111111111111111111111101 180 -3 11111111111111111111111111111111 184 -1 11111111111111111111111111111110 188 -2 00000000000000000000000000000011 192 3 00000000000000000000000000000000 196 0 00000000000000000000000000000000 200 0 00000000000000000000000000000101 204 5 11111111111111111111111111111011 208 -5 00000000000000000000000000000110 212 6 00000000000000000000000000000000 216 0 00000000000000000000000000000000 220 0 00000000000000000000000000000000 224 0 00000000000000000000000000000000 228 0 00000000000000000000000000000000 232 0 00000000000000000000000000000000 236 0 00000000000000000000000000000000 240 0 00000000000000000000000000000000 244 0 00000000000000000000000000000000 248 0 00000000000000000000000000000000 252 0 00000000000000000000000000000000 256 0 00000000000000000000000000000010 260 2 00000000000000000000000000000000 264 0 51 test3_sim.txt Page 1 of 32 ==================== cycle:1 100 ADDI R1, R0, #10 registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:2 104 SW R1, 264(R0) registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:3 112 LW R1, 264(R0) registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:4 116 BLTZ R1, #48 registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:5 120 SLL R10, R1, #2 registers: r00: 0 10 0 0 0 0 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 52 test3_sim.txt Page 2 of 32 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:6 124 LW R3, 172(R10) registers: r00: 0 10 0 6 0 0 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:7 128 LW R4, 216(R10) registers: r00: 0 10 0 6 0 0 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:8 132 LW R5, 260(R0) registers: r00: 0 10 0 6 0 2 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:9 136 BLTZ R3, #8 registers: r00: 0 10 0 6 0 2 0 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:10 140 SUB R6, R4, R5 registers: 53 test3_sim.txt Page 3 of 32 r00: 0 10 0 6 0 2 -2 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:11 144 J #152 registers: r00: 0 10 0 6 0 2 -2 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 6 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:12 152 SW R6, 172(R10) registers: r00: 0 10 0 6 0 2 -2 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:13 156 ADDI R1, R1, #-1 registers: r00: 0 9 0 6 0 2 -2 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 10 ==================== cycle:14 160 SW R1, 264(R0) registers: r00: 0 9 0 6 0 2 -2 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 54 test3_sim.txt Page 4 of 32 ==================== cycle:15 164 J #112 registers: r00: 0 9 0 6 0 2 -2 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:16 112 LW R1, 264(R0) registers: r00: 0 9 0 6 0 2 -2 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:17 116 BLTZ R1, #48 registers: r00: 0 9 0 6 0 2 -2 0 r08: 0 0 40 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:18 120 SLL R10, R1, #2 registers: r00: 0 9 0 6 0 2 -2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:19 124 LW R3, 172(R10) registers: r00: 0 9 0 -5 0 2 -2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 55 test3_sim.txt Page 5 of 32 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:20 128 LW R4, 216(R10) registers: r00: 0 9 0 -5 0 2 -2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:21 132 LW R5, 260(R0) registers: r00: 0 9 0 -5 0 2 -2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:22 136 BLTZ R3, #8 registers: r00: 0 9 0 -5 0 2 -2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:23 148 ADD R6, R4, R5 registers: r00: 0 9 0 -5 0 2 2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 -5 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:24 152 SW R6, 172(R10) 56 test3_sim.txt Page 6 of 32 registers: r00: 0 9 0 -5 0 2 2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:25 156 ADDI R1, R1, #-1 registers: r00: 0 8 0 -5 0 2 2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 9 ==================== cycle:26 160 SW R1, 264(R0) registers: r00: 0 8 0 -5 0 2 2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:27 164 J #112 registers: r00: 0 8 0 -5 0 2 2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:28 112 LW R1, 264(R0) registers: r00: 0 8 0 -5 0 2 2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 57 test3_sim.txt Page 7 of 32 236: 0 0 0 0 0 0 2 8 ==================== cycle:29 116 BLTZ R1, #48 registers: r00: 0 8 0 -5 0 2 2 0 r08: 0 0 36 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:30 120 SLL R10, R1, #2 registers: r00: 0 8 0 -5 0 2 2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:31 124 LW R3, 172(R10) registers: r00: 0 8 0 5 0 2 2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:32 128 LW R4, 216(R10) registers: r00: 0 8 0 5 0 2 2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:33 132 LW R5, 260(R0) registers: r00: 0 8 0 5 0 2 2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 58 test3_sim.txt Page 8 of 32 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:34 136 BLTZ R3, #8 registers: r00: 0 8 0 5 0 2 2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:35 140 SUB R6, R4, R5 registers: r00: 0 8 0 5 0 2 -2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:36 144 J #152 registers: r00: 0 8 0 5 0 2 -2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: 5 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:37 152 SW R6, 172(R10) registers: r00: 0 8 0 5 0 2 -2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:38 156 ADDI R1, R1, #-1 59 test3_sim.txt Page 9 of 32 registers: r00: 0 7 0 5 0 2 -2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 8 ==================== cycle:39 160 SW R1, 264(R0) registers: r00: 0 7 0 5 0 2 -2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:40 164 J #112 registers: r00: 0 7 0 5 0 2 -2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:41 112 LW R1, 264(R0) registers: r00: 0 7 0 5 0 2 -2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:42 116 BLTZ R1, #48 registers: r00: 0 7 0 5 0 2 -2 0 r08: 0 0 32 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 60 test3_sim.txt Page 10 of 32 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:43 120 SLL R10, R1, #2 registers: r00: 0 7 0 5 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:44 124 LW R3, 172(R10) registers: r00: 0 7 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:45 128 LW R4, 216(R10) registers: r00: 0 7 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:46 132 LW R5, 260(R0) registers: r00: 0 7 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:47 136 BLTZ R3, #8 registers: r00: 0 7 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 61 test3_sim.txt Page 11 of 32 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:48 140 SUB R6, R4, R5 registers: r00: 0 7 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:49 144 J #152 registers: r00: 0 7 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 0 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:50 152 SW R6, 172(R10) registers: r00: 0 7 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== cycle:51 156 ADDI R1, R1, #-1 registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 7 ==================== 62 test3_sim.txt Page 12 of 32 cycle:52 160 SW R1, 264(R0) registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:53 164 J #112 registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:54 112 LW R1, 264(R0) registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:55 116 BLTZ R1, #48 registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 28 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:56 120 SLL R10, R1, #2 registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 63 test3_sim.txt Page 13 of 32 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:57 124 LW R3, 172(R10) registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:58 128 LW R4, 216(R10) registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:59 132 LW R5, 260(R0) registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:60 136 BLTZ R3, #8 registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:61 140 SUB R6, R4, R5 registers: r00: 0 6 0 0 0 2 -2 0 64 test3_sim.txt Page 14 of 32 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:62 144 J #152 registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 0 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:63 152 SW R6, 172(R10) registers: r00: 0 6 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:64 156 ADDI R1, R1, #-1 registers: r00: 0 5 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 6 ==================== cycle:65 160 SW R1, 264(R0) registers: r00: 0 5 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 65 test3_sim.txt Page 15 of 32 ==================== cycle:66 164 J #112 registers: r00: 0 5 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:67 112 LW R1, 264(R0) registers: r00: 0 5 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:68 116 BLTZ R1, #48 registers: r00: 0 5 0 0 0 2 -2 0 r08: 0 0 24 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:69 120 SLL R10, R1, #2 registers: r00: 0 5 0 0 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:70 124 LW R3, 172(R10) registers: r00: 0 5 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 66 test3_sim.txt Page 16 of 32 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:71 128 LW R4, 216(R10) registers: r00: 0 5 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:72 132 LW R5, 260(R0) registers: r00: 0 5 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:73 136 BLTZ R3, #8 registers: r00: 0 5 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:74 140 SUB R6, R4, R5 registers: r00: 0 5 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:75 144 J #152 registers: 67 test3_sim.txt Page 17 of 32 r00: 0 5 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 3 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:76 152 SW R6, 172(R10) registers: r00: 0 5 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:77 156 ADDI R1, R1, #-1 registers: r00: 0 4 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 5 ==================== cycle:78 160 SW R1, 264(R0) registers: r00: 0 4 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:79 164 J #112 registers: r00: 0 4 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 68 test3_sim.txt Page 18 of 32 ==================== cycle:80 112 LW R1, 264(R0) registers: r00: 0 4 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:81 116 BLTZ R1, #48 registers: r00: 0 4 0 3 0 2 -2 0 r08: 0 0 20 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:82 120 SLL R10, R1, #2 registers: r00: 0 4 0 3 0 2 -2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:83 124 LW R3, 172(R10) registers: r00: 0 4 0 -2 0 2 -2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:84 128 LW R4, 216(R10) registers: r00: 0 4 0 -2 0 2 -2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 69 test3_sim.txt Page 19 of 32 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:85 132 LW R5, 260(R0) registers: r00: 0 4 0 -2 0 2 -2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:86 136 BLTZ R3, #8 registers: r00: 0 4 0 -2 0 2 -2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:87 148 ADD R6, R4, R5 registers: r00: 0 4 0 -2 0 2 2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 -2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:88 152 SW R6, 172(R10) registers: r00: 0 4 0 -2 0 2 2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:89 156 ADDI R1, R1, #-1 70 test3_sim.txt Page 20 of 32 registers: r00: 0 3 0 -2 0 2 2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 4 ==================== cycle:90 160 SW R1, 264(R0) registers: r00: 0 3 0 -2 0 2 2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:91 164 J #112 registers: r00: 0 3 0 -2 0 2 2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:92 112 LW R1, 264(R0) registers: r00: 0 3 0 -2 0 2 2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:93 116 BLTZ R1, #48 registers: r00: 0 3 0 -2 0 2 2 0 r08: 0 0 16 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 71 test3_sim.txt Page 21 of 32 236: 0 0 0 0 0 0 2 3 ==================== cycle:94 120 SLL R10, R1, #2 registers: r00: 0 3 0 -2 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:95 124 LW R3, 172(R10) registers: r00: 0 3 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:96 128 LW R4, 216(R10) registers: r00: 0 3 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:97 132 LW R5, 260(R0) registers: r00: 0 3 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:98 136 BLTZ R3, #8 registers: r00: 0 3 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 72 test3_sim.txt Page 22 of 32 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:99 148 ADD R6, R4, R5 registers: r00: 0 3 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 -1 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:100 152 SW R6, 172(R10) registers: r00: 0 3 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:101 156 ADDI R1, R1, #-1 registers: r00: 0 2 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 3 ==================== cycle:102 160 SW R1, 264(R0) registers: r00: 0 2 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:103 164 J #112 73 test3_sim.txt Page 23 of 32 registers: r00: 0 2 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:104 112 LW R1, 264(R0) registers: r00: 0 2 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:105 116 BLTZ R1, #48 registers: r00: 0 2 0 -1 0 2 2 0 r08: 0 0 12 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:106 120 SLL R10, R1, #2 registers: r00: 0 2 0 -1 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:107 124 LW R3, 172(R10) registers: r00: 0 2 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 74 test3_sim.txt Page 24 of 32 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:108 128 LW R4, 216(R10) registers: r00: 0 2 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:109 132 LW R5, 260(R0) registers: r00: 0 2 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:110 136 BLTZ R3, #8 registers: r00: 0 2 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:111 148 ADD R6, R4, R5 registers: r00: 0 2 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 -3 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:112 152 SW R6, 172(R10) registers: r00: 0 2 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 75 test3_sim.txt Page 25 of 32 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:113 156 ADDI R1, R1, #-1 registers: r00: 0 1 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 2 ==================== cycle:114 160 SW R1, 264(R0) registers: r00: 0 1 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:115 164 J #112 registers: r00: 0 1 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:116 112 LW R1, 264(R0) registers: r00: 0 1 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== 76 test3_sim.txt Page 26 of 32 cycle:117 116 BLTZ R1, #48 registers: r00: 0 1 0 -3 0 2 2 0 r08: 0 0 8 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:118 120 SLL R10, R1, #2 registers: r00: 0 1 0 -3 0 2 2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:119 124 LW R3, 172(R10) registers: r00: 0 1 0 3 0 2 2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:120 128 LW R4, 216(R10) registers: r00: 0 1 0 3 0 2 2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:121 132 LW R5, 260(R0) registers: r00: 0 1 0 3 0 2 2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 77 test3_sim.txt Page 27 of 32 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:122 136 BLTZ R3, #8 registers: r00: 0 1 0 3 0 2 2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:123 140 SUB R6, R4, R5 registers: r00: 0 1 0 3 0 2 -2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:124 144 J #152 registers: r00: 0 1 0 3 0 2 -2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 3 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:125 152 SW R6, 172(R10) registers: r00: 0 1 0 3 0 2 -2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:126 156 ADDI R1, R1, #-1 registers: r00: 0 0 0 3 0 2 -2 0 78 test3_sim.txt Page 28 of 32 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 1 ==================== cycle:127 160 SW R1, 264(R0) registers: r00: 0 0 0 3 0 2 -2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:128 164 J #112 registers: r00: 0 0 0 3 0 2 -2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:129 112 LW R1, 264(R0) registers: r00: 0 0 0 3 0 2 -2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:130 116 BLTZ R1, #48 registers: r00: 0 0 0 3 0 2 -2 0 r08: 0 0 4 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 79 test3_sim.txt Page 29 of 32 ==================== cycle:131 120 SLL R10, R1, #2 registers: r00: 0 0 0 3 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:132 124 LW R3, 172(R10) registers: r00: 0 0 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:133 128 LW R4, 216(R10) registers: r00: 0 0 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:134 132 LW R5, 260(R0) registers: r00: 0 0 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:135 136 BLTZ R3, #8 registers: r00: 0 0 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 80 test3_sim.txt Page 30 of 32 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:136 140 SUB R6, R4, R5 registers: r00: 0 0 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:137 144 J #152 registers: r00: 0 0 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: 1 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:138 152 SW R6, 172(R10) registers: r00: 0 0 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -2 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:139 156 ADDI R1, R1, #-1 registers: r00: 0 -1 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -2 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 0 ==================== cycle:140 160 SW R1, 264(R0) registers: 81 test3_sim.txt Page 31 of 32 r00: 0 -1 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -2 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 -1 ==================== cycle:141 164 J #112 registers: r00: 0 -1 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -2 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 -1 ==================== cycle:142 112 LW R1, 264(R0) registers: r00: 0 -1 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -2 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 -1 ==================== cycle:143 116 BLTZ R1, #48 registers: r00: 0 -1 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -2 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 -1 ==================== cycle:144 168 BREAK registers: r00: 0 -1 0 1 0 2 -2 0 r08: 0 0 0 0 0 0 0 0 r16: 0 0 0 0 0 0 0 0 r24: 0 0 0 0 0 0 0 0 data: 172: -2 -2 2 2 2 -2 -2 -2 204: -2 2 -2 0 0 0 0 0 236: 0 0 0 0 0 0 2 -1 82 test3_sim.txt Page 32 of 32 83
Overview The purpose of this assignment is to gain familiarity with MIPS assembly language programming. For this assignment, you will create a MIPS implementation of a bubble sort algorithm. The program must run in the MIPS simulator and print out the sorted array. Simulator We will use the MARS simulator found here: https://courses.missouristate.edu/KenVollmar/MARS/download.htm Other MIPS simulators (notably SPIM and its Javascript implementation JSPIM) out there. However, our assignemnt requires random number generation which is provided by the MARS simulator, and not the others. MARS is also a bit more intuitive. You solution must run in the MARS simulator with no errors and it must print out the sorted numbers. To invoke the simulator on a linux machine, use the flag to execute a JAR file: java -jar Mars4_5.jar Algorithm You may implement any sort routine you like. Bubble sort is the simplest to implement, so I suggest that one. A C implementaion of bubble sort is as follows: Create MIPS assembly code for the following C function. It is an insertion sort program. #include #include int main(){ int array [5000]; for( int i = 0; i < 5000; i++ ) array[i] = rand()%100; for(int i = 0; i < 4999; i++) for( int j =0; j < 4999-i; j++ ) if( array[j+1] < array[j] ){ 1 int tmp = array[j]; array[j] = array[j+1]; array[j+1] = tmp; } for( int i = 0; i < 5000; i++ ) printf( “%i ”, array[i]); } MIPS Implementation Notes I am providing a basic file to get you started. You should be able to paste the file into the MARS simulator and run it. a couple of notes: syscall Printing output and generating random numbers are donw via system calls. To execute a system call, you must put the correct arguments into the $v0, $a0, and $a1 registers. once those are in the correct place, you simply call syscall. The system call basically just calls a function, and chooses which function to execute (print, get a random number, etc) via the arguments. The full list of system calls and their arguments are found here: https://courses.missouristate.edu/kenvollmar/mars/help/syscallhelp.html We will use the following system calls: • code 1 to print integers (put the integer to print in $a0) • code 4 to print a string (a newline character in our case) • code 42 to get a random number The following code shows how to get a random number, then print it li $a0, 0 # set up random number system call. # use generator 0 li $a1, 100 # max random number is 100 li $v0, 42 # syscall 42 is random number in a range. # Result goes in $a0 syscall li $v0, 1 # set up to print int with syscall number 1 syscall Allocating Arrays To allocate an array, we will simply create space on the stack and use that. Remember that the stack starts at a high address, and grows downwards towards lower addresses. Therefore, to reserve space for 5 integers on the stack, you simply subtact 20 (4 bytes per integer times 5 integers) from the stack pointer $sp. you may then use that space to load and store numbers to. Remember, it is up to you if you want element 0 of your array to be at the higher address of the space, or the lower address! 2 Labels To create functions, or points in the code for you to use a branch or jump targets, you must create a label. A label is simply some text followed by a colon. The following block demonstrates. beq $t1, $t0, TARGET addi $t1, $t1, 5 TARGET: addi $t0, $t0, 2 To make a function, create a label where the function starts, and then jump and link to that function: # the function FUNC1: addi $t0, $a1, 10 jr $ra #call the function jal FUNC1 Assignment You must create a MIPS program that runs in the MARS simulator. The program must do a sort (bubble sort is probably the easiest) and it must sort an array of 30 integers. The array must be allocated ON THE STACK, as shown above. The program must print the sorted numbers at the end. Basic Example Program The following is a basic example program that allocates an array of 2 integers on the stack, puts random numbers in it, and prints it out. .data NEWLINE: .asciiz “ ” .text .globl main #required main: #required addi $sp, $sp, -8 # make space for 2 ints, 8 bytes on the stack li $a0, 0 #set up random number system call. use generator 0 li $a1, 100 # max random number is 100 li $v0, 42 # syscall 42 is random number in a range syscall # random number is now in $a0 3 sw $a0, 0($sp) # store the number in array location [0] syscall # random number is now in $a0 sw $a0, 4($sp) # store the number in array location [1] # NOTE I am making the array atart at a lower address # and go UP in memory! #now print out the numbers li $v0, 1 #set up to print int lw $a0, 0( $sp ) #get the int to print syscall li $v0, 4 #set up to print a string la $a0, NEWLINE #la is LOAD ADDRESS. newline is declared #at the start of the program syscall li $v0, 1 #set up to print int lw $a0, 4( $sp ) #get the int to print syscall li $v0, 4 #set up to print a string la $a0, NEWLINE #la is LOAD ADDRESS. newline is declared #at the start of the program syscall li $v0, 10 # last 2 lines are required to make program exit syscall
Now that we have our full-fledged Characters and a game world, let’s implement some game functionality. In this project you will modify the classes youhave implemented thus far to allow Characters to eat Tainted Stew. Eating tainted stew will affect each Character subclass differently. Have your Barbarians, Mages, Rangers and Scoundrels feast in the Tavern by eating Tainted Stew and see what happens to them!!! The link to accept the GitHub Classroom assignment can be found on Blackboard This project consists of two parts: You will modify the Character class and its subclasses to enable them to eat Tainted Stew and display their current status. You will modify the Tavern to store Character subclasses polymorphically and access each subclass-specific tainted stew actions. Some additional resources Abstract Classes and Polymorphism: Pure Virtual and Abstract Classes Early vs Late binding Reading from input file: ifstream stringstream getline String manipulation: C++ Substring of a given string C++ find position of a string inside another string Dynamic memory allocation: new and delete key words Implementation Work through the tasks sequentially (implement and test). Only move on to a task when you are positive that the previous one has been completedcorrectly. Remember that the names of classes and methods must exactly match those in this specification (FUNCTION NAMES, PARAMETERTYPES, RETURNS, PRE AND POST CONDITIONS MUST MATCH EXACTLY). Remember, you must thoroughly document your code!!! Task 1: Modify the Character class and its Subclasses Modify the Character class as follows: Make the display() function pure virtual Add the following public member function and also make it pure virtual Modify the Barbarian class as follows: Add the following public member functions /** @post: Modifies the character’s private member variables (the exact modifications will be subclass specific) */ eatTaintedStew /** @post : displays Barbarian data in the form: “[NAME] is a Level [LEVEL] [RACE] BARBARIAN. Vitality: [VITALITY] Armor: [ARMOR] They are [an enemy/not an enemy]. Modify the Mage class as follows: Add the following public member functions Main Weapon: [MAINWEAPON] Offhand Weapon: [OFFHANDWEAPON] Enraged: [TRUE/FALSE] ” Example: BONK is a Level 5 HUMAN BARBARIAN. Vitality: 11 Armor: 5 They are an enemy. Main Weapon: MACE Offhand Weapon: ANOTHERMACE Enraged: TRUE */ display /** @post: If the character is UNDEAD, gain 3 Vitality points. Nothing else happens. If the character is NOT UNDEAD, Vitality is set to 1. In addition, as a Barbarian: Become enraged if the character was not enraged, and not enraged if they were already enraged. If they have now become enraged, the offhand weapon is replaced with “TABLE”. If they are now not enraged, the main weapon is replaced with “BUCKET”. */ eatTaintedStew /** @post : displays Mage data in the form: “[NAME] is a Level [LEVEL] [RACE] MAGE. Vitality: [VITALITY] Armor: [ARMOR] They are [an enemy/not an enemy]. School of Magic: [SCHOOL] Weapon: [WEAPON] They [can/cannot] summon an Incarnate. ” Example: SPYNACH is a Level 4 ELF MAGE. Vitality: 6 Armor: 4 They are not an enemy. School of Magic: ILLUSION Weapon: WAND They can summon an Incarnate. */ display Modify the Ranger class as follows: Add the following public member functions /** @post: If the character is UNDEAD, gain 3 Vitality points. Nothing else happens. If the character is NOT UNDEAD, Vitality is set to 1. In addition, as a Mage: If the character is equipped with a wand or staff, they cast a healing ritual and recover vitality points – 2 points with a wand, 3 with a staff. If they can summon an incarnate, the emotional support allows the character to recover 1 Vitality point. */ eatTaintedStew /** @post : displays Ranger data in the form: “[NAME] is a Level [LEVEL] [RACE] RANGER. Vitality: [VITALITY] Armor: [ARMOR] They are [an enemy/not an enemy]. Animal Companion: [TRUE/FALSE] Arrows: [ARROW TYPE]: [QUANTITY] Affinities: [AFFINITY1], [AFFINITY2],…, [AFFINITYN] ” Example: MARROW is a Level 6 UNDEAD RANGER. Vitality: 9 Armor: 4 They are not an enemy. Animal Companion: TRUE Arrows: WOOD: 30 FIRE: 5 Affinities: FIRE, POISON */ display /** @post: If the character is UNDEAD, gain 3 Vitality points. Nothing else happens. If the character is NOT UNDEAD and, as a Ranger has POISON affinity, their Vitality is halved (round down to the nearest integer). Otherwise (not UNDEAD and not POISON affinity), their Vitality is set to 1. In either case, if they have an animal companion, the emotional support allows the character to recover 1 Vitality point. */ Modify the Scoundrel class as follows: Add the following public member functions eatTaintedStew /** @post : displays Scoundrel data in the form: “[NAME] is a Level [LEVEL] [RACE] SCOUNDREL. Vitality: [VITALITY] Armor: [ARMOR] They are [an enemy/not an enemy]. Dagger: [DAGGER] Faction: [FACTION] Disguise: [TRUE/FALSE] ” Example: FLEA is a Level 4 DWARF SCOUNDREL. Vitality: 6 Armor: 4 They are an enemy. Dagger: ADAMANT Faction: CUTPURSE Disguise: TRUE */ display /** @post: If the character is UNDEAD, gain 3 Vitality points. Nothing else happens. If the character is NOT UNDEAD, their Vitality is set to 1. In addition, as a Scoundrel: If the character is of the CUTPURSE faction, they steal a health potion and recover 3 Vitality points. If they are of the SILVERTONGUE faction, they talk the cook into redoing their stew as follows: they have a 70% chance of recovering 4 Vitality points, but a 30% chance of resetting their Vitality to 1, and they lose their daggers, which are replaced with WOOD daggers. (If their daggers were already WOOD, nothing happens to the daggers). */ eatTaintedStew Task 2: Modify the Tavern class – Modify the Tavern to now store pointers to Characters, rather than Character objects Implement a parameterized constructor /** @param: the name of an input file @pre: Formatting of the csv file is as follows (each numbered item appears separated by comma, only one value for each numbered item): 1. Name: An uppercase string 2. Race: An uppercase string [HUMAN, ELF, DWARF, LIZARD, UNDEAD] 3. Subclass: An uppercase string [BARBARIAN, MAGE, SCOUNDREL, RANGER] 4. Level/Vitality/Armor: A positive integer 5. Enemy: 0 (False) or 1 (True) 6. Main: Uppercase string or strings representing the main weapon (Barbarian and Mage), Dagger type (Scoundrel), or arrows (Ranger). A ranger’s arrows are of the form [TYPE] [QUANTITY];[TYPE] [QUANTITY], where each arrow type is separated by a semicolon, and the type and its quantity are separated with a space. Add the following public member functions Testing How to compile with your Makefile: In terminal, in the same directory as your Makefile and your source files, use the following command This assumes you did not rename the Makefile and that it is the only one in the current directory. 7. Offhand: An uppercase string that is only applicable to Barbarians, and may be NONE if the Barbarian does not have an offhand weapon, or if the character is of a different subclass. 8. School/Faction: Uppercase strings that represent a Mage’s school of magic: [ELEMENTAL, NECROMANCY, ILLUSION] or a Scoundrel’s faction: [CUTPURSE, SHADOWBLADE, SILVERTONGUE], and NONE where not applicable 9. Summoning: 0 (False) or 1 (True), only applicable to Mages (summoning an Incarnate) and Rangers (Having an Animal Companion) 10. Affinity: Only applicable to Rangers. Affinities are of the form [AFFINITY1];[AFFINITY2] where multiple affinities are separated by a semicolon. Th value may be NONE for a Ranger with no affinities, or characters of other subclasses. 11. Disguise: 0 (False) or 1 (True), only applicable to Scoundrels, representing if they have a disguise. 12. Enraged: 0 (False) or 1 (True), only applicable to Barbarians, representing if they are enraged. @post: Each line of the input file corresponds to a Character subclass and dynamically allocates Character derived objects, adding them to the Tavern. */ /** @post: For every character in the tavern, displays each character’s information */ displayCharacters /** @param: a string reference to a race @post: For every character in the tavern of the given race (only exact matches to the input string), displays each character’s information */ displayRace /** @post: Every character in the tavern eats a tainted stew. */ taintedStew make rebuild You must always implement and test your programs INCREMENTALLY!!! What does this mean? Implement and TEST one method at a time. For each class: Implement one function/method and test it thoroughly (write a main file with multiple test cases + edge cases if applicable). Only when you are certain that function works correctly and matches the specification, move on to the next. Implement the next function/method and test in the same fashion. How do you do this? Write your own main() function to test your classes. Choose the order in which you implement your methods so that you cantest incrementally: i.e. implement constructors then accessor functions, then mutator functions. Thoroughly test with valid and invalid input to check thet your function behaves as expected in each case. Pay special attention to edge cases. Sometimes functions depend on one another. If you need to use afunction you have not yet implemented, you can use stubs: a dummy implementation that always returns a single value for testing. Don’t forget to goback and implement the stub!!! If you put the word STUB in a comment, some editors will make it more visible. Grading Rubric Correctness 80% (distributed across unit testing of your submission) Documentation 15% Style and Design 5% (proper naming, modularity, and organization) Important: You must start working on the projects as soon as they are assigned to detect any problems with submitting your code and to address them withus well before the deadline so that we have time to get back to you before the deadline. There will be no negotiation about project grades after the submission deadline. Submission: We will grade the following : Character.hpp Character.cpp Barbarian.hpp Barbarian.cpp Mage.hpp Mage.cpp Ranger.hpp Ranger.cpp Scoundrel.hpp Scoundrel.cpp Tavern.hpp Tavern.cpp Although Gradescope allows multiple submissions, it is not a platform for testing and/or debugging and it should not be used for that. You MUST test anddebug your program locally. To help you not rely too much on Gradescope for testing, we will only allow 5 submissions per day. Before submitting toGradescope you MUST ensure that your program compiles using the provided Makefile and runs correctly on the Linux machines in the labs at Hunter(see detailed instructions on how to upload, compile and run your files in the “Programming Guidelines” document). That is your baseline, if it runscorrectly there it will run correctly on Gradescope, and if it does not, you will have the necessary feedback (compiler error messages, debugger or programoutput) to guide you in debugging, which you don’t have through Gradescope. “But it ran on my machine!” is not a valid argument for a submissionthat does not compile. Once you have done all the above you submit it to Gradescope. Due date: This project is due on October 27 at 11pm. No late submissions will be accepted. Important You must start working on the projects as soon as they are assigned to detect any problems and to address them with us well before the deadline so that we have time to get back to you before the deadline. There will be no extensions and no negotiation about project grades after the submission deadline. Help Help is available via drop-in tutoring in Lab 1001B (see website for schedule). You will be able to get help if you start early and go to the lab early. We onlyhave 2 UTAs in the lab, the days leading up to the due date will be crowded and you will not be able to get much help then.
It is now time to define the game setting. In this project you will implement the Tavern class, where your Characters will contend with each other. The link to accept the GitHub Classroom assignment can be found on Blackboard This project consists of two parts: You will modify the Character class You will implement Tavern, as a subclass of ArrayBag that holds Character objects Some additional resources Abstract Data Types: Geeks for Geeks Neso Academy Template Classes: CPP Manual Geeks for Geeks Tutorials Point Operator Overloading: CPP Manual Geeks for Geeks Programiz Implementation Work through the tasks sequentially (implement and test). Only move on to a task when you are positive that the previous one has been completedcorrectly. Remember that the names of classes and methods must exactly match those in this specification (FUNCTION NAMES, PARAMETERTYPES, RETURNS, PRE AND POST CONDITIONS MUST MATCH EXACTLY). Remember, you must thoroughly document your code!!! Task 1: Modify the Character class Define and implement the following additional public member functions: /** @param : A const reference to the right hand side of the == operator. @return : Returns true if the right hand side character is “equal”, false otherwise. Two characters are equal if they have the same name, same race, same level and are either both an enemy or not. NOTE: By this definition, only the aforementioned subset of the character’s attributes must be equal for two characters to be deemed “equal”. Example: In order for character1 to be == to character2 we only need: – The same name – The same race – The same level – They must either be both an enemy or not */ operator== /** @param `: A const reference to the right hand side of the != operator. @return : Returns true if the right hand side character is NOT “equal” (!=), false #### Task 2: Modify the ArrayBag class Define and implement the following additional public member functions: Task 3: Implement the Tavern class as a subclass of ArrayBag The Tavern is subclass of ArrayBag that stores Character objects Data Types The Tavern class must have the following private member variables: otherwise. Two characters are NOT equal if any of their name, race or level are not equal, or if one is an enemy and the other is not. NOTE: By this definition, one or more of the aforementioned subset of the character’s attributes only must be different for two characters to be deemed “NOT equal”. */ operator!= /** @post : displays Character data in the form: “[name_] is a Level [level_] [race_]. Vitality: [vitality_] Max Armor: [armor_] [They are / They are not] an enemy. ” */ display /** @param: A const reference to another ArrayBag object @post: Combines the contents from both ArrayBag objects, EXCLUDING duplicates. Example: [1, 2, 3] += [1, 4] will produce [1, 2, 3, 4] */ operator/= /** @param: A const reference to another ArrayBag object @post: Combines the contents from both ArrayBag objects, including duplicates, adding items from the argument bag as long as there is space. Example: [1, 2, 3] += [1, 4] will produce [1, 2, 3, 1, 4] */ operator+= – An integer sum of the levels of all the characters currently in the tavern – An integer count of all the characters currently in the Tavern that are marked as enemies The Tavern class must have the following public member functions: Constructor Unique Methods /** Default constructor. Default-initializes all private members. */ /** @param: A const reference to a Character entering the Tavern @return: returns true if a Character was successfully added to the tavern (i.e. items_), false otherwise @post: adds Character to the Tavern and updates the level sum and the enemy count if the character is an enemy. **/ enterTavern /** @param: A const reference to a Character leaving the Tavern @return: returns true if a character was successfully removed from the tavern (i.e. items_), false otherwise @post: removes the character from the Tavern and updates the level sum and the enemy count if the character is an enemy. **/ exitTavern /** @return: The integer level count of all the characters currently in the Tavern **/ getLevelSum /** @return: The average level of all the characters in the Tavern @post: Computes the average level of the Tavern rounded to the NEAREST integer. **/ calculateAvgLevel /** @return: The integer enemy count of the Tavern **/ getEnemyCount Testing How to compile with your Makefile: In terminal, in the same directory as your Makefile and your source files, use the following command This assumes you did not rename the Makefile and that it is the only one in the current directory. You must always implement and test your programs INCREMENTALLY!!! What does this mean? Implement and TEST one method at a time. For each class: Implement one function/method and test it thoroughly (write a main file with multiple test cases + edge cases if applicable). /** @return: The percentage (double) of all the enemy characters in the Tavern @post: Computes the enemy percentage of the Tavern rounded up to 2 decimal places. **/ calculateEnemyPercentage /** @param: A const reference to a string representing a character Race with value in [“NONE”, “HUMAN”, “ELF”, “DWARF”, “LIZARD”, “UNDEAD”] @return: An integer tally of the number of characters in the Tavern of the given race. If the argument string does not match one of the expected race values, the tally is zero. NOTE: no pre-processing of the input string necessary, only uppercase input will match. **/ tallyRace /** @post: Outputs a report of the characters currently in the tavern in the form: “Humans: [x] Elves: [x] Dwarves: [x] Lizards: [x] Undead: [x] The average level is: [x] [x]% are enemies. ” Note that the average level should be rounded to the NEAREST integer, and the enemy percentage should be rounded to 2 decimal places. Example output: Humans: 3 Elves: 5 Dwarves: 8 Lizards: 6 Undead: 0 The average level is: 7 46.67% are enemies. */ tavernReport make rebuild Only when you are certain that function works correctly and matches the specification, move on to the next. Implement the next function/method and test in the same fashion. How do you do this? Write your own main() function to test your classes. Choose the order in which you implement your methods so that you cantest incrementally: i.e. implement constructors then accessor functions, then mutator functions. Thoroughly test with valid and invalid input to check thet your function behaves as expected in each case. Pay special attention to edge cases. Sometimes functions depend on one another. If you need to use afunction you have not yet implemented, you can use stubs: a dummy implementation that always returns a single value for testing. Don’t forget to goback and implement the stub!!! If you put the word STUB in a comment, some editors will make it more visible. Grading Rubric Correctness 80% (distributed across unit testing of your submission) Documentation 15% Style and Design 5% (proper naming, modularity, and organization) Important: You must start working on the projects as soon as they are assigned to detect any problems with submitting your code and to address them withus well before the deadline so that we have time to get back to you before the deadline. There will be no negotiation about project grades after the submission deadline. Submission: We will grade the following : Character.hpp Character.cpp ArrayBag.hpp ArrayBag.cpp Tavern.hpp Tavern.cpp Although Gradescope allows multiple submissions, it is not a platform for testing and/or debugging and it should not be used for that. You MUST test anddebug your program locally. To help you not rely too much on Gradescope for testing, we will only allow 5 submissions per day. Before submitting toGradescope you MUST ensure that your program compiles using the provided Makefile and runs correctly on the Linux machines in the labs at Hunter(see detailed instructions on how to upload, compile and run your files in the “Programming Guidelines” document). That is your baseline, if it runscorrectly there it will run correctly on Gradescope, and if it does not, you will have the necessary feedback (compiler error messages, debugger or programoutput) to guide you in debugging, which you don’t have through Gradescope. “But it ran on my machine!” is not a valid argument for a submissionthat does not compile. Once you have done all the above you submit it to Gradescope. Due date: This project is due on October 13 at 11pm. No late submissions will be accepted. Important You must start working on the projects as soon as they are assigned to detect any problems and to address them with us well before the deadline so that we have time to get back to you before the deadline. There will be no extensions and no negotiation about project grades after the submission deadline. Help Help is available via drop-in tutoring in Lab 1001B (see website for schedule). You will be able to get help if you start early and go to the lab early. We onlyhave 2 UTAs in the lab, the days leading up to the due date will be crowded and you will not be able to get much help then.
You will now define different roles for your character. You will implement 4 new classes that derive from the Character class but also have additional attributes and methods specific to their type: Mages are scholars of magic and spells, and are able to manipulate the world around them with elemental magic. Each Mage has an affinity to a specificschool of magic, a choice of weapon, and an optional choice of summoning an incarnate. Scoundrels live by their wits, take chances when they must, and land on their feet against all odds. Every Scoundrel has a set of daggers made from acertain material (e.g. wood, bronze, iron, steel, mithril, adamant, and rune) faction that they belong to: Cutpurse, Nightblades, or Silvertongue (e.g, thieves, assassins, and swindlers). Scoundrels may or may not have a disguise. Rangers specialize in hunting the monsters that threaten the edges of civilization. Each Ranger is equipped with a crossbow that fires from a limitedsource of ammunition. Arrows can come with different effects, so a ranger’s ammunition source includes a range of different arrows. A Ranger may ormay not be able to recruit an animal companion. Each Ranger also has a list of affinities and advantages, which increases the damage dealt with anarrow of the same element. Barbarians are the archetypal warriors who use brute strength and raw fury to excel in combat. A Barbarian can wield either a two-handed weapon, or aone-handed weapon and a shield. They may or may not be enraged, which allows them to deal double damage. The link to accept the GitHub Classroom assignment can be found on Blackboard Some additional resources on Inheritance, enums and struct Code Beauty on inheritance programming Caleb Curry on base and subclass inheritance enums struct Implementation Work through the tasks sequentially (implement and test). Only move on to a task when you are positive that the previous one has been completedcorrectly. Remember that the names of classes and methods must exactly match those in this specification (FUNCTION NAMES, PARAMETERTYPES, RETURNS, PRE AND POST CONDITIONS MUST MATCH EXACTLY). Remember, you must thoroughly document your code!!! Task 1: Sublcasses The Mage class Data Types The Mage class must have the following private member variables: Constructors – A string that represents their school of magic – A string that represents their weapon – A boolean indicating if they are able to summon an incarnate /** Default constructor. Default-initializes all private members. Default character name: “NAMELESS”. Booleans are default-initialized to False. Default school of magic and weapon: “NONE”. */ /** Parameterized constructor. @param : The name of the character (a const string reference) @param : The race of the character (a const string reference) @param : The character’s vitality (an integer). Default to 0 @param : The character’s max armor level (an integer). Default to 0 @param : The character’s level (an integer). Default to 0 @param : A flag indicating whether the character is an enemy. Default to false. @param : The character’s school of magic (a const string reference). Valid schools: [ELEMENTAL, NECROMANCY, ILLUSION]. String inputs can be in lowercase, but must be converted to uppercase when setting the variable. If the school name is invalid, set it to “NONE” Unique methods @param : The character’s choice of weapon (a const string reference). Valid weapons: [WAND, STAFF] String inputs can be in lowercase, but must be converted to uppercase when setting the variable. If the weapon is invalid, set it to “NONE” @param : A flag indicating whether the character is able to summon an incarnate. Default to false. @post : The private members are set to the values of the corresponding parameters. REMEMBER: If the school of magic or weapon is not provided or invalid, the variables should be set to “NONE”. */ /** @param : a reference to a string representing the school of magic @post : sets the private member variable to the value of the parameter. If the provided school of magic is not one of the following: [ELEMENTAL, NECROMANCY, ILLUSION], do nothing and return false. String inputs can be in lowercase, but must be converted to uppercase when setting the variable. @return : true if setting the variable was successful, false otherwise. **/ setSchool /** @return : the string indicating the character’s school of magic **/ getSchool /** @param : a reference to a string representing the character’s weapon @post : sets the private member variable to the value of the parameter. String inputs can be in lowercase, but must be converted to uppercase when setting the variable. If the provided weapon is not one of the following: [WAND, STAFF], do nothing and return false. @return : true if setting the variable was successful, false otherwise. **/ setCastingWeapon /** @return : the string indicating the character’s weapon **/ getCastingWeapon /** @param : a reference to boolean @post : sets the private member variable indicating whether the character can summon an incarnate **/ setIncarnateSummon The Scoundrel class Data Types The Scoundrel class must have the following type: /** @return : the summon-incarnate flag **/ hasIncarnateSummon The Scoundrel class must have the following private member variables: The Scoundrel class must have the following public member functions: Constructors Unique Methods – an enum Dagger with the values {WOOD, BRONZE, IRON, STEEL, MITHRIL, ADAMANT, RUNE} – A dagger – A string that represents their Faction – A boolean indicating if they have a disguise /** Default constructor. Default-initializes all private members. Default character name: “NAMELESS”. Booleans are default-initialized to False. Default dagger: WOOD. Default faction: “NONE”. */ /** Parameterized constructor. @param : The name of the character (a const string reference) @param : The race of the character (a const string reference) @param : The character’s vitality (an integer). Default to 0 @param : The character’s max armor level (an integer). Default to 0 @param : The character’s level (an integer). Default to 0 @param : A flag indicating whether the character is an enemy. Default to false. @param : The character’s dagger type (a const string reference). String inputs can be in lowercase, but must be converted to uppercase when setting the dagger enum. Default to WOOD @param : The character’s Faction (a const string reference). Valid Factions: [CUTPURSE, SHADOWBLADE, SILVERTONGUE] String inputs can be in lowercase, but must be converted to uppercase when setting the variable. Default to “NONE” @param : A flag indicating whether the character has a disguise. Default to false @post : The private members are set to the values of the corresponding parameters. If the dagger type is not provided or invalid, the variable should be set to WOOD. If the Faction is not provided or invalid, the variable should be set to “NONE”. */ /** @param : a reference to a string representing the dagger type @post : sets the private member variable to the value of the parameter. String inputs can be in lowercase, but must be converted to uppercase when setting the variable. If the dagger type is not valid (i.e, is one of the following: [WOOD, BRONZE, IRON, STEEL, MITHRIL, ADAMANT, RUNE]), the variable should be set to WOOD. **/ setDagger /** @return : the string indicating the character’s dagger type **/ getDagger /** @param : a reference to a string representing the character’s Faction @post : sets the private member variable to the value of the parameter. String inputs can be in lowercase, but must be converted to uppercase when setting the variable. If the provided faction is not one of the following: [NONE, CUTPURSE, SHADOWBLADE, SILVERTONGUE], do nothing and return false. @return : true if setting the variable was successful, false otherwise. **/ setFaction /** @return : the string indicating the character’s Faction **/ getFaction /** @param : a reference to boolean @post : sets the private member variable indicating whether the character has a disguise **/ setDisguise /** @return : the visual aid flag **/ hasDisguise The Ranger class Data Types The Ranger class must have the following type: The Ranger class must have the following private member variables: struct Arrows { string type_; int quantity_; }; Constructors Unique Methods – A vector of arrows – A vector of affinities (strings) – A boolean indicating if they are able to recruit an animal companion /** Default constructor. Default-initializes all private members. Default character name: “NAMELESS”. Booleans are default-initialized to False. */ /** Parameterized constructor. @param : The name of the character (a const string reference) @param : The race of the character (a const string reference) @param : The character’s vitality (an integer). Default to 0 @param : The character’s max armor level (an integer). Default to 0 @param : The character’s level (an integer). Default to 0 @param : A flag indicating whether the character is an enemy. Default to false @param : A vector of arrows. Valid arrow types are: [WOOD, FIRE, WATER, POISON, BLOOD] Lowercase valid arrow types are retained but converted to uppercase. Invalid arrows are those with non-positive quantities or invalid types. If the vector contains invalid arrows, those arrows are discarded. Default to empty vector @param : A vector of affinities. Valid Affinities: [FIRE, WATER, POISON, BLOOD] String inputs can be in lowercase, but must be converted to uppercase. If the vector contains invalid affinities, those affinities are discarded. Default to empty vector @param : A flag indicating whether the character is able to recruit an animal companion. Default to false @post : The private members are set to the values of the corresponding parameters */ /** @return : a vector of the Character’s arrows **/ getArrows /** @param : a reference to string representing the arrow type @param : a reference to an integer quantity @post : If the character already has that type of arrow, the quantity in the vector is updated. If not, the arrow is added to the vector. Valid arrow types are: [WOOD, FIRE, WATER, POISON, BLOOD] Lowercase valid arrow types are retained but converted to uppercase. Quantity of arrows must be greater than 0 Invalid arrows are those with non-positive quantities or invalid types. ## Note: This class uses a struct type. Struct is just short for “structure” and it is a user-defined type, much like a class. It is not a full fledge class, however. We use it to bundle data items with no or very few functions (non in this case). When initializing a new review you may use review as the data type. Forexample for int x, int is our primitive data type. If we want to make a review called x we would write review x. More on structs here: https://www.educative.io/edpresso/what-is-a-cpp-struct ## If the arrows are invalid, they are not added. @return : True if the arrows were added successfully, false otherwise **/ addArrows /** @param : a reference to string representing the arrow type @post : If the character has the listed arrow AND enough arrows to fire one, the quantity of remaining arrows in the vector is updated. @return : True if the character had the listed arrow AND enough arrows, False otherwise. **/ fireArrow /** @param : a reference to string representing an affinity @post : If the affinity does not already exist in the vector, add it to the vector. Valid Affinities: [FIRE, WATER, POISON, BLOOD] String inputs can be in lowercase, but must be converted to uppercase when setting the variable. There should be no duplicate affinities. If the affinity is invalid, it is NOT added. @return : True if the affinity was added successfully, false otherwise **/ addAffinity /** @return : a vector of the Character’s affinities **/ getAffinities /** @param : a reference to a boolean indicating whether the character is able to recruit an animal companion @post : sets the private member variable to the value of the parameter. **/ setCompanion /** @return : a boolean indicating whether the character is able to recruit an animal companion **/ getCompanion The Barbarian class Data Types The Barbarian class must have the following private member variables: Constructors – A string representing their main weapon – A string representing their offhand weapon – A boolean indicating if they are enraged /** Default constructor. Default-initializes all private members. Default character name: “NAMELESS”. Booleans are default-initialized to False. Default weapons: “NONE”. */ /** Parameterized constructor. @param : The name of the character (a const string reference) @param : The race of the character (a const string reference) @param : The character’s vitality (an integer). Default to 0 @param : The character’s max armor level (an integer). Default to 0 @param : The character’s level (an integer). Default to 0 @param : A flag indicating whether the character is an enemy. Default to false @param : The character’s main weapon (a string). String inputs can be in lowercase, but must be converted to uppercase when setting the variable. Only alphabetical characters are allowed. Default to “NONE” @param : The character’s offhand weapon (a string). Unique Methods String inputs can be in lowercase, but must be converted to uppercase when setting the variable. Only alphabetical characters are allowed. Default to “NONE” @param : A flag indicating whether the character is enraged. Default to false @post : The private members are set to the values of the corresponding parameters. : If the main and secondary weapons are not provided or invalid, the variables are set to “NONE”. */ /** @param : a reference to a string representing the Character’s main weapon @post : sets the private member variable to the value of the parameter. Only alphabetical characters are allowed. String inputs can be in lowercase, but must be converted to uppercase when setting the variable. If the given input is invalid (i.e contains numbers), do nothing and return false @return : true if setting the variable was successful, false otherwise. **/ setMainWeapon /** @return : a string of the Character’s main weapon **/ getMainWeapon /** @param : a reference to a string representing the Character’s secondary weapon @post : sets the private member variable to the value of the parameter. Only alphabetical characters are allowed. String inputs can be in lowercase, but must be converted to uppercase when setting the variable. : If the given input is invalid (i.e contains numbers), do nothing and return false @return : true if setting the variable was successful, false otherwise. **/ setSecondaryWeapon /** @return : a string of the Character’s secondary weapon **/ getSecondaryWeapon /** @param : a reference to a bool @post : sets the private member variable to the value of the parameter. **/ setEnrage /** @return : a boolean of whether the Character is enraged **/ getEnrage Task 2: Testing To give you some feel and guide into testing, we have added a testing task to this project. To be clear, the task below is not a thorough test of your classes. You must test every function you implement with valid and invalid values, and make sure it behaves as specified. This is only intended to motivate you intobuilding a methodical and incremental testing practice. Write a testing program as described below and name the file test.cpp. Instantiate the following character objects that derive from the Character class. For each character object, print out the character’s information – includingprivate member variables that are unique to their subclass – using the appropriate getter functions in the format: “[NAME] is a Level [LEVEL] [RACE]. Vitality: [VITALITY] Armor: [ARMOR] They are [an enemy/not an enemy] [SUBCLASS ATTRIBUTE] : [SUBCLASS ATTRIBUTE VALUE]…”, where [SUBCLASS ATTRIBUTE] : [SUBCLASS ATTRIBUTE VALUE]… means that you add any additional subclass-specific variable name and itsvalue after a colon, each on a new line. /** @post : sets the enraged variable to the opposite of what it was **/ toggleEnrage 2.1.1 Instantiate a default Mage – Sets its name, race, vitality, and armor levels using the appropriate setter functions 2.1.2 Instantiate a Mage with the parameterized constructor with the following character details: Name: SPYNACH Race: ELF Level: 4 Vitality: 6 Enemy: FALSE 2.1.3 Set SPYNACH’s unique private member variables to the following: School of Magic: Illusion Weapon: Wand Summon Incarnate: FALSE 2.2.1 Instantiate a default Scoundrel – Sets its name, race, vitality, and armor levels using the appropriate setter functions 2.2.2 Instantiate a Scoundrel with the parameterized constructor with the following character details: Name: FLEA Race: DWARF Level: 7 Vitality: 12 Enemy: FALSE 2.2.3 Set FLEA’s unique private member variables to the following: Dagger: Adamant Faction: Cutpurse Disguise: TRUE Testing How to compile with your Makefile: In terminal, in the same directory as your Makefile and your source files, use the following command This assumes you did not rename the Makefile and that it is the only one in the current directory. You must always implement and test your programs INCREMENTALLY!!! What does this mean? Implement and TEST one method at a time. For each class: Implement one function/method and test it thoroughly (write a main file with multiple test cases + edge cases if applicable). Only when you are certain that function works correctly and matches the specification, move on to the next. 2.3.1 Instantiate a default Ranger – Sets its name, race, vitality, and armor levels using the appropriate setter functions 2.2.2 Instantiate a Ranger with the parameterized constructor with the following character details: Name: MARROW Race: UNDEAD Level: 6 Vitality: 9 Enemy: TRUE 2.2.3 Set MARROW’s unique private member variables to the following: Vector of arrows: Wood, 30, Fire, 5, Water, 5, Poison, 5 Affinities: Fire, Poison Animal Companian: TRUE 2.4.1 Instantiate a default Barbarian – Sets its name, race, vitality, and armor levels using the appropriate setter functions 2.4.2 Instantiate a Barbarian with the parameterized constructor with the following character details: Name: BONK Race: HUMAN Level: 5 Vitality: 11 Enemy: TRUE 2.4.3 Set BONK’s unique private member variables to the following: Main Weapon: MACE Offhand Weapon: ANOTHERMACE Enraged: TRUE make rebuild Implement the next function/method and test in the same fashion. How do you do this? Write your own main() function to test your classes. Choose the order in which you implement your methods so that you cantest incrementally: i.e. implement constructors then accessor functions, then mutator functions. Thoroughly test with valid and invalid input to check thet your function behaves as expected in each case. Pay special attention to edge cases. Sometimes functions depend on one another. If you need to use afunction you have not yet implemented, you can use stubs: a dummy implementation that always returns a single value for testing. Don’t forget to goback and implement the stub!!! If you put the word STUB in a comment, some editors will make it more visible. Grading Rubric Correctness 80% (distributed across unit testing of your submission) Documentation 15% Style and Design 5% (proper naming, modularity, and organization) Important: You must start working on the projects as soon as they are assigned to detect any problems with submitting your code and to address them withus well before the deadline so that we have time to get back to you before the deadline. There will be no negotiation about project grades after the submission deadline. Submission: We will grade the following : Barbarian.hpp Barbarian.cpp Mage.hpp Mage.cpp Ranger.hpp Ranger.cpp Scoundrel.hpp Scoundrel.cpp test.cpp Although Gradescope allows multiple submissions, it is not a platform for testing and/or debugging and it should not be used for that. You MUST test anddebug your program locally. To help you not rely too much on Gradescope for testing, we will only allow 5 submissions per day. Before submitting toGradescope you MUST ensure that your program compiles using the provided Makefile and runs correctly on the Linux machines in the labs at Hunter(see detailed instructions on how to upload, compile and run your files in the “Programming Guidelines” document). That is your baseline, if it runscorrectly there it will run correctly on Gradescope, and if it does not, you will have the necessary feedback (compiler error messages, debugger or programoutput) to guide you in debugging, which you don’t have through Gradescope. “But it ran on my machine!” is not a valid argument for a submissionthat does not compile. Once you have done all the above you submit it to Gradescope. Due date: This project is due on September 22 at 11pm. No late submissions will be accepted. Important You must start working on the projects as soon as they are assigned to detect any problems and to address them with us well before the deadline so that we have time to get back to you before the deadline. There will be no extensions and no negotiation about project grades after the submission deadline. Help Help is available via drop-in tutoring in Lab 1001B (see website for schedule). You will be able to get help if you start early and go to the lab early. We onlyhave 2 UTAs in the lab, the days leading up to the due date will be crowded and you will not be able to get much help then.
This semester we will have some fun setting up our very own role playing game (inspired by the Divinity series). We hope you’ll enjoy this adventure asmuch as we did setting it up for you! This is a baseline project whose objective is to get you acquainted with the platforms we will use in this course and to refresh your knowledge of basicOOP. You will implement the Character class. In order to successfully complete this project, we strongly recommend that you look back to your CSCI135 coursework as a reference. First of all, this project will introduce you to GitHub Classroom so you can work with git version control. All projects in this course will be distributed viaGitHub Classroom and submitted to Gradescope via GitHub. We truly hope you will start establishing best practices of version control, INCREMENTAL coding, testing and debugging (i.e. code, test and debug one function at a time); you will need it in the near future, so better start now! Part 1 – getting started with GitHub Classroom: If you don’t already have one, go to https://github.com/ and create a GitHub account. You will likely use your GitHub account professionally in thefuture, so choose a username you will want to keep. Next, watch this video to brush-up on or learn the basics of git and GitHub: For this project we will use GitHub Classroom. The following video will guide you through the entire process – from accepting an assignment tosubmitting your solution: Although the video is about a different course, the instructions are the same (with different repo and file names). The only difference is that we will not add a distribution branch, so you can ignore the part where it says to execute the two git commands in the readme file; there are not extrainstructions in the readme file on our repo). The link to accept the GitHub Classroom assignment can be found on Blackboard The above video will also show you how to submit to Gradescope via GitHub. Make sure to refer back to these instructions when it’s time to submit. Some additional resources if you need to brush up on basic OOP Code Beauty on constructors and class methods thenewboston on classes and objects McProgramming on .hpp and .cpp files Part 2 – The Character Class : You will implement the Character class. You must always separate interface from implementation(Character.hpp and Character.cpp), and you ONLY EVER include a class’ interface(.hpp) This will be an implicit assumption in this course going forward. Work through the tasks sequentially (implement and test). Only move on to atask when you are positive that the previous one has been completed correctly. Remember that the names of classes and methods must exactly match thosein this specification (FUNCTION NAMES, PARAMETER TYPES, RETURNS, PRE AND POST CONDITIONS MUST MATCH EXACTLY). Thisclass has only accessor and mutator functions for its public data members. Recall that accessor functions (e.g. getName()) are used to access the privatedata members (e.g. all getName() will do is return name_, the private data member) and are therefore declared const, while mutator functions (e.g. setName()) give a value to the data members, and do not modify it’s parameters, which will be passed by const reference. Remember, you must thoroughly document your code!!! Task: The Character class Every Character has a Name, Race, Health, Armor, Level, and a boolean if the character is an enemy. The Character class must define the following types: An enum named Race with values {NONE, HUMAN, ELF, DWARF, LIZARD, UNDEAD} The Character class must have the following private member variables: The Character class must have the following public member functions: Constructors private: – The name of the character (a string in UPPERCASE) – The race of the character (an enum) – The character’s vitality (a non-negative integer) – The character’s max armor level (a non-negative integer) – The character’s level (a non-negative integer) – A flag indicating whether the character is an enemy /** Default constructor. Default-initializes all private members. Default character name: “NAMELESS”. Booleans are default-initialized to False. Default enum value: NONE. Default Vitality, Max Armor, and Level: 0. */ Hint: Notice the default argument in the parameterized constructor. Accessors (get) and Mutators (set) /** Parameterized constructor. @param : The name of the character (a string in UPPERCASE) @param : The race of the character (a string) @param : The character’s vitality (a non-negative integer) , with default value 0 @param : The character’s max armor level (a non-negative integer), with default value 0 @param : The character’s level (non-negative integer), with default value 0 @param : A flag indicating whether the character is an enemy, with default value false @post : The private members are set to the values of the corresponding parameters. */ /** @param : the name of the Character @post : sets the Character’s title to the value of the parameter, in UPPERCASE. Only alphabetical characters are allowed. For example, attempting to create a character named “TW3EDLEDUM2” should create a character named “TWEDLEDUM”. : If the given parameter does not have any valid alphabetical characters, the character’s name should be set to “NAMELESS”. */ setName /** @return : the name of the Character */ getName /** @param : the race of the Character (a string) @post : sets the Character’s race to the value of the parameter. If the given race was invalid, set race_ to NONE. */ setRace /** @return : the race of the Character (a string) */ getRace /** @param : an integer vitality @pre : vitality >= 0 : Characters cannot have negative health @post : sets the vitality private member to the value of the parameter if vitality is negative, do nothing. */ setVitality /** @return : the value stored in vitality_ */ getVitality /** @param : an integer armor level @pre : armor >= 0 : Characters cannot have negative armor @post : sets the armor private member to the value of the parameter if armor is negative, do nothing. */ setArmor /** @return : the value stored in armor_ */ getArmor /** @param : an integer level @pre : level >= 0 : Characters cannot have a negative @post : sets the level private member to the value of the parameter if level is negative, do nothing. */ setLevel /** @return : the value stored in level_ */ getLevel /** @post : sets the enemy flag to true */ setEnemy /** @return true if the character is an enemy, false otherwise Note: this is an accessor function and must follow the same convention as all accessor functions even if it is not called getEnemy */ isEnemy Submission: You will submit your solution to Gradescope via GitHub Classroom (see video linked above). The autograder will grade the following files only: Character.hpp Character.cpp Although Gradescope allows multiple submissions, it is not a platform for testing and/or debugging and it should not be used for that. You MUST test anddebug your program locally. To help you not rely too much on Gradescope for testing, we will only allow 5 submissions per day. Before submitting toGradescope you MUST ensure that your program compiles using the provided Makefile and runs correctly on the Linux machines in the labs at Hunter(see detailed instructions on how to upload, compile and run your files in the “Programming Guidelines” document). That is your baseline, if it runscorrectly there it will run correctly on Gradescope, and if it does not, you will have the necessary feedback (compiler error messages, debugger or programoutput) to guide you in debugging, which you don’t have through Gradescope. “But it ran on my machine!” is not a valid argument for a submissionthat does not compile. Once you have done all the above you submit it to Gradescope. Testing How to compile with your Makefile: In terminal, in the same directory as your Makefile and your source files, use the following command This assumes you did not rename the Makefile and that it is the only one in the current directory. You must always implement and test your programs INCREMENTALLY!!! What does this mean? Implement and TEST one method at a time. For each class: Implement one function/method and test it thoroughly (write a main file with multiple test cases + edge cases if applicable). Only when you are certain that function works correctly and matches the specification, move on to the next. Implement the next function/method and test in the same fashion. How do you do this? Write your own main() function to test your classes (we will provide one with this first project, but you can always add to it). Inthis course we will not grade your test program, but you must always write one to test your classes. Choose the order in which you implement yourmethods so that you can test incrementally: i.e. implement constructors then accessor functions, then mutator functions. Sometimes functions depend onone another. If you need to use a function you have not yet implemented, you can use stubs: a dummy implementation that always returns a single valuefor testing. Don’t forget to go back and implement the stub!!! If you put the word STUB in a comment, some editors will make it more visible. Grading Rubrics Correctness 80% (distributed across unit testing of your submission) Documentation 15% Style and Design 5% (proper naming, modularity, and organization) make rebuild Due date: This project is due on September 8 at 11pm. No late submissions will be accepted. Important You must start working on the projects as soon as they are assigned to detect any problems and to address them with us well before the deadline so that we have time to get back to you before the deadline. There will be no extensions and no negotiation about project grades after the submission deadline. Help Help is available via drop-in tutoring in Lab 1001B (see website for schedule). You will be able to get help if you start early and go to the lab early. We onlyhave 2 UTAs in the lab, the days leading up to the due date will be crowded and you will not be able to get much help then.
Define and implement the Skill and Profemon class with a header and a source file (From now on, when you implement a class, assume you will implement two files: .hpp and .cpp).The Skill class must have the following information stored in their private member variables:The Skill class must have the following in it’s public access modifier:The header file of the Profemon class must have an enumeration type called “Specialty” with three possible values: ML, SOFTWARE, and HARDWARE in this exact order. The enum should be defined outside the class.The Profemon class must have the following information stored in their private member variables:The Profemon class must have the following in it’s public access modifier:You will submit the following files to gradescope:profemon.hpp, profemon.cpp, skill.hpp, skill.cppHere is a sample main file. Make sure to add your own code to test all functions.Compile your code locally using:
1 Clustering. [100 points total. Each part is 25 points.] [a-b] Given N data points xn(n = 1, . . . , N), K-means clustering algorithm groups them into K clusters by minimizing the distortion function over {r nk, µk} J = X N n=1 X K k=1 r nkkx n − µ k k 2 , where r nk = 1 if xn belongs to the k-th cluster and r nk = 0 otherwise. (a) Prove that using the squared Euclidean distance kx n − µ kk 2 as the dissimilarity function and minimizing the distortion function, we will have µ k = P n r nkx n P n r nk . That is, µ k is the center of k-th cluster. [5 pts] (b) Prove that K-means algorithm converges to a local optimum in finite steps. [5 pts] [c-d] In class, we discussed bottom-up hierarchical clustering. For each iteration, we need to find two clusters {x1, x2, . . . , xm} and {y1 , y2 , . . . , yp} with the minimum distance to merge. Some of the most commonly used distance metrics between two clusters are: • Single linkage: the minimum distance between any pairs of points from the two clusters, i.e. min i=1,…,m j=1,…,p kxi − yjk • Complete linkage: the maximum distance between any parts of points from the two clusters, i.e. max i=1,…,m j=1,…,p kxi − yjk • Average linkage: the average distance between all pair of points from the two clusters, i.e. 1 mp Xm i=1 Xp j=1 kxi − yjk 1 (c) When we use the bottom up hierarchical clustering to realize the partition of data, which of the three cluster distance metrics described above would most likely result in clusters most similar to those given by K-means? (Suppose K is a power of 2 in this case). [5 pts] (d) For the following data (two moons), which of these three distance metrics (if any) would successfully separate the two moons? [5 pts] −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 −2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 21 Image compression using clustering [40 points] In this programming assignment, you are going to apply clustering algorithms for image compression. To ease your implementation, we provide a skeleton code containing image processing part. homework2.m is designed to read an RGB bitmap image file, then cluster pixels with the given number of clusters K. It shows converted image only using K colors, each of them with the representative color of centroid. To see what it looks like, you are encouraged to run homework2(‘beach.bmp’, 3) or homework2(‘football.bmp’, 2), for example. Your task is implementing the clustering parts with two algorithms: K-means and K-medoids. We learned and demonstrated K-means in class, so you may start from the sample code we distributed. The file you need to edit is mykmeans.m and mykmedoids.m, provided with this homework. In the files, you can see it calls Matlab function kmeans initially. Comment this line out, and implement your own in the files. You would expect to see similar result with your implementation of K-means, instead of kmeans function in Matlab. K-medoids In class, we learned that the basic K-means works in Euclidean space for computing distance between data points as well as for updating centroids by arithmetic mean. Sometimes, however, the dataset may work better with other distance measures. It is sometimes even impossible to compute arithmetic mean if a feature is categorical, e.g, gender or nationality of a person. With K-medoids, you choose a representative data point for each cluster instead of computing their average. Given N data points xn(n = 1, …, N), K-medoids clustering algorithm groups them into K clusters by minimizing the distortion function J = PN n=1 PK k=1 r nkD(xn, µk ), where D(x, y) is a distance measure between two vectors x and y in same size (in case of K-means, D(x, y) = kx − yk 2 ), µ k is the center of k-th cluster; and r nk = 1 if xn belongs to the k-th cluster and r nk = 0 otherwise. In this exercise, we will use the following iterative procedure: • Initialize the cluster center µ k , k = 1, …, K. • Iterate until convergence: – Update the cluster assignments for every data point xn: r nk = 1 if k = arg minj D(xn, µj ), and r nk = 0 otherwise. – Update the center for each cluster k: choosing another representative if necessary. There can be many options to implement the procedure; for example, you can try many distance measures in addition to Euclidean distance, and also you can be creative for deciding a better representative of each cluster. We will not restrict these choices in this assignment. You are encouraged to try many distance measures as well as way of choosing representatives. 1 Formatting instruction Both mykmeans.m and mykmedoids.m take input and output format as follows. You should not alter this definition, otherwise your submission will print an error, which leads to zero credit. Input • pixels: the input image representation. Each row contains one data point (pixel). For image dataset, it contains 3 columns, each column corresponding to Red, Green, and Blue component. Each component has an integer value between 0 and 255. • K: the number of desired clusters. Too high value of K may result in empty cluster error. Then, you need to reduce it. Output • class: cluster assignment of each data point in pixels. The assignment should be 1, 2, 3, etc. For K = 5, for example, each cell of class should be either 1, 2, 3, 4, or 5. The output should be a column vector with size(pixels, 1) elements. • centroid: location of K centroids (or representatives) in your result. With images, each centroid corresponds to the representative color of each cluster. The output should be a matrix with K rows and 3 columns. The range of values should be [0, 255], possibly floating point numbers. Hand-in Both of your code and report will be evaluated. Upload codes with your implementation. In your report, answer to the following questions: 1. Within the K-medoids framework, you have several choices for detailed implementation. Explain how you designed and implemented details of your K-medoids algorithm, including (but not limited to) how you chose representatives of each cluster, what distance measures you tried and chose one, or when you stopped iteration. 2. Attach a picture of your own. We recommend size of 320 × 240 or smaller. 3. Run your K-medoids implementation with the picture you chose above, with several different K. (e.g, small values like 2 or 3, large values like 16 or 32) What did you observe with different K? How long does it take to converge for each K? 4. Run your K-medoids implementation with different initial centroids/representatives. Does it affect final result? Do you see same or different result for each trial with different initial assignments? (We usually randomize initial location of centroids in general. To answer this question, an intentional poor assignment may be useful.) 5. Repeat question 2 and 3 with K-means. Do you see significant difference between K-medoids and K-means, in terms of output quality, robustness, or running time? Note • You may see some error message about empty clusters even with Matlab implementation, when you use too large K. Your implementation should treat this exception as well. That is, do not terminate even if you have an empty cluster, but use smaller number of clusters in that case. • We will grade using test pictures which are not provided. We recommend you to test your code with several different pictures so that you can detect some problems that might happen occasionally. 2 • If we detect copy from any other student’s code or from the web, you will not be eligible for any credit for the entire homework, not just for the programming part. Also, directly calling Matlab function kmeans or other clustering functions is not allowed. 2 Spectral clustering [40 points] 1. Consider an undirected graph with non-negative edge weights wij and graph Laplacian L. Suppose there are m connected components A1, A2, . . . , Am in the graph. Show that there are m eigenvectors of L corresponding to eigenvalue zero, and the indicator vectors of these components IA1 , . . . , IAm span the zero eigenspace. 2. Real data: political blogs dataset. We will study a political blogs dataset first compiled for the paper Lada A. Adamic and Natalie Glance, “The political blogosphere and the 2004 US Election”, in Proceedings of the WWW-2005 Workshop on the Weblogging Ecosystem (2005). The dataset nodes.txt contains a graph with n = 1490 vertices (“nodes”) corresponding to political blogs. Each vertex has a 0-1 label (in the 3rd column) corresponding to the political orientation of that blog. We will consider this as the true label and try to reconstruct the true label from the graph using the spectral clustering on the graph. The dataset edges.txt contains edges between the vertices. Here we assume the number of clusters to be estimated is k = 2. Using spectral clustering to find the 2 clusters. Compare the clustering results with the true labels. What is the false classification rate (the percentage of nodes that are classified incorrectly). 3 PCA: Food consumption in European area [20 points] The data food-consumption.csv contains 16 countries in the European area and their consumption for 20 food items, such as tea, jam, coffee, yoghurt, and others. There are some missing data entries: you may remove the rows “Sweden”, “Finland”, and “Spain”. Implement PCA by writing your own code. 1. Find the first two principal directions, and plot them. 2. Compute the reduced representation of the data point (which are sometimes called the principal components of the data). Draw a scatter plot of two-dimensional reduced representation for each country. Do you observe some pattern? 31. Density estimation: Psychological experiments. (50 points) The data set n90pol.csv contains information on 90 university students who participated in a psychological experiment designed to look for relationships between the size of different regions of the brain and political views. The variables amygdala and acc indicate the volume of two particular brain regions known to be involved in emotions and decision-making, the amygdala and the anterior cingulate cortex; more exactly, these are residuals from the predicted volume, after adjusting for height, sex, and similar body-type variables. The variable orientation gives the students’ locations on a five-point scale from 1 (very conservative) to 5 (very liberal). (a) Form 2-dimensional histogram for the pairs of variables (amygdala, acc). Decide on a suitable number of bins so you can see the shape of the distribution clearly. (b) Now implement kernel-density-estimation (KDE) to estimate the 2-dimensional with a two-dimensional density function of (amygdala, acc). Use a simple multi-dimensional Gaussian kernel, for x = x1 x2 ∈ R 2 , where x1 and x2 are the two dimensions respectively K(x) = 1 √ 2π e − (x 2 1+x 2 2 ) 2 . Recall in this case, the kernel density estimator (KDE) for a density is given by p(x) = 1 m Xm i=1 1 h K x i − x h where x i are two-dimensional vectors, h > 0 is the kernel bandwidth. Set an appropriate h so you can see the shape of the distribution clearly. Plot of contour plot (like the ones in slides) for your estimated density. (c) Plot the condition distribution of the volume of the amygdala as a function of political orientation: p(amygdala|orientation = a), a = 1, . . . , 5. Do the same for the volume of the acc. Plot p(acc|orientation = a), a = 1, . . . , 5. You may either use histogram or KDE to achieve the goal. 2. Implementing EM algorithm for MNIST dataset. (50 points) Implement the EM algorithm for fitting a Gaussian mixture model for the MNIST dataset. We reduce the dataset to be only two cases, of digits “2” and “6” only. Thus, you will fit GMM with C = 2. Use the data file data.mat or data.dat on Canvas. True label of the data are also provided in label.mat and label.dat The matrix images is of size 784-by-1990, i.e., there are totally 1990 images, and each column of the matrix corresponds to one image of size 28-by-28 pixels (the image is vectorized; the original image can be recovered, e.g., using MATLAB code, reshape(images(:,1),28, 28). 1 (a) Select from data one raw image of “2” and “6” and visualize them, respectively. (b) Use random Gaussian vector with zero mean as initial means, and identity matrix as initial covariance matrix for the clusters. Please plot the log-likelihood function versus the number of iterations to show your algorithm is converging. (c) Report the finally fitting GMM model when EM terminates: the weights for each component, the mean vectors (please reformat the vectors into 28-by-28 images and show these images in your submission). Ideally, you should be able to see these means corresponds to “average” images. No need to report the covariance matrices. (d) (Optional). Use the pic to infer the labels of the images, and compare with the true labels. Report the miss classification rate for digits “2” and “6” respectively. Perform K-means clustering with K = 2. Find out the miss classification rate for digits “2” and “6” respectively, and compare with GMM. Which one achieves the better performance?1. Basic optimization. (50 points.) The background of logistic regression will be discussed in the next lecture. Here, we just focus on finding out the property of the optimization problem, related to training a logistic regression. Consider a simplified logistic regression problem. Given n training samples (xi , yi), i = 1, . . . , n. The data xi ∈ R, and yi ∈ {0, 1}. To train/fit a logistic regression model for classification, we solve the following optimization problem, where θ ∈ R is a parameter we aim to find: max θ `(θ), (1) where the log-likelhood function `(θ) = Xn i=1 {− log(1 + exp{−θxi}) + (yi − 1)θxi} . (a) (15 points) Derive the gradient of the cost function `(θ) in (1) and write a pseudo-code for performing gradient descent to find the optimizer θ ∗ . This is essentially what the training procedure does. pseudo-code means you will write down the steps of the algorithm, not necessarily any specific programming language. (b) (15 points) Present a stochastic gradient descent algorithm to solve the training of logistic regression problem (1). (c) (20 points) We will show that the training problem in basic logistic regression problem is concave. Derive the Hessian matrix of `(θ) and based on this, show the training problem (1) is concave. Explain why the problem can be solved efficiently and gradient descent will achieve a unique global optimizer, as we discussed in class. 2. Comparing Bayes, logistic, and KNN classifiers. (50 points) In lectures we learn three different classifiers. This question is to implement and compare them. We are suggest use Scikit-learn, which is a commonly-used and powerful Python library with various machine learning tools. But you can also use other similar library in other languages of your choice to perform the tasks. 1 Part One (Divorce classification/prediction). (30 points) This dataset is about participants who completed the personal information form and a divorce predictors scale. The data is a modified version of the publicly available at https://archive.ics.uci.edu/ml/datasets/ Divorce+Predictors+data+set (by injecting noise so you will not replicate the results on uci website). There are 170 participants and 54 attributes (or predictor variables) that are all real-valued. The dataset q3.csv. The last column of the CSV file is label y (1 means “divorce”, 0 means “no divorce”). Each column is for one feature (predictor variable), and each row is a sample (participant). A detailed explanation for each feature (predictor variable) can be found at the website link above. Our goal is to build a classifier using training data, such that given a test sample, we can classify (or essentially predict) whether its label is 0 (“no divorce”) or 1 (“divorce”). Build three classifiers using (Naive Bayes, Logistic Regression, KNN). Use the first 80% data for training and the remaining 20% for testing. If you use scikit-learn you can use train test split to split the dataset. (a) Report testing accuracy for each of the three classifiers. Comment on their performance: which performs the best and make a guess why they perform the best in this setting. (b) Use the first two features to train three new classifiers. Plot the data points and decision boundary of each classifier. Comment on the difference between the decision boundary for the three classifiers. Please clearly represent the data points with different labels using different colors. Part Two (Handwritten digits classification). (20 points) Repeat the above using the MNIST Data in our Homework 3. Here, give “digit” 6 label y = 1, and give “digit” 2 label y = 0. All the pixels in each image will be the feature (predictor variables) for that sample (i.e., image). Our goal is to build classifier to such that given a new test sample, we can tell is it a 2 or a 6. Using the first 80% of the samples for training and remaining 20% for testing. Report the classification accuracy on testing data, for each of the three classifiers. Comment on their performance: which performs the best and make a guess why they perform the best in this setting. 3. Deriving M-step in EM for GMM. (Bonus question: 10 points) Consider the Q function in EM for GMM: Q(θ|θk) = Xn i=1 X C c=1 pi,c log πc + Xn i=1 X C c=1 pi,c log φ(xi |µc, Σc) where θ = {πc, µc, Σc}c=1,…,C , and pi,c ∝ π (k) c φ(xi |µ (k) c , Σ (k) c ), i = 1, . . . , n, c = 1, . . . , C depend on the parameters from the previous iteration. Here φ(x|µ, Σ) denotes the pdf of a multi-variate Gaussian with mean vector µ and covariance matrix Σ. Solve πc, µc and Σc that maximize the Q(θ|θk) function. In other words, we want to find θk+1 := arg max θ Q(θ|θk) subject to X C c=1 πc = 1. Show that the solution θk+1 is given by the following expression π (k+1) c = Pn i=1 pi,c n µ (k+1) c = Pn i=1 P pi,cxi c i=1 pi,c 2 Σ (k+1) c = Pn i=1 pi,c(xi − µ (k) c )(xi − µ (k) c ) T Pn i=1 pi,c (Hint: Consider the Lagrangian function for solving this constrained optimization problem. You only need to introduce one Lagrangian multiplier because you only have one constraint. Then solve it from there.) 4. Naive Bayes for spam filtering. (Bonus question: 20 points) In this problem we will use the Naive Bayes algorithm to fit a spam filter by hand. This will enhance your understanding to Bayes classifier and build intuition. Spam filters are used in all email services to classify received emails as “Spam” or “Not Spam”. A simple approach involves maintaining a vocabulary of words that commonly occur in “Spam” emails and classifying an email as “Spam” if the number of words from the dictionary that are present in the email is over a certain threshold. We are given the vocabulary consists of 15 words V = {secret, offer, low, price, valued, customer, today, dollar, million, sports, is, for, play, healthy, pizza}. We will use Vi to represent the ith word in V . As our training dataset, we are also given 3 example spam messages, • million dollar offer • secret offer today • secret is secret and 4 example non-spam messages • low price for valued customer • play secret sports today • sports is healthy • low price pizza Recall that the Naive Bayes classifier assumes the probability of an input x = [x1, x2, . . . , xn] T depends on its class y. In our case the input vector x corresponding to each message has length n = 15 equal to the number of words in the vocabulary V , where each entry xi is equal to the number of times word Vi occurs in x. (a) Calculate P(y = 0) and P(y = 1) from the training data, where y = 0 corresponds to spam messages, and y = 1 corresponds to non-spam messages. (b) List the feature vector x for each spam and non-spam message. (c) In the Naive Bayes model, the likelihood of a sentence with feature vector x given a class c is P(x|y = c) = Yn k=1 θ xk c,k where θc,k ∈ (0, 1) is the weight of word k in class c, which satisfies Pn k=1 θc,k = 1, ∀c. Calculate the maximum likelihood estimates of θ0,1, θ0,7, θ1,1, θ1,15 by maximizing P(x|y = c) with respect to θc,k and given data. (Hint: Consider the Lagrangian function for solving this constrained optimization problem. You only need to introduce one Lagrangian multiplier because you only have one constraint. Consider log-likelihood (i.e., taking log of the cost function). Then solve it from there.) (d) Given a new message “today is secret”, decide whether it is spam or not spam, based on the Naive Bayes classifier, learned from the above data.
1 Lab 3.6 from ISLR Go through the lab exercise in Section 3.6 of ISLR. The book is written to use the programming language R for these exercises. If you like, you can use R to complete these exercises (in this case, I highly recommend using the IDE rstudio). Alternatively, since we will later use python for other assignments, I have provided a corresponding set of python commands that you can use. These are given in the form of a Jupyter notebook. I recommend that you open this notebook for reference (run the command jupyter notebook in a terminal from the directory containing the notebook, and open the notebook from there), but type all the commands into a new notebook of your own, to make sure that you pay attention to what each cell is doing. If you use python, you will need the various packages listed at the top of the notebook (these packages are extremely common at companies, becoming proficient with pandas, numpy, sklearn is highly recommended). The easiest way to get these is to install the anaconda package from here: https://www.anaconda.com/products/individual. You do not need to turn in your code. The goal of this exercise is to practice your data skills. Questions 1. Compare the plots of the residuals vs. the fitted values for the regression medv lstat + np.square(lstat) and the regression using only lstat as a predictor. What’s the qualitative difference? 2. Does the fifth-order polynomial from your python regression correspond to the one from the ISLR book? If not, why might this occur? 2 EDA with the Spam Filtering Data Set The csv file spam.csv contains a data set for emails that were categorized as spam or not spam. The documentation for this data set is in the file spam-info.pdf. 1. Look at the documentation. What is the variable of interest, i.e. the dependent variable? 1 IEOR E4525 Christian Kroer Assignment 1 Due: Oct 1st, at 11:59pm 2. For each of the independent variables, report something about it. Specifically, you should report on each variable’s relationship with the response, i.e dependent, variable. Pay special attention to variable type (binary, ordinal, real) when doing this. Your comments should contain at least some tables and graphs. 3. Investigate the variable ’spampct’. (a) How many missing values does it have? (b) Compare graphically the distribution for time.of.day for the cases where spampct is missing against the distribution of time.of.day when spampct is present. Do you see any differences? (c) Plot a scatter plot of time of day vs. spampct. How many unique points (x,y coordinates) are plotted? Explain a technique you might use to deal with the overplotting. 3 Exploring the Relationship Between Overfitting and Noise Do exercise 13 from Section 3.7 of ISLR. The example codes are for R, but below I provide a table of translations to python. You will need to use the numpy documentation to look up how to use the various commands. Make sure you look up the documentation for your version of numpy. R command python command set.seed(1) np.random.seed(1) rnorm() np.random.randn() rnorm() np.random.randn() 4 Naive Bayes and Spam Filtering 1. Use the spam data from Question 2 and Naive Bayes to build a classifier that distinguishes spam from non-spam. You can use Naive Bayes from sklearn for this. Your code should split the data into training and test sets and then estimate the generalization error of your classifier. 2. Randomly assign 80% of your data to the training set, 20% to the test set and now estimate the test error, Etest, of your classifier. Repeat this 10 times. How much variability do you see in Etest? What conclusions can you draw from this? 3. There are two types of error that a spam classifier can make. Should these errors be treated equally when constructing a classifier. Can we adapt our naive Bayes classifier to reflect this? 5 Least Squares Linear Regression is MLE for Gaussian noise Consider the linear regression model Y = XT β + , where β, X ∈ R d , are fixed, and the error ∼ N (0, σ2 ) is distributed according to a Gaussian distribution. In class we saw how to derive the least squares estimator. In this exercise, you just must prove that the least squares estimator is also the maximum-likelihood estimator, given that the error is Gaussian. 6 k Nearest Neighbors and the Curse of Dimensionality Solve exercise 4 from Section 4.7 of ISLR.1 ISLR Classification Lab Complete the lab from Section 4.6 of ISLR. Feel free to utilize the provided worked jupyter notebook as inspiration. You do not need to submit your code for this question, it is purely for you to practice with. 2 Classification Models for Stock Market Data Solve exercise 10 from Section 4.7 of ISLR. The Weekly dataset can be found in the Data folder. A description of it can be found here on page 14. 3 Reduced-Rank LDA Let B and W be positive definite matrices and consider the following problem of maximizing the Rayleigh quotient: max a a T Ba a T W a (1) 1. Use the method of Lagrange multipliers to solve this problem. In particular, show that the optimal solution a ∗ is an eigenvector of a certain matrix related to B and W. What is this matrix, and which eigenvector does a ∗ correspond to? Hint: Use the scale invariance of the Rayleigh quotient to rewrite the unconstrained maximization as a constrained maximization problem where B appears in the objective, and W appears in the constraint. 1 IEOR E4525 Christian Kroer Assignment 2 Due: Oct 15th, at 11:59pm 2. By identifying B and W with the between-class and within-class covariance matrices, we can interpret the problem in (1) as the problem of finding the linear combination a ∗T x so as to maximize the between-class variance relative to the within-class variance. Show that a ∗T x is the first discriminant variable. Hint: First note that W = Σ from the lecture slides, and that B∗ = D−1/2U T BUD−1/2 4 Logistic Regression 1. Show that binary classification using logistic regression yields a linear classifier. Consider a naive Bayes classifier for a binary classification problem where all the class-conditional distributions are assumed to be Gaussian with the variance of each feature Xj being equal across the two classes. That is we assume (Xj |Y = k) ∼ N(µjk, σ2 j ) 2. Show that the decision boundary is a linear function of X = (X1, …, Xd) and hence that it has the same parametric form as the decision boundary given by logistic regression. 3. Does the result of part (2) imply that in this case, Gaussian naive Bayes and logistic regression will find the same decision boundary? Justify your answer. 4. If indeed the class conditional distributions are Gaussian with (Xj |Y = k) ∼ N(µjk, σ2 j ) and the assumptions of naive Bayes are true, which classifier do you think will be “better”: the naive Bayes classifier of part (2) or logistic regression? Justify your answer. 5 Bootstrap Probabilities Solve exercise 2 from Section 5.4 of ISLR.1 Bootstrap Description Suppose that we have a dataset (xi , yi) n i=1, and we fix a value ¯x. Further suppose that we are going to build a predictor for the response ¯y associated to ¯x using some statistical learning method. Describe how we might estimate the standard deviation of our prediction. You must explicitly define every variable and equation. 2 Bootstrap for Estimating Standard Errors of Logistic Regression Coefficients Solve exercise 6 from Chapter 5 of ISLR, in python. You can use either the statsmodels or sklearn package for this. I recommend using statsmodels, since it has better support for this kind of statistical analysis (note also that sklearn regularizes by default, so you must turn that off if you use sklearn). For statsmodels, after building a model m, you can use m.summary() to get the standard errors of the coefficients. For 6.b, write a function boot fn that works as described in ISLR. Instead of the R library function boot, you must write your own: write a function boot(data, fn, R) where data is a pandas dataframe, fn is a function that computes a statistic, and R is the number of replicates. You can use resample from sklearn to generate individual bootstrap samples. 3 Cross-Validation on Simulated Data Solve exercise 8 from Chapter 5 of ISLR, in python. 4 Ridge Regression Effect of λ Solve exercise 4 in Chapter 6 of ISLR 1 IEOR E4525 Christian Kroer Assignment 3 Due: Oct 29th, at 11:59pm 5 Comparing Lasso, Ridge, and Least Squares Solve exercise 9 from Chapter 6 of ISLR. You only need to complete questions (a),(b),(c),(d), and (g). For question (g), you only need to compare the three approaches from (b), (c), and (d). You can use scikitlearn LinearRegression, Ridge, Lasso. If you prefer statsmodels, then you can use regularized for lasso and ridge.1 SVMs 1.1 Scaling the Inputs True or false: in training an SVM it is generally a good idea to scale all input variables so that, for example, they all lie in some fixed interval or so that they all have the same mean, µ, and variance, σ 2 , e.g (µ, σ2 ) = (0, 1). Justify your answer. 1.2 Classifying Tumors 1. Load the breast cancer dataset using sklearn.datasets. Construct an SVM classifier for this data. You should randomly assign t% of your data to the training set and the remainder of your data to the test set. Then use cross-validation on your training set to build your classifier. You can take t = 70% initially. 2. Repeat part (1) N = 50 times to get N samples of the performance of the trained classifier on the test set. (Note that each of the N samples will have different training and test sets.) Compute the mean and standard deviation of the test- set-performance. 3. Repeat part (b) for values of t = 50%, 55%, . . . , 95% and plot the mean test-set performance together with 95% confidence intervals for this performance against t. What conclusions can you draw? 1.3 SVMs and Cross-Validation Suppose you have successfully trained an SVM with 10,000 training points and a Gaussian kernel where the values of C and σ were selected via cross-validation. Recall that the Gaussian kernel has the form K(x, x0 ) = exp − kx − x 0k 2 2σ 2 You are then given an additional 40,000 training points and so you wish to retrain your SVM using the entire 50,000 training points that you now have. However, you wish to avoid the heavy computational 1 IEOR E4525 Christian Kroer Assignment 4 Due: Nov 19th, at 11:59pm expense associated with repeating the cross-validation exercise that you previously used to pick C and σ. Instead, you simply use the C and σ that you found using the first 10,000 training points, and then retrain your SVM using those hyperparameters, but on the new set of 50,000 data points. Do you see any potentially major problem with this? If so, what is it? 2 PyTorch Practice 1. Install PyTorch. I recommend using anaconda, in which case you can do it with the conda package manager: conda install pytorch torchvision -c pytorch 2. Do the PyTorch 60-minute blitz tutorial 3. Your jupyter notebook should have a section where you run each command from the PyTorch 60-minute blitz (this will only be lightly graded). You do not need to run the GPU commands. 4. Create a neural network with two hidden layers (the notebook shows how to create one with one hidden layer), both layers should be ReLU layers (you may simply take the one from the notebook and add a second layer with 256 output features, but feel free to get more creative) 5. Try SGD, Adam, and at least one other optimization algorithm from torch.optim. Try at least 3 different stepsizes for each algorithm (for Adam you should also try the default stepsize). Report on your experience with finding a reasonable stepsize for each algorithm (e.g. how sensitive is each algorithm to stepsize), and how the algorithms compare on loss minimization, training accuracy, and test accuracy. 6. If you pick the best setup from all your experiments above, based on either loss or training accuracy performance, do you get the best algorithm on test accuracy? 3 Function Approximation 1. Consider a ReLU network with a single hidden layer, with W(1) ∈ R 2×2 , x, b(1) ∈ R 2 and W(2) ∈ R 1×2 , b(2), y ∈ R: • h 1 = σ(W(1)x + b (1)) • yˆ = W(1)h 1 + b (2) Show that this network is a piecewise linear function. Specify the set of pieces and the value on each piece. 2. Consider a continuous piecewise-linear function f(x) = x + 3 if x < 5 2x − 2 if 5 ≤ x < 10 −1x + 28 if 10 ≤ x Show how to represent it with a ReLU network that uses a single hidden layer.
Using what you’ve learned, we will create a dictionary. Later, we will implement Hangman using the dictionary.Things you need to know for this project: Arrays, Functions, Global-Variables, String-Manipulation
This project specification is subject to change at any time for clarification. For this project you will be writing several Java classes to implement a programming language scanner and a CSV parser. The classes developed in this project will be used in subsequent projects. A Token class has been provided for you and an interface the Scanner will rely on for input has also been provided. The Scanner class must take in a PeekableCharacterStream and a List of keywords and must produce a stream of Tokens that are queried from either peekNextToken or getNextToken methods. The rules for the tokens defined below. You will also develop the CSVParser class that will parse CSV files into Maps, one for each row. Your first task is to develop a class that implements the PeekableCharacterStream interface for a FileInputStream. public interface PeekableCharacterStream{ // Returns true if more characters are available, false otherwise public boolean moreAvailable(); // Returns the next character that would be returned without consuming // the character. If no more characters are available -1 is returned. public int peekNextChar(); // Returns the character ahead in the stream without consuming the // the character. peekAheadChar(0) returns the same character as // peekNextChar(). If no more characters are available at that position // -1 is returned. public int peekAheadChar(int ahead); // Returns the next character and consumes it. If no more characters are // available -1 is returned. public int getNextChar(); // Closes the stream. public void close(); } Your second task is to develop the Scanner class that will rely on the PeekableCharacterStream interface and the Token class. The Scanner must have the following minimal interface. public class Scanner{ // Constructor that takes in a stream and a list of keywords. public Scanner(PeekableCharacterStream stream, List keywordlist); // Returns the next token without consuming it. If no more tokens are // available a None token is returned. public Token peekNextToken(); // Returns the next token and consumes it. If no more tokens are // available a None token is returned. ECS140A FQ20 October 7, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 1 2 of 3 public Token getNextToken(); } Token Rules: Identifier := ( _ | Alpha ) { ( _ | Digit | Alpha ) } Operator := ( | , | ) | { | } | = | == | < | > | = | != | + | – | * | / | ; IntConstant := [ – ] Digit { Digit } FloatConstant := [ – ] Digit { Digit } [ . Digit { Digit } ] StringConstant := ” { ( CharacterLiteral | EscapedCharacter ) } ” Digit := 0 – 9 Alpha := A – Z | a – z WhiteSpace := Space | Tab | CarriageReturn | NewLine CharacterLiteral := Space – ! | # – [ | ] – ~ EscapedCharacter := b | | r | t | \ | ’ | ” Additional Tokenizing Rules: • Keywords are identifiers that are in the List provided to the Scanner. • Whitespace must be skipped during the tokenizing. • A negative sign immediately preceding an integer or float constant must be tokenized as an operator if the previous token was a constant or identifier. For example, “A -5” must be tokenized as Identifier “A”, Operator “-”, and IntConstant “5”, not as Identifier “A”, and IntConstant “-5”. • If an underscore or Alpha character immediately follows a constant, the token is considered Invalid. All Alpha, Digit, and underscore characters will be part of the Invalid token. • If an invalid character is in a string constant, the characters are consumed until the next non-escaped ” is reached or the end of stream is reached. The token type is considered Invalid. • Any invalid character beginning a token will be considered an Invalid token by itself. For example, “@#4” must be tokenized as Invalid “@”, Invalid “#”, and IntConstant “4”. Implementation Requirements: • You may use java.io.FileInputStream. • You may use java.util.Set, java.util.HashSet, java.util.Arrays and similar containers. • You may not use java.util.regex or similar packages. • You may not use java.util.StringTokenizer or similar library classes. Your final task is to develop the CSVParser class that will rely on the PeekableCharacterStream interface. The CSVParser must have the following minimal interface. public class CSVParser{ // Constructor that takes in a stream. public CSVParser(PeekableCharacterStream stream); // Returns the next row without consuming it. If no more rows are // available null is returned. public Map peekNextRow(); // Returns the next row and consumes it. If no more rows are // available null is returned. public Map getNextRow(); } ECS140A FQ20 October 7, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 1 3 of 3 CSV Format Rules • CSV files must have a header row, and no column in the header may be repeated or empty. • Each row is terminated by a newline character. • Each column is terminated by a comma character. • Any whitespace (space, tab, carriage return, or newline) character that is part of a column must be a double quoted ” column. The escape sequence for a double quote in a double quoted column is two double quotes in a row. • Any empty columns or missing columns will return a value of null for the corresponding value in the returned Map. • Valid CSV files are not allowed to have more columns in a data row than the header row but may have fewer. Your Scanner and CSVParser classes must have a main function that takes in a filename as an argument and outputs the contents similar to the examples in /home/cjnitta/ecs140a/proj1 on the CSIF. You can run the provided solutions by running the shell scripts Scanner.sh or CSVParser.sh and providing a filename to open. Your code will be tested on the CSIF and is expected to compile and run on the CSIF. You must submit the source file(s), a Makefile, and README.txt file, in a tgz archive. Do a make clean prior to zipping up your files so the size will be smaller. You will want to be in the parent directory of the project directory when creating the tgz archive. You can tar gzip a directory with the command: tar -zcvf archive-name.tgz directory-name You should avoid using existing source code as a primer that is currently available on the Internet. You must specify in your readme file any sources of code that you have viewed to help you complete this project. You must also provide the URL any code sources in comments of your source code. All class projects will be submitted to MOSS to determine if students have excessively collaborated. Excessive collaboration, or failure to list external code sources will result in the matter being referred to Student Judicial Affairs.Overview This project specification is subject to change at any time for clarification. For this project you will be writing a Java program to parse language X. Language X is similar to the C language, but is simpler and slightly different. For this program you will be reading in an X language file and converting it to a syntax highlighted XHTML version. A CSV file will describe the font and color settings for the XHTML output file. The CSV file will have a .csv extension and the X language file will have a .x extension. You will be creating several classes that will solve the various portions of the problem, and reusing the classes developed in Project 1. X Programming Language The following EBNF describes the X language. The Non-terminals on the right are identified by italics. Literal values are specified in bold. The operators not in bold or italics describe the options of the EBNF. These operators are {} for repetition, [] for optional, () for grouping, and | for or. EBNF Rules: Program := {Declaration} MainDeclaration {FunctionDefinition} Declaration := DeclarationType (VariableDeclaration | FunctionDeclaration) MainDeclaration := void main ( ) Block FunctionDefinition := DeclarationType ParameterBlock Block DeclarationType := DataType Identifier VariableDeclaration := [= Constant] ; FunctionDeclaration := ParameterBlock ; Block := { {Declaration} [(Statement {Statement} {FunctionDefinition})] } ParameterBlock := ( [Parameter {, Parameter}] ) DataType := IntegerType | FloatType Constant := IntConstant | FloatConstant Statement := Assignment | WhileLoop | IfStatement | ReturnStatement | (Expression 😉 Parameter := DataType Identifier IntegerType := [unsigned] ( char | short | int | long ) FloatType := float | double Assignment := Identifier = {Identifier =} Expression ; WhileLoop := while ( Expression ) Block IfStatement := if ( Expression ) Block ReturnStatement := return Expression ; Expression := SimpleExpression [ RelationOperator SimpleExpression ] SimpleExpression := Term { AddOperator Term } Term := Factor { MultOperator Factor } Factor := ( ( Expression ) ) | Constant | (Identifier [ ( [ Expression {, Expression}] ) ] ) RelationOperator := ( == ) | < | > | ( = ) | ( != ) AddOperator := + | – MultOperator := * | / ECS140A FQ20 October 22, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 2 2 of 4 Token Rules: Identifier := ( _ | Alpha ) { ( _ | Digit | Alpha ) } IntConstant := [ – ] Digit { Digit } FloatConstant := [ – ] Digit { Digit } [ . Digit { Digit } ] Digit := 0 – 9 Alpha := A – Z | a – z The X programming language allows for nested function declarations. Variables and functions must be declared before they are referenced or called. Identifiers have the same scoping rules as C. An identifier may only be declared once per block but may be declared more than once per file. Output XHTML The output XHTML will output the X language file with fonts and colors specified by the CSV file. All token types that are not specified in the CSV file will be assumed to be the same as the browser defaults. The token classes are IntConstants, FloatConstants, Keywords, Operators, Variables, and Functions. The keywords of language X are: unsigned, char, short, int, long, float, double, while, if, return, void, and main. The operators of language X are: (, ,, ), {, }, =, ==, , =, !=, +, -, *, /, and ;. All Variable and Function references must link back to the declaration of the Variable or Function. Each nested block should be indented another level, the amount of indention will 4 spaces. Input CSV File The input CSV description file describes the formatting to be used for the XHTML file. The default will be specified by the DEFAULT ELEMENT_TYPE. The attribute types are: BACKGROUND, FOREGROUND, and STYLE. The DEFAULT BACKGROUND, FOREGROUND, and STYLE attributes set the global background color, foreground color and font face for the output XHTML. The indention is 4 spaces for each nested block. The FOREGROUND and STYLE may be specified for the token types: FUNCTION, VARIABLE, FLOAT_CONSTANT, INT_CONSTANT, OPERATOR, and KEYWORD. Below is an example of CSV formatting file. ELEMENT_TYPE,BACKGROUND,FOREGROUND,STYLE,FONT DEFAULT,navy,yellow,,”Courier New” FUNCTION,,orange,, VARIABLE,,yellow,, FLOAT_CONSTANT,,aqua,b, INT_CONSTANT,,aqua,b, OPERATOR,,white,b, KEYWORD,,white,b, ECS140A FQ20 October 22, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 2 3 of 4 Suggested Approach 1. Write a class that will open and read the .csv file as specified on the command line. You should use your CSVParser class from Project 1. Make sure you can determine the formatting for each element type. 2. Use your Scanner class from Project 1 to write a recursive descent parser class that will parse the .x file by making repeated calls to the getNextToken() method. (You may need to use peekNextToken() to look ahead in some instances.) This class should implement one method per EBNF rule. It should also be able to validate that the .x file does conform to the grammar specified. If an error is found in the .x file, the line number of the error and the grammar rule where the error was found should be printed with an appropriate message. A working example is available on the CSIF to test against. 3. Using the classes written in steps 1 and 2, write a program that reads in a .x source file and a .csv file and outputs the XHTML formatting. The colors and styles of tokens in the XHTML file should match the specification in the .csv file. In addition, references to variables and functions in the source file should be hyperlinks back to their respective declarations. Do not worry about matching overloaded functions with appropriate call, just link back to the function that would be in current scope as if it were a variable. The XHTML should be a valid HTML that can be loaded into any web browser and viewed. You will probably need to begin your XHTML with: You may also need to have attributes in your html element. So it should look like:Examples of valid input files .x and .csv files are in /home/cjnitta/ecs140a/proj2 on the CSIF. Scripts that run the example classes can be found there too. All scripts take one argument except for XFormatter.sh which takes two arguments, the format CSV file and then the X file to format. Your code will be tested on the CSIF and is expected to compile and run on the CSIF. You must submit the source file(s), a Makefile, and README.txt file, in a tgz archive. Do a make clean prior to zipping up your files so the size will be smaller. You will want to be in the parent directory of the project directory when creating the tgz archive. You can tar gzip a directory with the command: tar -zcvf archive-name.tgz directory-name You should avoid using existing source code as a primer that is currently available on the Internet. You must specify in your readme file any sources of code that you have viewed to help you complete this project. You must also provide the URL any code sources in comments of your source code. All class projects will be submitted to MOSS to determine if students have excessively collaborated. Excessive collaboration, or failure to list external code sources will result in the matter being referred to Student Judicial Affairs. ECS140A FQ20 October 22, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 2 4 of 4 Helpful Hints • The use of peekNextToken()will be helpful in some instances where it isn’t clear what the next rule should be parsed. The language is designed so that only a single look ahead is necessary. • Obviously, a hash table of some form should be used for the symbol table, but there are two clear options for handling of the scope blocks. Either a stack (or list) of hash tables can be used for the blocks, or a hash table list values could be used. Using a stack of hash tables makes the cleaning up of the scope easy when the block ends but requires traversing through the outer scope hash tables if the symbol is not found in the inner table. Using a hash table of list values makes finding the correct symbol easy but complicates the scope cleanup when the block ends. • In order to implement the X formatter there are two general approaches. One is to use an observer pattern https://en.wikipedia.org/wiki/Observer_pattern that observes the parsing of your class developed in step 2. You can create an observer interface that has methods to notify of beginning/ending of parsing and rules, as well as parsing of a token and if there is an error. The formatter would then just need to implement the observer interface. Another option is to subclass the X parser and to call the equivalent super method after any pre work is completed, and any post work can be done after the method returns. There are advantages/disadvantages to both approaches.Overview This project specification is subject to change at any time for clarification. For this programming assignment you will be programming in LISP. This entire project must be purely functional; you are not allowed to use setq, loop or similar constructs. Points may be docked for use of such constructs. Fibonacci Sequence Less Than N Write a function fibo-lt to return a list as the Fibonacci sequence such that all numbers in the sequence are less than n (NOT the first n numbers of the sequence). For example: > (fibo-lt ’50) (0 1 1 2 3 5 8 13 21 34) Your function must run in sublinear time and may not hardcode results except for those of the beginning of the sequence such as ‘(), ‘(0) and ‘(0 1). Pattern Matching Program Before we start building the pattern matching function, let us first build a set of routines that will allow us to represent facts, called assertions. For instance, we can define the following assertions: (this is an assertion) (color apple red) (supports table block1) Patterns are like assertions, except that they may contain certain special atoms not allowed in assertions, the single ! character, for instance, or may have strings containing the * character. (color ! red) (su*ts table block1) Write a function match, which compares a pattern and an assertion. When a pattern containing no special atoms is compared to an assertion, the two match only if they are exactly the same, with each corresponding position occupied by the same atom. > (match ‘(color apple red) ‘(color apple red)) T > (match ‘(color apple red) ‘(color apple green)) NIL ECS140A FQ20 September 28, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 3 2 of 4 The special symbol ‘!’ expands the capability of match by matching zero or more atoms. > (match ‘(! table !) ‘(this table supports a block)) T Here, the first symbol ‘!’ matches this, table matches table, and second symbol ‘!’ matches supports a block. > (match ‘(this table !) ‘(this table supports a block)) T > (match ‘(! brown) ‘(green red brown yellow)) NIL In the last example, the special symbol ‘!’ matches ‘green red’. However, the match fails because yellow occurs in the assertion after brown, whereas it does not occur in the assertion. However, the following example succeeds: > (match ‘(! brown) ‘(green red brown brown)) T In this example, ‘!’ matches list (green red brown), whereas the brown matches the last element. > (match ‘(red green ! blue) ‘(red green blue)) T In this example, the ‘!’ matches the empty list. The * character matches zero or more characters inside a string. > (match ‘(red gr*n blue) ‘(red green blue)) T > (match ‘(t* table is *n) ‘(this table is blue)) NIL In the first example the *, matches ee. In the second example the first * matches his, but the second one fails to match because of the n. The lone * will match any single atom. > (match ‘(color apple *) ‘(color apple red)) T > (match ‘(color * red) ‘(color apple red)) T > (match ‘(color * red) ‘(color apple green)) NIL In the last example, color * red and (color apple green) do not match because red and green do not match. Erasure Code Reconstruction Erasure codes are used when a value can be detected as incorrect (or erased). They are commonly used in storage (like in RAID-5) as well as in communication. Write a function ECS140A FQ20 September 28, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 3 3 of 4 to return a reconstructed message. Each message will be a set of words each with a parity bit followed by a parity word. Each word is represented as a list of 0, 1 or NIL values. The parity bit will be the final value in each word, the parity word will be the final word in the list of words. An example output should be something like: > (parity-correction ‘((0 1 1 NIL 0) (0 0 0 NIL 1) (NIL 1 1 1 0) (1 0 1 NIL 0) (0 NIL 1 0 1))) (T ((0 1 1 0 0) (0 0 0 1 1) (1 1 1 1 0) (1 0 1 0 0) (0 0 1 0 1))) This would be equivalent to the following message. D0 D1 D2 D3 P W0 0 1 1 0 0 W1 0 0 0 1 1 W2 1 1 1 1 0 W3 1 0 1 0 0 WP 0 0 1 0 1 If the message is incorrect or can’t be solved NIL should be returned with the original message. For example: > (parity-correction ‘((0 1 1 NIL 0) (0 0 0 NIL 1) (NIL 1 1 1 0) (1 0 1 NIL 0) (0 NIL 1 0 0))) (NIL ((0 1 1 NIL 0) (0 0 0 NIL 1) (NIL 1 1 1 0) (1 0 1 NIL 0) (0 NIL 1 0 0))) The parity-correction function should return results for modest size messages within a second or two. Submissions that take significant amount of time may lose points. Scripts of working solutions are in /home/cjnitta/ecs140a/proj3 on the CSIF. Passing the arguments may require care on bash as a single quote may need to be passed in to the test script. You must submit the source file(s), and README.txt file, in a tgz archive. Do a make clean prior to zipping up your files so the size will be smaller. You will want to be in the parent directory of the project directory when creating the tgz archive. You can tar gzip a directory with the command: tar -zcvf archive-name.tgz directory-name You should avoid using existing source code as a primer that is currently available on the Internet. You must specify in your readme file any sources of code that you have viewed to help you complete this project. You must also provide the URL any code sources in comments of your source code. All class projects will be submitted to MOSS to determine if students have excessively collaborated. Excessive collaboration, or failure to list external code sources will result in the matter being referred to Student Judicial Affairs. ECS140A FQ20 September 28, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 3 4 of 4 Helpful Hints • The command to use Common LISP is clisp. • Appendix A of LISPcraft summarizes LISP’s built-in functions. Each function is explained briefly. You will find this a very useful reference as you write and debug your programs. Also, you can get help about clisp by typing: man clisp • You may define additional helper functions that your main functions use. Be sure, though, to name the main functions as specified since the test program uses those names. • If you place a init.lsp file in the directory in which you execute LISP (or your home directory), LISP will load that file automatically when it starts execution. Such a file is useful to define your own environment. For instance, you will probably want to put the following command in that fiile: (setq *print-case* :downcase) • When developing your program, you might find it easier to test your functions first interactively before using the test program. You might find trace, step, print, etc. functions useful in debugging your functions. • A few points to help the novice LISP programmer: • Watch your use of (, ), “, and ‘. Be sure to quote things that need to be quoted. • To see how lisp reads your function, use pretty printing. For example, > (pprint (symbol-function ‘foo)) will print out the definition of function foo, using indentation to show nesting. This is useful to locate logically incorrect nesting due to, e.g., wrong parenthesizing. • If you cause an error, Common Lisp places you into a mode in which debugging can be performed (LISPcraft section 11.2). To exit any level, except the top level, type :q. To exit the top level, type > (bye)Overview This project specification is subject to change at any time for clarification. For this project you will be writing Datalog queries and will be developing a non-recursive Datalog interpreter. The EBNF rules that will be used for the dialect of Datalog are in the following section. For this project Datalog queries will be constructed of a set of rules that will each have a rule head. Each rule head specifies the rule name and a set of variables. Each rule that does not have a rule body is considered a fact (also known as external rule); these facts will be backed by CSV data files that have the tuples that specify when the rule is true. Any other tuples do not satisfy the fact rules. Rules that have a rule body are to be calculated and attempt to find the values that will satisfy all of the subgoals. If multiple rules exist with the same name, then rule invocations of that name are considered to be a union of the multiple rules. The goal of the Datalog interpreter is to find all tuples that satisfy all of the rules specified, with the base values coming from the fact rules. Non-Recursive Datalog Programming Language The following EBNF describes the non-recursive Datalog language. The Non-terminals on the right are identified by italics. Literal values are specified in bold. The operators not in bold or italics describe the options of the EBNF. These operators are {} for repetition, [] for optional, () for grouping, and | for or. EBNF Rules: Query := Rule { Rule } Rule := [ RuleHead [ := RuleBody ] ] NewLine RuleHead := RuleName ( HeadVariableList ) RuleBody := SubGoal { AND SubGoal } RuleName := Identifier HeadVariableList := Identifier { , Identifier } SubGoal := RuleInvocation | NegatedRuleInvocation | EqualityRelation RuleInvocation := RuleName ( BodyVariableList ) NegatedRuleInvocation := NOT RuleInvocation EqualityRelation := InequalityRelation { EQOperator InequalityRelation } BodyVariableList := InvocationVariable { , InvocationVariable } InequalityRelation := Term { IEQOperator Term } EQOperator := != | = InvocationVariable := Identifier | EmptyIdentifier Term := SimpleTerm { AddOperator SimpleTerm } IEQOperator := < | > | = SimpleTerm := UnaryExpression { MultOperator UnaryExpression } AddOperator := + | – UnaryExpression := ( UnaryOperator UnaryExpression ) | PrimaryExpression MultOperator := * | / | % UnaryOperator := ! | – | + PrimaryExpression:= ( EqualityRelation ) | Constant | Identifier Constant := IntConstant | FloatConstant | StringConstant ECS140A FQ20 December 10, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 4 2 of 8 Token Rules: Identifier := Alpha { ( Digit | Alpha ) } EmptyIdentifier := _ IntConstant := Digit { Digit } FloatConstant := Digit { Digit } [ . Digit { Digit } ] StringConstant := ” { ( CharacterLiteral | EscapedCharacter ) } ” Comment := # { ( WhiteSpace | Printable ) } NewLine Digit := 0 – 9 Alpha := A – Z | a – z WhiteSpace := Tab | CarriageReturn | NewLine | Space Printable := Space – ~ CharacterLiteral := Space – ! | # – [ | ] – ~ EscapedCharacter := b | | r | t | \ | ’ | ” Semantics: There are several requirements for the non-recursive Datalog queries that cannot be expressed in the EBNF. The following must be true of all valid queries: • All subgoals must be defined prior to their use • All repeated rule names must be defined contiguously • All repeated rule names must have the same number of variables • Any variable in the rule head or rule body must appear in a non-negated rule invocation • A rule invocation may not appear in its own rule definition (i.e. recursive definition) • Fact rules may only appear once • Every rule invocation must provide the same number of variables as its definition CSV Data Files The facts can be stored in a CSV files. The name of the file without .csv extension will be the name of the facts loaded into the environment. Each column is a different attribute, with the header row containing the attribute name. Four data types are allowed in the CSV files: string, float, integer, and boolean. Rows two and on, each have one tuple in them. The following is an example of fact R that would be in the file R.csv: “a” “b” “c” “d” 3 “Hello” 3.4 true 4 “World” 1.1 false 6 “Goodbye” 8.8 false 7 “None” 9.3 true Datalog Examples The following are a few Datalog query examples using the grammar described in the previous sections using the R fact described. Simple fact query: ECS140A FQ20 December 10, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 4 3 of 8 R(a,b,c,d) This will result in the following output: a b c d 3 Hello 3.4 true 4 World 1.1 false 6 Goodbye 8.8 false 7 None 9.3 true Simple filtering for a being greater than 5: R(a,b,c,d) S(x) := R(x,_,_,_) AND x > 5 This will result in the following output: x 6 7 Simple filtering for c being greater than 8.0 or less than 3.0: R(a,b,c,d) S(x) := R(_,_,x,_) AND x > 8.0 S(x) := R(_,_,x,_) AND x < 3.0 This will result in the following output: x 1.1 8.8 9.3 Find all b associated with c not being the smallest in R: R(a,b,c,d) S(x) := R(_,x,c1,_) AND R(_,_,c2,_) AND c1 > c2 This will result in the following output: x Hello Goodbye None ECS140A FQ20 December 10, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 4 4 of 8 Datalog Queries Write Datalog queries for the questions posed below. Name your query files query_X.nrdl where X is the number of the question below. You may assume the following facts will be available: Product(maker, model, year) Car(model, city, highway, style, passengers, trunk, msrp) Pickup(model, city, highway, passengers, cargo, towing, msrp) EV(model, range, battery, passengers, msrp) 1) What Car models have a highway less than 35miles? The final rule head should be Answer(model). 2) Find all of the Pickup models that have a cargo capacity of at least 75cu ft. and a highway fuel economy less than 25MPG. The final rule head should be Answer(model). 3) Find all automakers that sell at least one vehicle that msrp less than $27,000 and at least one vehicle greater than $55,000. The final rule head should be Answer(maker). 4) Find the passenger capacities that exist for three or more vehicles. That is three or more different vehicles should have that passenger capacity. The final rule head should be Answer(passengers). 5) Find the automaker(s) of the highest combined fuel economy (55% city, 45% highway) of conventional vehicles (cars and pickups). The final rule head should be Answer(maker). 6) Find the vehicle model with the highest miles per gallon gasoline equivalent (MPGGE). For this problem assume combined fuel economy formula from above, and that a gallon of gasoline is equivalent to 33.1kWh. The final rule head should be Answer(model). 7) Find automaker(s) that sell a pickup with a highway fuel economy lower than all the cars it sells. The final rule head should be Answer(maker). Datalog Interpreter The Datalog interpreter will be constructed from several classes. A given NRDatalog class provides a class that can parse command line arguments and instantiate and call the appropriate classes. The following classes should be created with at least the specified interface. // Class that parses the non-recursive Datalog language public class NRDatalogParser{ // Constructor for the parser public NRDatalogParser(PeekableCharacterStream stream); // Parses a query and returns true if valid query public boolean parseQuery(); // Prints out the error if one has occurred public void printError(PrintStream ostream); } ECS140A FQ20 December 10, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 4 5 of 8 // Class that will construct the parse tree from public class NRDatalogParseTree extends NRDatalogParser{ // Constructor for the parse tree public NRDatalogParseTree(PeekableCharacterStream stream); // Parses a query and returns true if valid query public boolean parseQuery(); // Prints out the error if one has occurred public void printError(PrintStream ostream); // Outputs a tree of the parsed query public void outputParseTree(PrintStream ostream); } // Class that will construct and execution tree from the parse // tree. Executes the query and displays the results. public class NRDatalogExecutionTree extends NRDatalogParseTree{ // Constructor for the execution tree public NRDatalogExecutionTree(PeekableCharacterStream stream); // Parses a query and returns true if valid query public boolean parseQuery(); // Prints out the error if one has occurred public void printError(PrintStream ostream); // Outputs a tree of how execution would occur for queries public void outputExecutionTree(PrintStream ostream); // Sets the verbosity setting for execution public void setVerbose(boolean verb); // Sets the number of threads during execution public void setThreadCount(int threadcount); // Sets the path to the data files public void setDataPath(String datapath); // Executes the parsed query public boolean executeQuery(); }Suggested Approach 1. Work on writing the Datalog queries first. An example program is available to execute the queries. This will provide you the basic understanding of how the operations should work from the query writing perspective. 2. Update your Scanner to accommodate the new set of operators. Make sure to change the newline to an operator, if a whitespace newline is desired, it must be preceded with a backslash. There are only two keywords AND and NOT. 3. Write the NRDatalogParser class to recognize the non-recursive Datalog language. Similar to project 2 you will want one method per rule. Creating an enum class that represents each rule and creating an isFirst method may aid in the development of the rules. ECS140A FQ20 December 10, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 4 6 of 8 4. Write the NRDatalogParseTree class to construct the parsed query as a tree. You will likely want to create some type of node class within the NRDatalogParseTree that holds the rule type, associated token (if any), parent, and children. You may find using a stack data member will be helpful in the recursive decent parsing. The top of the stack can hold the current parent node. 5. Write a class to hold the data sets. These should be able to hold a set of list of objects. Each list of objects can act as data tuple. It should also have a list of column names. This class should be able to be constructed from a PeekableCharacterStream (so to load using CSV), or from list of column names and the set of list of objects. The data set class should support: a. Appending tuples at least privately (this will help constructing new data sets) b. Filtering (or selecting) tuples given an object that has function returning if the tuple should be in the new data set or not (this will essentially remove rows from the data set) c. Projecting specified columns to a new data set (this is similar to cutting off columns that are not desired, may want to consider supporting renaming of the columns with this as well) d. Cartesian product with another data set to create a new data set e. Natural join with another data set to create new data set f. Filter (or selecting) tuples given tuples do not appear in another data set (the tuple should be kept if the combination of columns specified do not appear in the other data set. g. Union with another data set to create a new data set (it is assumed that both have the same column names) h. Reordering columns to create a new data set (may be helpful when projecting down temporary work into final rule) 6. Write the NRDatlogExecutionTree class to construct the tree to that it can easily be executed. You will likely want to create some type of node class within the NRDatlogExecutionTree that holds the rule type, associated token (if any), parent, children, and possibly a node index (may be helpful for sorting step). a. Translate the parse tree into an execution tree. The difference is that infix operators will be migrated up to be the parent of the operands instead of being children of the larger rule. You will likely want to write a function that recursively visits the nodes of the parse tree and returns a converted execution tree node equivalent. Tokens like open/close paren, assignment, and commas should be omitted. The body sub goals should also be sorted for simplicity of execution. Rule invocations should be first, followed by negated rule invocations and finally equality relations. b. Write a method to validate the semantic rules for the execution tree. This will make sure that execution should be able to proceed. c. Write a method to execute a rule without a rule body. This will essentially just load a data set and add it to the calculated rules. d. Write a method to execute a rule with a rule body. This will create a temporary calculated data set that will either be added to the calculated rules or unioned with the previously calculated rule of the same name. i. Create the ability to invoke rules by projecting them based upon the invocation. This projected invocation will then be either naturally joined or ECS140A FQ20 December 10, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 4 7 of 8 cartesian product with the temporary calculated data set depending if there are any overlap in column names. ii. Create the ability to do a negated invocation, this is similar to invoking rules; however, this will potentially be removing tuples from the temporary data set instead of adding tuples. iii. Create the ability to filter the tuples based upon the equality relations. A filter object should be created using the subgoal and should be able to calculate if a tuple should be kept or omitted. Examples of valid and invalid nrdl files are in /home/cjnitta/ecs140a/proj4 on the CSIF. Script that runs the NRDatalog class can be found there too. Runnig the script without options, or with the –help option will print the help. Your code will be tested on the CSIF and is expected to compile and run on the CSIF. You must submit the source file(s), a Makefile, and README.txt file, in a tgz archive. Do a make clean prior to zipping up your files so the size will be smaller. You will want to be in the parent directory of the project directory when creating the tgz archive. You can tar gzip a directory with the command: tar -zcvf archive-name.tgz directory-name You should avoid using existing source code as a primer that is currently available on the Internet. You must specify in your readme file any sources of code that you have viewed to help you complete this project. You must also provide the URL any code sources in comments of your source code. All class projects will be submitted to MOSS to determine if students have excessively collaborated. Excessive collaboration, or failure to list external code sources will result in the matter being referred to Student Judicial Affairs. Helpful Hints • The use of peekNextToken()will be helpful in some instances where it isn’t clear what the next rule should be parsed. The language is designed so that only a single look ahead is necessary. • Creating an enum class for the rule names will be helpful in multiple levels and is highly encouraged. • Use the working example to output the parse tree and the execution tree. Being able to match them will help prior to executing the queries. • Using Integer, Float, String, and Boolean types will allow the data sets to hold lists of Objects. The instanceof operator will be helpful in detecting the type during calculations of evaluations. Extra Credit After successfully creating a Datalog interpreter convert the NRDatalogExecutionTree into a multithreaded version. The executeQuery method needs to use multiple threads to execute the ECS140A FQ20 December 10, 2020 This content is protected and may not be shared, uploaded, or distributed. Project 4 8 of 8 query in parallel on systems with multiple cores. Larger data sets with a query that requires significant time will be provided later.
1 A warm-up for using Xpress This exercise is an introduction to Xpress, the main modeling language that we will use in this class. The best way to learn Xpress is to use it to model and solve optimization problems. We will solve the following facility location problem using Xpress. Suppose A = (x1, y1) and B = (x2, y2) are the coordinates of two new facilities you want to locate. There are three existing facilities at C = (300, 1200), D = (0, 600), and E = (600, 0). You want to minimize the sum of `1 distances between A and B, A and C, A and D, B and D, and B and E. The two new facilities should be located within the rectangular region (0, 900) for the x-axis and (0, 1500) for the y-axis. 1. Write down an optimization model for the above facility location problem using absolute-value functions in the objective. 2. Write down a linear programming reformulation of the above problem. 3. We have implemented your LP model in Xpress (see hw1prob1Fac.mos). The code is quite selfexplanatory and I have added comments in the code to help you understand its key features. Go over the code and make sure you understand how the variables are declared and how the objective and constraints are modeled. Now run the code by clicking on the green button with a right arrow on it. The optimal solution should be printed out in the “Output/Input” tab on the right panel of the Xpress window. Write down the optimal solution in your homework submission. Do the two facilities overlap each other? 4. Now suppose facility A needs to be located close to the facility C within a distance of 500, and the facility B needs to be located close to the facility E within a distance of 700. First, model these conditions by constraints involving absolute value functions. Write down these constraints in your homework paper submission. Second, reformulate them as linear constraints. Also write down your reformulation in the paper submission. Third, modify the hw1prob1Fac.mos code to implement the new linear constraints. Solve the new model and write down the optimal solution. Do the two new facilities overlap? You need to submit all your code on t-square. 2 Quadratic programming and piecewise linear approximation Consider the following nonlinear optimization problem involving absolute value functions: (P) min f(x) = (x1 − 1.5)2 + (x2 − 0.8)2 s.t. |x1| + |x2| ≤ 1. (1) 1 1. First, reformulate (P) by rewriting the constraints in (P) as linear constraints. This feasible region is called the `1 unit ball, because it is defined by all the points with `1-distance being 1 from the origin. Such a program is called a quadratic program (QP) with linear constraints. 2. Suppose you are given 10 feasible points, denoted as x i = (x i 1 , xi 2 ) for i = 1, …, 10. Form a convex piecewise linear function ˆf(x) using these 10 points such that ˆf(x) ≤ f(x) for all x ∈ R 2 , and ˆf(x i ) = f(x i ) for i = 1, . . . , 10, i.e. ˆf(x) is a supporting lower envelope of f(x). 3. Formulate a linear program of minimizing ˆf(x) over the `1 unit ball. 4. Generate 10 feasible points for (P), write down their numerical values. You can do this by a computer or by hand. You need to make sure the 10 points are all feasible. 5. Implement the above linear program in question 3 with the 10 points you generate in question 4 using Xpress. You need to submit the mos file on t-square and write down the optimal solution in your paper submission. NOTICE: Xpress assumes that all the mpvar variables are constrained to be nonnegative! So there is no need to specify non-negativity on variables. However, if you want to have a free variable, you need to specify it explicitly as “x(i) is free”. 6. Now, implement the quadratic program with linear constraints that you formulated in question 1 and solve it in Xpress. Use the code hw1prob2QP.mos as the template (notice the code calls “mmquad” and uses “qexp”). Compare the optimal solution and objective values of the QP and the LP approximation. Later in the course, you will implement an automatic cutting plane algorithm that generates supporting linear cuts on-fly to approximate a nonlinear function. 3 Production planning by a computer manufacturer This is a problem that Digital Equipment Corporation (DEC) had faced in 1988, which illustrates the usefulness of mathematical modeling for making important strategic decisions. DEC was a major US computer company in 1950s-1990s. It was acquired by Compaq in 1998, which was at the time the largest merger in the computer industry. Compaq later merged with HP in 2002. In the beginning of 1988, DEC introduced a new family of single CPU computer systems and workstations: GP-1, GP-2, and GP-3, which are general purpose computer systems with different memory, disk storage, and expansion capabilities, as well as WS-1 and WS-2, which are workstations. In the following table, we list the models, the list prices, and the memory usage. For example, GP-1 uses four 256k memory boards, and sells at $60,000. System Price #256K boards GP-1 $60,000 4 GP-2 $40,000 2 GP-3 $30,000 2 WS-1 $30,000 2 WS-2 $15,000 1 The following difficulties were anticipated: 1. The in-house supplier of CPUs could provide at most 7,000 units, due to debugging problems. 2 2. The supply of 256K memory boards was limited to be no more than 8,000 units. On the demand side, the marketing department established the following: 1. The maximum demand for the first quarter of 1989 would be 1,800 for GP-1 system, 300 for GP-3 system, 3,800 for the whole GP family, and 3,200 for the whole WS family. 2. Included in these projects were 500 orders for GP-2 system, 500 orders for WS-1, and 400 orders for WS-2 that had already been received and had to be fulfilled in the first quarter of 1989. DEC needed to make a production plan for the first quarter of 1989 to consider all the above production limitations and demand projections and to maximize the revenue. (a) In order to model the problem that DEC faced, we introduce variables x1, x2, x3, x4, x5 that represent the number (in thousands) of GP-1, GP-2, GP-3, WS-1, and WS-2 systems, respectively, to be produced in the first quarter of 1989. Strictly speaking, the 1000xi should be an integer to represent the number of units. But for now, we ignore the integrality constraint on 1000xi . Formulate the problem as a linear program. Fill in objective and constraints according to the descriptions. max (total revenue): s.t. (CPU availability): (256K availability): (max demand for GP-1): (max demand for GP-3): (max demand for GP family): (max demand for WS family): (min demand for GP-2): (min demand for WS-1): (min demand for WS-2): (any other constraints?): (b) Create an XpressMP model to solve the above complete LP problem. Hand in a printout of your .mos and .dat files as well as the solution output. 4 Convex functions 1. Prove the piecewise linear function f(x) = max{a > 1 x+b1, . . . , a > mx+bm} for x ∈ R n is convex using the Jensen’s inequality. 2. Prove the following functions are convex, using one of your favorite conditions (Jensen’s inequality, the first-order condition, or the second-order condition): (a) f(x) = e ax for x ∈ R and any a ∈ R. Plot it. (b) f(x) = x a for x > 0 and a ≥ 1 or a ≤ 0. Is such a function convex or concave for 0 ≤ a ≤ 1? Plot f(x) for a ≥ 1, a ≤ 0, and 0 ≤ a ≤ 1, separately. 3 (c) f(x) = − log(x) on x > 0. Plot it. (d) f(x) = x log(x) for x > 0. This function is called the negative entropy function. Plot it. (e) f(x, y) = x 2/y over x ∈ R and y > 0. Plot it. (f) f(x1, x2) = (|x1| p + |x2| p ) 1/p for (x1, x2) ∈ R 2 and p ≥ 1. This is called the `p norm. We have seen examples for p = 1, p = 2.1 Convex Sets 1. Prove that if sets S1 and S2 are convex sets in R n , then their intersection S1 ∩ S2 is a convex set, using the definition of convex sets. Hint: This should be an easy argument following from the definition. Also, we always take an empty set as a convex set. Remark: This exercise tells us an important property that the intersection of two convex sets is a convex set, in short we say, intersection preserves convexity. 2. Extend the above result to argue that S1 ∩ S2 ∩ … ∩ Sm is convex if all S1, . . . , Sm are convex sets in R n for any m ≥ 1. 3. Show that a halfspace H+ = {x ∈ R n : a >x ≥ b} is a convex set using the definition of a convex set. 4. Show that a hyperplane H = {x ∈ R n : a >x = b} is a convex set by using the conclusion of questions 1 and 3. 5. Argue that a polyhedron is always a convex set by using the results from questions 2 and 3. 6. Show that the convex hull S of a finite number of points x 1 , . . . , x m in R n is a convex set. Hint: Use the definition of a convex set. 7. Give an example to show that a nonempty convex set may not be the convex hull of a finite number of points. 8. Draw the following sets by hands using ruler and pencil (your circles do not have to be perfect). State if each set is a convex set, and list all the extreme points of each set. If a set does not have an extreme point, just say so. Hint: The definition of an extreme point applies to nonconvex sets. (a) T1 = {x ∈ R 2 : x1 + x2 ≤ 1}. (b) T2 = conv −1 1 , −2 0 , −1 −1 , 1 −1 , 2 0 , 1 1 , 0 0 . (c) T3 = {x ∈ R 2 : |x1| + |x2| ≤ 1}. (d) T4 = {x ∈ R 3 : |x1| + |x2| + |x3| ≤ 1}. (e) T5 = {x ∈ R 2 : 1 ≤ x 2 1 + x 2 2 ≤ 4}. (f) T6 = {0, 1, 2, 3, 4} ⊂ R. 1 2 Convex Functions 1. Give an example of a function f : R → R that is both convex and concave. 2. Let f1 : R n → R and f2 : R n → R be two convex functions. Prove that the sum f = f1 + f2 is always a convex function. What about af1 + bf2 for any nonnegative constants a, b? Hint: This should be a straightforward argument from the Jensen’s inequality, and this shows linear combination of convex functions with nonnegative weights is again a convex function. 3. Let f1 = |x| and f2 = |x+1|. Notice that both f1 and f2 are convex functions. Let f = f1−f2. Draw f1(x), f2(x), and f(x) for x ∈ [−2, 2] on the same plot by hand, i.e. using ruler and pencil on a piece of paper by hand. Is f a convex function on [−2, 2]? Hint: Your answer should show that the difference of two convex functions may not be a convex function any more. Contrast this with the previous question. 4. Let f0 = x 2 , f1 = (x − 1)2 , …, fk = (x − k) 2 for all x ∈ R. Let gk(x) = max{f1(x), …, fk(x)} for all x ∈ R. Draw f0, f1, f2, and g1(x) and g2(x) on two separate plots. Are they convex sets? 5. Let f1(x), …, fk(x) be convex functions on R. Prove g(x) = max{f1(x), . . . , fk(x)} is a convex function. Hint: Use the Jensen’s inequality and a similar argument that you used in Homework 1 for proving a similar result. This exercise shows that the pointwise max operation preserves convexity of functions. 6. Define a set S = {(x, y) ∈ R 2 : y = sin(x)}, i.e. S is the graph of the sine function on the entire real axis. Draw S. Is S a convex set in R 2 ? Find the convex hull of S by drawing it. Write down the inequalities that define the convex hull of S. Hint: This exercise requires you to find the convex hull of an infinite number of points. Your intuition of how to form the convex hull of a finite number of points should suffice to guide you. 3 LP geometry and the simplex method 1. Consider the following linear program: min −2×1 − 3×2 s.t. x1 + x2 ≤ 4 −x1 + x2 ≤ 2 x1 − x2 ≤ 2 x1 ≥ 0, x2 ≥ 0. Draw the feasible region of this linear program. 2. To solve this problem using the simplex method, first transform it into a standard form LP. Denote x = [x1, x2, x3, x4, x5] > as the vector of variables, and use the standard form notation: min c >x s.t. Ax = b, x ≥ 0, specify c, A, b for the above problem. 2 3. Now we want to solve the above standard-form linear program by the simplex method. If in an iteration of the Simplex method, there is any ambiguity about which nonbasic variable to enter the basis or which basic variable to exit the basis, use the Bland’s rule. (a) For each iteration of the simplex method, write down the following information in the format given below: • Iteration k = numerical value, e.g. 1, 2, …; • Basis B = [Ai , Aj , Ak] = numerical value (i.e. you need to specify i, j, k as well as the numerical values of the columns); • Basis inverse B−1 = numerical value; • Basic variable xB = [xi , xj , xk] = numerical value (you need to specify i, j, k and numerical values of xi , xj , xk); • Nonbasic variable xN = [xp, xq] = numerical value (you need to specify p, q and numerical values of xp, xq); • Reduced cost for each nonbasic variable ¯cp = cp − c > BB−1A? = numerical values; Same for ¯cq; (you need to the index “?” for A?); • Is the current solution optimal? If not, which nonbasic variable to enter the basis? • Direction dB = −B−1A? = numerical value. Does the simplex method terminate with an unbounded optimum? • Min-ratio test θ ∗ = mini:dB(i) = c > BB−1 . To take the inverse of B, you can use your favorite calculator or computer program. In this iteration, you could solve this LP by hand. But we ask you to set up the code in Xpress and solve it using Xpress. This code will be used in later iterations. 3. Write down the pricing problem, i.e. the knapsack problem using the above data and the optimal dual solution you found. 4. Solve the pricing problem in Xpress. Should we terminate the column generation algorithm at this point? Explain. If the column generation should continue, what is the new pattern generated by the pricing problem? 5. If the column generation should continue, then augment (RMP) with the new column and solve it in Xpress again. You can easily modify your Xpress code by incorporating the new column. Write down the optimal solution, the optimal basis B, and the inverse B−1 . Compute the dual variable. Then solve the pricing problem again by modifying the date in your code. Should you terminate the column generation at this iteration? Explain. If the column generation should continue, do the same for all the following iterations until the column generation terminates. 1 6. Write down the final optimal solution, the optimal basis, and the optimal objective value. 7. For this problem, you need to submit on t-square all your codes for all the steps separately. Name them as question1 RMP step1.mos, question1 Pricing step1.mos, question1 RMP step2.mos, and so on. Question 2: Dantzig-Wolfe decomposition Consider the following linear program: min x1 − 2×2 + 3×3 − 4×4 (1) s.t. x1 − x2 + x3 − x4 ≤ 1 (2) x1 + x2 ≤ 1 (3) − x1 + x2 ≤ 1 (4) x2 ≥ 0 (5) 0 ≤ x3 ≤ 1 (6) 0 ≤ x4 ≤ 1 (7) Consider constraint (2) as the complicating constraint and the rest (3)-(7) as easy constraints. Notice that the easy constraints are separable, i.e. constraints (3)-(5) only involve x1 and x2, while constraints (6)-(7) only involve x3 and x4. Therefore, define the polyhedron for variables x1 and x2 defined by (3)-(5) as P1, and P2 as the polyhedron for x3 and x4 defined by (6)-(7). 1. First, prove that both P1 and P2 are bounded. 2. Write down the master problem of the Dantzig-Wolfe decomposition. Since P1 and P2 are separable, we will use extreme point representation for P1 and P2 separately. That is, for every (x1, x2) ∈ P1, we write (x1, x2) = PN1 i=1 λi(x i 1 , xi 2 ) and λi ’s sum to one and are nonnegative, where (x i 1 , xi 2 ) is an extreme point of P1 and N1 is the number of extreme points of P1. Similarly, for every (x3, x4) ∈ P2, we write (x3, x4) = PN2 j=1 βj (x j 3 , x j 4 ) and βj ’s are convex weights. Use this representation to write down the Dantzig-Wolfe decomposition’s master problem. How many columns are there in the master problem? 3. Write down a reduced master problem (RMP) by choosing the smallest number of columns (i.e. just enough to form a basis) to start the Dantzig-Wolfe decomposition. Here, you need to write explicitly the numerical values of the objective coefficients and the matrix of the RMP. You do not need to solve this RMP. 4. Write down the pricing problem. Note that since the easy constraints are separable into P1 and P2, your pricing problem should also be separable into subproblems only involving P1 or P2. You can denote the dual variables of RMP using some letters like y or z, i.e. since you are not asked to solve the RMP, you can assume you know the dual variables and denote them as some vectors. 5. Write down the condition under which the Dantzig-Wolfe decomposition should continue.1 Two-Stage Stochastic Programming Consider the following two-stage stochastic linear program appeared in the lecture note. Z ∗ = min x1,x2,y1,y2 100×1 + 150×2 + X 2 s=1 ps(q s 1 y s 1 + q s 2 y s 2 ) s.t. x1 + x2 ≤ 120, x1 ≥ 40, x2 ≥ 20, 6y s 1 + 10y s 2 ≤ 60×1, ∀s = 1, 2, 8y s 1 + 5y s 2 ≤ 80×2, ∀s = 1, 2, 0 ≤ y s 1 ≤ d s 1 , ∀s = 1, 2, 0 ≤ y s 2 ≤ d s 2 , ∀s = 1, 2, where the random data has two scenarios: (d 1 1 , d1 2 , q1 1 , q1 2 ) = (500, 100, −24, −28) with probability 0.4 and (d 2 1 , d2 2 , q2 1 , q2 2 ) = (300, 300, −28, −32) with probability 0.6, the first-stage decision is x = (x1, x2), and the second-stage decision is y s = (y s 1 , ys 2 ) for scenarios s = 1, 2. The lecture note implemented two iterations of the Benders decomposition algorithm. Now we want to carry on the steps until an optimal solution of the two-stage stochastic program is found. 1. Write an Xpress code to solve the subproblem in each iteration. You will use this code to solve all the subproblems by entering the corresponding parameter values. You need to submit your code on t-square and turn in a hard-copy with your homework. 2. Write an Xpress code to solve the restricted master problem (RMP) in each iteration. You will update this code by adding a Benders cut in each iteration. You should incorporate the two Benders cuts found in the first two iterations of the Benders decomposition. You need to submit your code on t-square and turn in a hard-copy with your homework. 3. Now carry out the Benders decomposition from the third iteration. Follow the format given in the numerical example of the lecture, i.e. write down the RMP, write down scenario subproblems, solve them by the codes your build above, write down the optimal solutions, find the upper bound UB and the lower bound LB, and construct the optimality cut. You should only terminate when UB is equal to LB. 4. Write an Xpress code to solve the extensive formulation. Record the optimal solution and objective value. Compare it to the result of the Benders decomposition. Are they the same? 1 2 IP modeling 1. Let three binary variables x1, x2, x3 represent “event i is chosen if xi = 1, and event i is not chosen if xi = 0” for i = 1, 2, 3. Write down all feasible binary solutions of the nonlinear constraint (1 − x1)·(1 − x2)· x3 = x1. Then, reformulate the nonlinear constraint using linear constraints to describe the same set of feasible binary solutions. 2. Write one linear constraint to model the disjunction “Either event 1 is chosen or event 2 is chosen or event 3 is not chosen”. Note that the “or” is inclusive without specification. Use binary variable xi = 1 if event i is chosen, and xi = 0 if event i is not chosen. 3. Write linear constraints for the disjunction: “2×1 + x2 ≤ 2 or −x1 + 2×2 ≥ 2”. Here x1, x2 are continuous variables in the box [−3, 3] × [−3, 3] (i.e. −3 ≤ x1 ≤ 3 and −3 ≤ x2 ≤ 3). If you need to use a big M in a constraint, then you need to find the smallest valid value for M for each constraint. Draw the feasible region by hand. 4. Write linear constraints for the exclusive or (xor) statement: “2×1+x2 ≤ 2 or −x1+2×2 ≥ 2 but not both” and x1, x2 are in the box [−2, 2] × [−2, 2]. Find the best big M’s in your formulation. Then, draw the feasible region defined by the above xor statement in the plane of (x1, x2). 5. We have to place n facilities in n locations. The data are the amount fkl of goods that has to be shipped from facility k to facility l, for k = 1, . . . , n and l = 1, . . . , n, and the distance dij between locations i, j, for i = 1, . . . , n and j = 1, . . . , n. The problem is to assign facilities to locations so as to minimize the total cumulative distance traveled by the goods. For example, if f12 = 2, d34 = 3 and facility 1 is placed at location 3 and facility 2 is placed at location 4, then the total distance traveled by the goods from facility 1 to facility 2 is f12d34 = 6. Now define variable xki to be 1 if facility k is placed at location i, and 0 otherwise. (a) Write down an integer programming model for this problem in the xki variables defined above. The objective function should be a quadratic function. You need to explain each constraint and the objective of your model. (b) Write an equivalent linear integer programming model that linearizes the quadratic objective function. You need to introduce new variables for this purpose. Call these new variables y and you can name each y with appropriate sub-index. 6. A company sets an auction for n objects numbered N = {1, 2, . . . , n}. Bidders submit their bids for some subsets of the n objects that they like. The auction house has received m bids, namely bids bj dollars for subset Sj ⊆ N of objects, for j = 1, . . . , m. We can use a matrix A to store the Sj ’s in the following way: each row Ai of A corresponds to bidder i’s bid and the component Aij = 1 if bidder i chooses object j and 0 otherwise. So A is a given 0-1 matrix of m rows and n columns. The auction house is faced with the problem of choosing the winning bidders so that profit is maximized and each of the 10 objects is given to at most one bidder (multiple winning bidders can be chosen). Formulate a linear integer program for the auction house to solve this problem using the matrix A defined above. You need to define all variables carefully and explain each constraint in your IP. 2 7. The demand for a product is known to be dt units in periods t = 1, . . . , n. If we produce the product in period t, we incur a machine setup cost ft which does not depend on the number of units produced plus a production cost pt per unit produced. We may produce any number of units in any period. Any inventory carried over from period t to period t + 1 incurs an inventory cost rt per unit carried over. Initial inventory is s0. Formulate a mixed integer linear program in order to meet the demand over the n periods while minimizing overall costs. 8. Sharpen your pencil and try if you can crack the following Sudoku by hand. You can take as long as you need. Now write down a binary program to solve the Sudoku. Then code it in Xpress, solve it, and fill in your answer in the blanks. For your Xpress code, you can define a list, e.g. {1, 3, 5}, inside declarations by L = [1, 3, 5]. 8 3 6 7 9 2 5 7 4 5 7 1 3 1 6 8 8 5 1 9 4 P.S. Either you cracked this Sudoku by hand or by your binary program, congratulations! You just solved the world’s hardest Sudoku. 3 Convex hull of mixed-integer program 1. Consider the following integer program. (P) max − x1 + 2×2 s.t. − x1 + x2 ≤ 1.2 − 2×1 + x2 ≤ 1 6×1 + x2 ≤ 12 x2 ≥ 0 x1 ∈ Z, x2 ∈ Z. (a) Draw the feasible region of (P). (b) Find the optimal solution and optimal objective value of the linear relaxation of (P) by inspection. (c) Draw the convex hull of the feasible region of (P), denoted as conv(P). (d) Write down the minimal set of linear constraints for conv(P). (e) Find the optimal solution and optimal objective value of (P) by inspection. 3 2. Consider the following mixed-integer set. S = (x1, x2, y) : y + x1 ≤ 1.5, y + x2 ≤ 2, x1 ∈ Z+, x2 ∈ Z+, y ≥ 0 , where x1, x2 are nonnegative integer variables and y is a continuous nonnegative variable. (a) Draw the set S in of (x1, x2, y), with y as the axis vertical to the plane (x1, x2). (b) On the same plot as in the above question, draw the convex hull of S, denoted as conv(S). You need to be careful about the facets of the conv(S). (c) Conv(S) should be a polyhedron. Write down a minimal set of linear constraints for conv(S). (d) On a different plot, draw the feasible region of the linear programming relaxation of S, denoted as S 0 . Is S 0 the same as conv(S)? 4 Integer Hull, Branch-and-Bound, and Cutting-Plane Consider the following integer programming problem: (P) min − x1 − 2×2 s.t. 14×1 + 10×2 ≤ 39 (1) − 5×1 + 4×2 ≤ 10 (2) 2×1 − 3×2 ≤ 3 (3) x1, x2 ≥ 0, integer. 1. Solve the LP relaxation of (P) graphically. What are the optimal cost and optimal solution of the linear programming relaxation? 2. Solve the IP graphically. What are the optimal cost and optimal solution of the integer programming problem? 3. What is the integer hull of the set of feasible integer solutions? Find the minimal set of defining linear inequalities for the integer hull. 4. Solve the problem by branch-and-bound. At the root node, branch on x1 variable. Solve the LP relaxations in the bounding process graphically. 5. Now we want to derive a Chvatal-Gomory cut following the lecture notes. First write (P) in standard form. After you solve the LP relaxation of (P) graphically, you will find that x1, x2 are both basic variables. Use the standard form of Eq. (1) and Eq. (2) to derive a new linear equality which has no x1 term and has x2 term with coefficient 1. Write down this linear equality. Now do rounding down on both sides of it and write down the resulting Chvatal-Gomory cut. Does it cut off the LP relaxation solution?Problem 1: Column Generation for the Diet Problem In this problem, we will solve a large diet problem. There are 7146 kinds of food listed, and 30 nutrients. We want to find a diet that has the minimum level of cholesterol intake and at the same time satisfies all the nutritional requirement. For each nutrient i, let mi be the minimum daily intake of i and let Mi be the maximum daily intake of i. For each food j, let aij be the amount of nutrient i in food j. Let cholj be the amount of cholesterol in food j, and define variables xj to be the amount of food j in the daily diet. Then, the formulation of the diet problem is: min 7146 X j=1 cholj · xj (1) s.t. 7146 X j=1 aijxj ≤ Mi , ∀i = 1, . . . , 27 (2) 7146 X j=1 aijxj ≥ mi , ∀i = 1, . . . , 27 (3) xj ≥ 0, ∀j = 1, . . . , 7146 (4) Suppose you wanted to solve it using the student (free) version of Xpress, which cannot handle 7146 variables. Apparently, you need to solve the problem in a more clever way. Column generation is an ideal approach. Attached file (diet.colgen.partial.mos) gives the general framework of column generation, but it is missing the crucial part. Questions: 1. Complete the construction of the column generation code and submit the mos file online and also submit a physical copy with the final report. 2. Print out the final solution from the code, and submit it in the final report. 3. Now, you want to find an optimal diet with the total calorie intake between 1800 cal and 2000 cal. Modify the data file diet.xls (the minimum and maximum levels in the calorie column), and resolve the above problem. Report in the final report the composition of optimal diet and the total amount of each nutrient provided by the diet. HINT: 1. getdual(name) is the Xpress function to retrieve the optimal dual variable (also called shadow price) of a constraint called “name”. To store that shadow price, you need to save that value in the array given to you. 2. A general way to add a new column to a constraint or the objective function is: name += XXX, where name is the name of the constraint or the objective function, and XXX is the part of the constraint that comes with the new variable. 2 Problem 2: Solution Strategies for the Cutting Stock Problem The cutting stock problem formulation that we learned in class is due to P. C. Gilmore and Ralph E. Gomory. They published this formulation in their 1961 paper “A linear programming approach to the cutting-stock problem”, Operations Research, 8 (1961), 849-859. We call this formulation the Gilmore-Gomory formulation. A little history here: Ralph E. Gomory is a renowned mathematician and a key figure in the development of both theoretical understandings and computational methods for integer programming. He has also been very influential in bringing operations research to the real world. He first worked at IBM as a research mathematician and later became IBM’s Senior Vice President for Science and Technology. He helped IBM attain some of the best minds for OR for over 20 years. He still maintains a blog in the Huffington Post on interesting topics such as technology development and industry research1 at an advanced age. An alternative optimization formulation for the cutting stock problem was proposed by the Soviet mathematician and economist Leonid V. Kantorovich in 1939. His paper was published in English in 1960 (“Mathematical Methods of Planning and Organising Production” Management Science, 6 (1960), 366-422. You can get the paper online through Tech’s library.) Later, Kantorovich won the Nobel Prize in Economics in 1975, shared with another pioneer of operations research Tjalling Koopmans, “for their contributions to the theory of optimal allocation of resources.” Kantorovich’s formulation for the cutting stock problem is very different from the Gilmore-Gomory formulation. In this problem, we lead you through steps to formulate the Kantorovich formulation and implement in Xpress. Then, we ask you to implement Column Generation on the Gilmore-Gomory formulation. The purpose is to compare these two different formulations and solution strategies for solving the same cutting stock problem. Through this exercise, you will learn that the compact formulation of an optimization problem might not always be computationally easier to solve than a larger formulation. You will also see the power of column generation as a solution strategy to solve large-scale linear programs. Problem 2.1: The Kantorovich Formulation: A Modelling Exercise The Kantorovich formulation uses the following data and decision variables: 1. Data: • wi , i = 1, . . . , m: the width of small roll item i. • bi , i = 1, . . . , m: the demand for item i. • W: the width of the large rolls. • K: the total number of large rolls. (Note that this data is not required in the Gilmore-Gomory formulation.) 2. Decision variables: • y k : if large roll k is cut then y k = 1, otherwise y k = 0, for k = 1, . . . , K. • x k i : number of times that item i of width wi is cut on the large roll k. Now we lead you through the steps to formulate the Kantorovich model. • The objective is given as to minimize the number of large rolls cut to satisfy demand: min X K k=1 y k 1https://www.huffingtonpost.com/ralph-gomory/ 3 • Constraint 1: Demand must be satisfied with equality for each item i = 1, . . . , m. • Constraint 2: The width of a large roll can not be exceeded for each large roll k = 1, . . . , K. Here you should consider that Constraint 2 is only needed when a large roll is cut, which involves the y k variable. • Constraint 3: Variable bounds and integrality constraints. Questions: 1. How many variables and constraints are there in this formulation? 2. Use the files cs.Kantorovich.partial.mos as a starting point. Finish the Xpress code and solve the problem using the data file kant1.dat. Submit the mos file online and also submit a physical copy with the final report. Print out the optimal objective value and the optimal solution in the report. Since there are many variables, to save space, only print out the non-zero optimal variables for both x k i ’s and y k ’s. 3. Next change K to 600 and change m to 10 and use the data file cs1.dat. What do you observe? Is it easy or hard to solve the problem using the Kantorovich formulation? You can keep the solver running as long as you want, when it terminates (or when you run out of patience and stops it manually, click on the red cross button), answer the following questions: • How many branch-and-bound (BB) nodes are searched by the BB algorithm? • What is the objective value of the best lower bound and the objective value of the best integer solution found? • What is the optimality gap given by the above two numbers? • How many integer solutions are found by the BB algorithm? • How many seconds did the algorithm take? For these questions, click on the “Stats” tab on the right-hand side panel in Xpress. Look at the results after “Current node”, “Best Bound”, “Best solution”, “Gap”, “Status”, and “Time”. Lastly, relax the integrality constraints. Write your answers in the report. 4 Problem 2.2: The Gilmore-Gomory Formulation: Use Column Generation You should encounter some difficulty with the Kantorovich formulation for the larger data set. In this part, we want to solve the Gilmore-Gomory formulation (exactly the same formulation we discussed in class) using column generation. The master problem is given below. min X N j=1 xj s.t. X N j=1 aijxj = bi , ∀i = 1, . . . , m (Demand Constraints) xj ≥ 0, ∀j = 1, . . . , N. Attached file (cs.colgen.partial.mos) gives the general framework of the column generation algorithm. It also has detailed instructions on how to complete the code. The programming experience you gained in solving Problem 1, the diet problem, will be useful here. You need to do the following. Questions: 1. Complete the column generation code. Submit the completed mos file online and also submit a physical copy with the final report. 2. Print out solution summary for each test instance (cs1.dat, cs2.dat, cs3.dat), following the instructions in the cs.colgen.partial.mos file. 3. In your report, discuss the differences between the LP solutions and the integer solutions obtained in the three test instances, also the differences between the integer solutions of the two approaches (rounding and resolving). Which approach produces better solution? 4. Now, change the demand constraint from = to ≥, that is, PN j=1 aijxj ≥ bi . Solve the LP relaxation of the resulting cutting stock problem and obtain the integer solutions using the same procedure as in the above question for kant1.dat, cs1.dat, cs2.dat, cs3.dat. Compare the results to the ones of the equality demand constraint. If the two formulations give different solutions, elaborate how the solutions are different for each test case. HINT: 1. The Xpress code uses dynamic array to implement an array whose size can change dynamically. To create a new variable, use create(variable name). 2. You can use a similar way to add new columns to constraints and the objective function as hinted in problem 1. Problem 2.3: An Alternative Objective – Minimize Total Waste Now let us consider an alternative objective of minimizing the total wasted paper for satisfying demand. More specifically, the total wasted paper of a cutting stock solution is the total amount of paper that is cut but not used to satisfy the demand. For instance, if a large roll of width 100 is cut into 3 copies of width 30 and the remaining 10 = 100 − 3 × 30 units of width is thrown away, then that 10 units of paper is counted as wasted paper of this roll. The total amount of wasted paper is the sum of wasted paper from each large roll that is cut to satisfy demand. We call the new cutting stock problem, CS-min-waste. Answer the following questions. Questions: 5 1. Modify the Gilmore-Gomory formulation (the LP version) to incorporate the objective of minimizing the total wasted paper. The demand constraint is still an equality constraint. Write down the new formulation. 2. Use column generation to solve CS-min-waste. Write down the reduced master problem (RMP) and the pricing problem. In particular, the pricing problem should be written first as an enumerate problem and then be transformed into an optimization problem. 3. Write a new .mos file (name it cs.colgen.minwaste.mod) to implement the column generation for CSmin-waste. Your code should also include the capability to get an integer solution through resolving. Hand in your code on t-square. 4. Run your code cs.colgen.minwaste.mod on all four test cases kant1.dat, cs1.dat, cs2.dat, cs3.dat. Compare CS-min-waste with the original cutting-stock formulation in Problem 2.2. For each test case, does the LP solution of CS-min-waste have the same optimal objective with that of the original cuttingstock problem? What about the optimal LP solution? Does the integer solution of CS-min-waste have the same objective with the original problem? Compare the total amount of wasted paper of the integer solution of the cs-min-waste model with that of the original model? How much different are they? 5. After you run the test cases, does your results suggest some interesting relation between the LP solutions of the two cutting stock models? Prove that these two models (both the LP relaxation and the integer version of the CS-min-waste and the original cutting-stock problems) are in fact equivalent, thus always give the same optimal objective value. Or argue that your experimental results disprove the above statement. 6. Now change the demand constraint to ≥ as in Problem 2.2 Question 4. Modify your cs.colgen.minwaste.mod code and rerun all the four test cases. Answer Problem 2.3 Question 4 again with the new results. Can you explain why the new results with demand constraint of ≥ are different or the same as the ones with demand constraints of = sign you obtained in Problem 2.3 Question 4? 6 Problem 3: TSP using Dantzig-Fulkerson-Johnson Formulation In this problem, we study the classic formulation for the TSP problem, namely the Dantzig-Fulkerson-Johnson (DFJ) formulation, and implement a constraint generation algorithm to solve it. DFJ Formulation The DFJ formulation was first proposed by G. Dantzig, D. Fulkerson, and S. Johnson. All three of them made significant contributions to operations research and optimization. The original paper where they first proposed the formulation has become a classic “Solutions of large scale travelling salesman problem”, Operations Research, 2:393-410, 1954. (DF J) min Xn i=1 Xn j=1 dijxij (5) s.t. Xn j=1 xij = 1 for all cities i = 1, 2 . . . , n (6) Xn j=1 xji = 1 for all cities i = 1, 2 . . . , n (7) X i∈S X j∈S xij ≤ |S| − 1 for all subset S ⊂ N and S 6= N (8) xij ∈ {0, 1} for all cities i, j = 1, 2, . . . , n The decision variables xij = 1 if the tour goes directly from city i to city j, xij = 0 otherwise. The objective is to minimize the total travel distance of the tour Pn j=1 dijxij , where dij is the distance between cities i and j. Constraint (6) ensures that the tour goes from city i to exactly one other city, and Constraint (7) ensures that the tour goes into city i exactly from one other city. These two types of constraints together are called the assignment constraints. However, as we know, they are not enough to characterize a correct tour, because the solution to the assignment constraints may contain subtours of smaller cycles around subsets of cities. We need constraints (8) to eliminate all the subtours. Here N = {1, 2, . . . , n} is the set of all n cities, and S is any strict subset of N with |S| number of cities. The DFJ formulation has n 2 binary variables. However, the number of constraints is astronomical! Indeed, we need a subtour elimination constraint (8) for each non-empty subset S of N, excluding N itself (why?). There are 2n − 2 such constraints. That’s why we say the number of constraints is exponentially many in the number of variables. For a problem of 48 cities, there will be 248 − 2 = 281, 474, 976, 710, 654 (281 trillion) subtour elimination constraints. It will be completely nonsensical to enter such a model into Xpress! Constraint Generation Algorithm In the following, we outline a constraint generation procedure that can be implemented quite easily in Xpress and can find an optimal (yah!) solution fairly fast. The strategy is as follows: 1. First form the restricted master problem with a subset of constraints (8). In the following we will explore different options for the initial set of constraints. 2. Solve the restricted master problem. 3. Check if any subtour elimination constraint is violated by the current solution. There are many ways to do this. We choose the following method: Find a tour starting at City 1 (which is Atlanta). Such a tour always exists and is unique (Why?). Then, if this tour contains less than 48 cities, then this is a subtour and we can add the corresponding subtour elimination constraint (8) to the restricted master 7 problem. Go to Step 2 and repeat. Otherwise if the tour has all 48 cities, then it is a legitimate TSP tour, and in fact optimal, then we are done. Steps to Implement the Algorithm The above outlined framework of constraint generation is partially coded in TSP-partial.mos. The data file US48.dat contains coordinates of the 48 state capitals. You need to fill in the parts that are marked “!!!!!!!!! fill in your code here !!!!!!!!!!!”. In particular, you need to do the following to complete the code: 1. Formulate the objective function, all the assignment constraints. 2. Write small subtour elimination constraints for subsets of 1 city, 2 cities, 3 cities, and 4 cities. 3. Finish the constraint generation part: (a) Write code to find a subtour starting at Atlanta (City 1). Hint: One way to find such a tour is to start at Atlanta and see which city comes next, then see which city comes after that, etc, until you trace back to Atlanta. Mark all the cities on the subtour. (b) Write code to generate the corresponding subtour elimination constraints and add it to the restricted problem. Computational Experiments 1. Begin with all of the small subtour elimination constraints commented out (So, these will be generated by your constraint generation code if necessary.) Use your Xpress code to solve the problem. If it runs for more than 20 minutes, stop Xpress in the middle — report “Reach time limit of 20 min”. If it does finish quickly, record how long it takes and how many constraints were generated. You can find the running time in the “Stats” tab in Xpress. 2. Now un-comment the 1-city subtour elimination constraints, so that the formulation begins with those constraints already in it. Use your Xpress code to solve the problem. If it runs for more than 20 minitues stop Xpress in the middle and report “Reach time limit of 20 min”. If it does finish quickly, record how long it takes and how many constraints were generated. 3. Now start the formulation with both 1-city and 2-city subtour elimination constraints. Use your Xpress code to solve the problem. If it runs for more than 20 minitues stop Xpress and report “Reach time limit of 20 min”. If it does finish quickly, record how long it takes and how many constraints were generated. 4. Now start the formulation with 1-city, 2-city, and 3-city subtour elimination constraints. Use your Xpress code to solve the problem. If it runs for more than 20 minitues stop Xpress and report “Reach time limit of 20 min”. If it does finish quickly, record how long it takes and how many constraints were generated. 5. Now start the formulation with 1-city, 2-city, 3-city, and 4-city subtour elimination constraints. Use your Xpress code to solve the problem. If it runs for more than 20 minitues stop Xpress and report “Reach time limit of 20 min”. If it does finish quickly, record how long it takes and how many constraints were generated. 6. For each of (i)-(v), how many subtour elimination constraints did the formulation begin with? 7. Plot the optimal TSP tour you found, using the Matlab code US48TourPlot.m. 8. Figure out each city’s name (City 1 is Atlanta). Write the TSP tour in the names of the state capitals. 8 9. Select 26 cities that you want to visit. Create a new data file US26.dat. You will need to use this data set for the next part as well. Run your Xpress code to find an optimal TSP tour. Report the optimal tour. Modify the Matlab code and generate a similar plot of the tour over these 26 cities. You need to hand-in the new data file for this part. Deliverables 1. Complete and well-commented Xpress codes (both .mod and US26.dat). Include all of the 1-city, 2-city, 3-city, and 4-city subtour constraints, but comment out the 3-city and 4-city subtour constraints. Data initialization part uses the 48-city data. 2. Matlab code for reading data and generating the plot for 26-city tour. 3. In the final report, answer all the questions in steps 1-6 of Section , and print out the optimal 48-city TSP tour and the optimal 26-city TSP tour, their total distances, and the plots generated by the Matlab code. For the 26-city tour, you need to clearly state the indices and the names of the 26 cities you select, and attach the data file you used in Xpress. Attach the complete Xpress and Matlab code in the end.
1 Probability [15 pts] (a) Stores A, B, and C have 50, 75, and 100 employees and, respectively, 50, 60, and 70 percent of these are women. Resignations are equally likely among all employees, regardless of stores and sex. Suppose an employee resigned, and this was a woman. What is the probability that she has worked in store C? [5 pts] (b) A laboratory blood test is 95 percent effective in detecting a certain disease when it is, in fact, present. The test also yields a false positive result for 1 percent of the healthy persons tested. That is, if a healthy person is tested then with probability 0.01 the test result will imply he has the disease. If 0.5 percent of the population actually has the disease, what is the probability a person has the disease given that his test result is positive? [5 pts] [c-d] On the morning of September 31, 1982, the won-lost records of the three leading baseball teams in the western division of the National League of the United States were as follows: Team Won Lost Atlanta Braves 87 72 San Francisco Giants 86 73 Los Angeles Dodgers 86 73 Each team had 3 games remaining to be played. All 3 of the Giants games were with the Dodgers, and the 3 remaining games of the Braves were against the San Diego Padres. Suppose that the outcomes of all remaining games are independent and each game is equally likely to be won by either participant. If two teams tie for first place, they have a playoff game, which each team has an equal chance of winning. (c) What is the probability that Atlanta Braves wins the division? [2 pts] (d) What is the probability to have an additional playoff game? [3 pts] 1Christopher M. Bishop, Pattern Recognition and Machine Learning, 2006, Springer. 1 2 Maximum Likelihood [15 pts] Suppose we have n i.i.d (independent and identically distributed) data samples from the following probability distribution. This problem asks you to build a log-likelihood function, and find the maximum likelihood estimator of the parameter(s). (a) Poisson distribution [5 pts] The Poisson distribution is defined as P(xi = k) = λ k e −λ k! (k = 0, 1, 2, …). What is the maximum likelihood estimator of λ? (b) Multinomial distribution [5 pts] The probability density function of Multinomial distribution is given by f(x1, x2, . . . , xk; n, θ1, θ2, . . . , θk) = n! x1!x2! · · · xk! Y k j=1 θ xj j , where Pk j=1 θj = 1, Pk j=1 xj = n. What is the maximum likelihood estimator of θj , j = 1, . . . k? (c) Gaussian normal distribution [5 pts] Suppose we have n i.i.d (Independent and Identically Distributed) data samples from a univariate Gaussian normal distribution N (µ, σ2 ), which is given by N (x; µ, σ2 ) = 1 σ √ 2π exp − (x − µ) 2 2σ 2 . What is the maximum likelihood estimator of µ and σ 2 ? 3 Principal Component Analysis [20 pts] In class, we learned that Principal Component Analysis (PCA) preserves variance as much as possible. We are going to explore another way of deriving it: minimizing reconstruction error. Consider data points xn(n = 1, …, N) in D-dimensional space. We are going to represent them in {u1, …, uD} orthonormal basis. That is, x n = X D i=1 α n i ui = X D i=1 (xnT ui)ui . Here, α n i is the length when xn is projected onto ui . Suppose we want to reduce the dimension from D to M < D. Then the data point xn is approximated by ˜x n = X M i=1 z n i ui + X D i=M+1 biui . 2 In this representation, the first M directions of ui are allowed to have different coefficient z n i for each data point, while the rest has a constant coefficient bi . As long as it is the same value for all data points, it does not need to be 0. Our goal is setting ui , z n i , and bi for n = 1, …, N and i = 1, …, D so as to minimize reconstruction error. That is, we want to minimize the difference between xn and ˜xn over {ui , zn i , bi}: J = 1 N X N n=1 kx n − ˜x n k 2 . (a) What is the assignment of z n j for j = 1, …, M minimizing J? [5 pts] (b) What is the assignment of bj for j = M + 1, …, D minimizing J? [5 pts] (c) Express optimal x˜ n and xn − x˜ n using your answer for (a) and (b). [2 pts] (d) What should be the ui for i = 1, …, D to minimize J? [8 pts] Hint: Use S = 1 N PN n=1(xn − ¯x)(xn − ¯x)T for sample covariance matrix. 4 Clustering [20 pts] [a-b] Given N data points xn(n = 1, . . . , N), K-means clustering algorithm groups them into K clusters by minimizing the distortion function over {r nk, µk} J = X N n=1 X K k=1 r nkkx n − µ k k 2 , where r nk = 1 if xn belongs to the k-th cluster and r nk = 0 otherwise. (a) Prove that using the squared Euclidean distance kx n − µ kk 2 as the dissimilarity function and minimizing the distortion function, we will have µ k = P n r nk P xn n r nk . That is, µ k is the center of k-th cluster. [5 pts] (b) Prove that K-means algorithm converges to a local optimum in finite steps. [5 pts] [c-d] In class, we discussed bottom-up hierarchical clustering. For each iteration, we need to find two clusters {x1, x2, . . . , xm} and {y1 , y2 , . . . , yp} with the minimum distance to merge. Some of the most commonly used distance metrics between two clusters are: • Single linkage: the minimum distance between any pairs of points from the two clusters, i.e. min i=1,…,m j=1,…,p kxi − yjk • Complete linkage: the maximum distance between any parts of points from the two clusters, i.e. max i=1,…,m j=1,…,p kxi − yjk 3 • Average linkage: the average distance between all pair of points from the two clusters, i.e. 1 mp Xm i=1 Xp j=1 kxi − yjk (c) When we use the bottom up hierarchical clustering to realize the partition of data, which of the three cluster distance metrics described above would most likely result in clusters most similar to those given by K-means? (Suppose K is a power of 2 in this case). [5 pts] (d) For the following data (two moons), which of these three distance metrics (if any) would successfully separate the two moons? [5 pts] −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 −2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 5 Programming: Image compression [30 pts] In this programming assignment, you are going to apply clustering algorithms for image compression. Before starting this assignment, we strongly recommend reading PRML Section 9.1.1, page 428 – 430. To ease your implementation, we provide a skeleton code containing image processing part. homework1.m is designed to read an RGB bitmap image file, then cluster pixels with the given number of clusters K. It shows converted image only using K colors, each of them with the representative color of centroid. To see what it looks like, you are encouraged to run homework1(‘beach.bmp’, 3) or homework1(‘football.bmp’, 2), for example. Your task is implementing the clustering parts with two algorithms: K-means and K-medoids. We learned and demonstrated K-means in class, so you may start from the sample code we distributed. The file you need to edit is mykmeans.m and mykmedoids.m, provided with this homework. In the files, you can see it calls Matlab function kmeans initially. Comment this line out, and implement your own in the files. You would expect to see similar result with your implementation of K-means, instead of kmeans function in Matlab. 4 K-medoids In class, we learned that the basic K-means works in Euclidean space for computing distance between data points as well as for updating centroids by arithmetic mean. Sometimes, however, the dataset may work better with other distance measures. It is sometimes even impossible to compute arithmetic mean if a feature is categorical, e.g, gender or nationality of a person. With K-medoids, you choose a representative data point for each cluster instead of computing their average. Given N data points xn(n = 1, …, N), K-medoids clustering algorithm groups them into K clusters by minimizing the distortion function J = PN n=1 PK k=1 r nkD(xn, µk ), where D(x, y) is a distance measure between two vectors x and y in same size (in case of K-means, D(x, y) = kx − yk 2 ), µ k is the center of k-th cluster; and r nk = 1 if xn belongs to the k-th cluster and r nk = 0 otherwise. In this exercise, we will use the following iterative procedure: • Initialize the cluster center µ k , k = 1, …, K. • Iterate until convergence: – Update the cluster assignments for every data point xn: r nk = 1 if k = arg minj D(xn, µj ), and r nk = 0 otherwise. – Update the center for each cluster k: choosing another representative if necessary. There can be many options to implement the procedure; for example, you can try many distance measures in addition to Euclidean distance, and also you can be creative for deciding a better representative of each cluster. We will not restrict these choices in this assignment. You are encouraged to try many distance measures as well as way of choosing representatives. Formatting instruction Both mykmeans.m and mykmedoids.m take input and output format as follows. You should not alter this definition, otherwise your submission will print an error, which leads to zero credit. Input • pixels: the input image representation. Each row contains one data point (pixel). For image dataset, it contains 3 columns, each column corresponding to Red, Green, and Blue component. Each component has an integer value between 0 and 255. • K: the number of desired clusters. Too high value of K may result in empty cluster error. Then, you need to reduce it. Output • class: cluster assignment of each data point in pixels. The assignment should be 1, 2, 3, etc. For K = 5, for example, each cell of class should be either 1, 2, 3, 4, or 5. The output should be a column vector with size(pixels, 1) elements. • centroid: location of K centroids (or representatives) in your result. With images, each centroid corresponds to the representative color of each cluster. The output should be a matrix with K rows and 3 columns. The range of values should be [0, 255], possibly floating point numbers. Hand-in Both of your code and report will be evaluated. Upload mykmeans.m and mykmedoids.m files with your implementation. In your report, answer to the following questions: 5 1. Within the K-medoids framework, you have several choices for detailed implementation. Explain how you designed and implemented details of your K-medoids algorithm, including (but not limited to) how you chose representatives of each cluster, what distance measures you tried and chose one, or when you stopped iteration. 2. Attach a picture of your own. We recommend size of 320 × 240 or smaller. 3. Run your K-medoids implementation with the picture you chose above, with several different K. (e.g, small values like 2 or 3, large values like 16 or 32) What did you observe with different K? How long does it take to converge for each K? 4. Run your K-medoids implementation with different initial centroids/representatives. Does it affect final result? Do you see same or different result for each trial with different initial assignments? (We usually randomize initial location of centroids in general. To answer this question, an intentional poor assignment may be useful.) 5. Repeat question 2 and 3 with K-means. Do you see significant difference between K-medoids and K-means, in terms of output quality, robustness, or running time? Note • You may see some error message about empty clusters even with Matlab implementation, when you use too large K. Your implementation should treat this exception as well. That is, do not terminate even if you have an empty cluster, but use smaller number of clusters in that case. • We will grade using test pictures which are not provided. We recommend you to test your code with several different pictures so that you can detect some problems that might happen occasionally. • If we detect copy from any other student’s code or from the web, you will not be eligible for any credit for the entire homework, not just for the programming part. Also, directly calling Matlab function kmeans or other clustering functions is not allowed.1 EM for Mixture of Gaussians Mixture of K Gaussians is represented as p(x) = X K k=1 πkN (x|µk, Σk), (1) where πk represents the probability that a data point belongs to the kth component. As it is probability, it satisfies 0 ≤ πk ≤ 1 and P k πk = 1. In this problem, we are going to represent this in a slightly different manner with explicit latent variables. Specifically, we introduce 1-of-K coding representation for latent 1https://matlab.mathworks.com/ 2You can also use the environment for developing by registering online with your university email. 3Christopher M. Bishop, Pattern Recognition and Machine Learning, 2006, Springer. 1 variables z (k) ∈ R K for k = 1, …, K. Each z (k) is a binary vector of size K, with 1 only in kth element and 0 in all others. That is, z (1) = [1; 0; …; 0] z (2) = [0; 1; …; 0] . . . z (K) = [0; 0; …; 1]. For example, if the second component generated data point x n, its latent variable z n is given by [0; 1; …; 0] = z (2). With this representation, we can express p(z) as p(z) = Y K k=1 πk zk , where zk indicates kth element of vector z. Also, p(x|z) can be represented similarly as p(x|z) = Y K k=1 N (x|µk, Σk) zk . By the sum rule of probability, (1) can be represented by p(x) = X z∈Z p(z)p(x|z). (2) where Z = {z (1), z(2), …, z(K)}. (a) Show that (2) is equivalent to (1). [5 pts] (b) In reality, we do not know which component each data point is from. Thus, we estimate the responsibility (expectation of z n k ) in the E-step of EM. Since z n k is either 1 or 0, its expectation is the probability for the point xn to belong to the component zk. In other words, we estimate p(z n k |xn). Derive the formula for this estimation by using Bayes rule. Note that, in the E-step, we assume all other parameters, i.e. πk, µk, and Σk, are fixed, and we want to express p(z n k |xn) as a function of these fixed parameters. [10 pts] (c) In the M-Step, we re-estimate parameters πk, µk, and Σk by maximizing the log-likelihood. Given N i.i.d (Independent Identically Distributed) data samples, derive the update formula for each parameter. Note that in order to obtain an update rule for the M-step, we fix the responsibilities, i.e. p(z n k |xn), which we have already calculated in the E-step. [15 pts] Hint: Use Lagrange multiplier for πk to apply constraints on it. (d) EM and K-Means [10 pts] K-means can be viewed as a particular limit of EM for Gaussian mixture. Considering a mixture model in which all components have covariance I, show that in the limit → 0, maximizing the expected complete data log-likelihood for this model is equivalent to minimizing objective function in K-means: J = X N n=1 X K k=1 γnkkxn − µkk 2 , where γnk = 1 if xn belongs to the k-th cluster and γnk = 0 otherwise. 2 2 Density Estimation Consider a histogram-like density model in which the space x is divided into fixed regions for which density p(x) takes constant value hi over ith region, and that the volume of region i in denoted as ∆i . Suppose we have a set of N observations of x such that ni of these observations fall in regions i. (a) What is the log-likelihood function? [8 pts] (b) Derive an expression for the maximum likelihood estimator for hi. [10 pts] Hint: This is a constrained optimization problem. Remember that p(x) must integrate to unity. Since p(x) has constant value hi over region i, which has volume ∆i . The normalization constraint is P i hi∆i = 1. Use Lagrange multiplier by adding λ ( P i hi∆i − 1) to your objective function. (c) Mark T if it is always true, and F otherwise. Briefly explain why. [12 pts] • Non-parametric density estimation usually does not have parameters. • The Epanechnikov kernel is the optimal kernel function for all data. • Histogram is an efficient way to estimate density for high-dimensional data. • Parametric density estimation assumes the shape of probability density. 3 Information Theory In the lecture you became familiar with the concept of entropy for one random variable and mutual information. For a pair of discrete random variables X and Y with the joint distribution p(x, y), the joint entropy H(X, Y ) is defined as H(X, Y ) = − X x∈X X y∈Y p(x, y) log p(x, y) (3) which can also be expressed as H(X, Y ) = −E[log p(X, Y )] (4) Let X and Y take on values x1, x2, …, xr and y1, y2, …, ys respectively. Let Z also be a discrete random variable and Z = X + Y . (a) Prove that H(X, Y ) ≤ H(X) + H(Y ) [4 pts] (b) Show that I(X; Y ) = H(X) + H(Y ) − H(X, Y ). [2 pts] (c) Under what conditions does H(Z) = H(X) + H(Y ). [4 pts] 4 Programming: Text Clustering In this problem, we will explore the use of EM algorithm for text clustering. Text clustering is a technique for unsupervised document organization, information retrieval. We want to find how to group a set of different text documents based on their topics. First we will analyze a model to represent the data. 3 Bag of Words The simplest model for text documents is to understand them as a collection of words. To keep the model simple, we keep the collection unordered, disregarding grammar and word order. What we do is counting how often each word appears in each document and store the word counts into a matrix, where each row of the matrix represents one document. Each column of matrix represent a specific word from the document dictionary. Suppose we represent the set of nd documents using a matrix of word counts like this: D1:nd = 2 6 … 4 2 4 … 0 . . . . . . = T This means that word W1 occurs twice in document D1 . Word Wnw occurs 4 times in document D1 and not at all in document D2. Multinomial Distribution The simplest distribution representing a text document is multinomial distribution(Bishop Chapter 2.2). The probability of a document Di is: p(Di) = Ynw j=1 µ Tij j Here, µj denotes the probability of a particular word in the text being equal to wj , Tij is the count of the word in document. So the probability of document D1 would be p(D1) = µ 2 1 · µ 6 2 · … · µ 4 nw · Mixture of Multinomial Distributions In order to do text clustering, we want to use a mixture of multinomial distributions, so that each topic has a particular multinomial distribution associated with it, and each document is a mixture of different topics. We define p(c) = πc as the mixture coefficient of a document containing topic c, and each topic is modeled by a multinomial distribution p(Di |c) with parameters µjc, then we can write each document as a mixture over topics as p(Di) = Xnc c=1 p(Di |c)p(c) = Xnc c=1 πc Ynw j=1 µ Tij jc EM for Mixture of Multinomials In order to cluster a set of documents, we need to fit this mixture model to data. In this problem, the EM algorithm can be used for fitting mixture models. This will be a simple topic model for documents. Each topic is a multinomial distribution over words (a mixture component). EM algorithm for such a topic model, which consists of iterating the following steps: 1. Expectation Compute the expectation of document Di belonging to cluster c: γic = πc Qnw j=1 µ Tij jc Pnc c=1 πc Qnw j=1 µ Tij jc 4 2. Maximization Update the mixture parameters, i.e. the probability of a word being Wj in cluster (topic) c, as well as prior probability of each cluster. µjc = Pnd i=1 P γicTij nd i=1 Pmw l=1 γicTil πc = 1 nd Xnd i=1 γic Task [20 pts] Implement the algorithm and run on the toy dataset data.mat. You can find detailed description about the data in the homework2.m file. Observe the results and compare them with the provided true clusters each document belongs to. Report the evaluation (e.g. accuracy) of your implementation. Hint: We already did the word counting for you, so the data file only contains a count matrix like the one shown above. For the toy dataset, set the number of clusters nc = 4. You will need to initialize the parameters. Try several different random initial values for the probability of a word being Wj in topic c, µjc. Make sure you normalized it. Make sure that you should not use the true cluster information during your learning phase. Extra Credit: Realistic Topic Models [20pts] The above model assumes all the words in a document belongs to some topic at the same time. However, in real world datasets, it is more likely that some words in the documents belong to one topic while other words belong to some other topics. For example, in a news report, some words may talk about “Ebola” and “health”, while others may mention “administration” and “congress”. In order to model this phenomenon, we should model each word as a mixture of possible topics. Specifically, consider the log-likelihood of the joint distribution of document and words L = X d∈D X w∈W Tdw log P(d, w), (5) where Tdw is the counts of word w in the document d. This count matrix is provided as input. The joint distribution of a specific document and a specific word is modeled as a mixture P(d, w) = X z∈Z P(z)P(w|z)P(d|z), (6) where P(z) is the mixture proportion, P(w|z) is the distribution over the vocabulary for the z-th topic, and P(d|z) is the probability of the document for the z-th topic. And these are the parameters for the model. The E-step calculates the posterior distribution of the latent variable conditioned on all other variables P(z|d, w) = P(z)P(w|z)P(d|z) P z 0 P(z 0)P(w|z 0)P(d|z 0) . (7) In the M-step, we maximizes the expected complete log-likelihood with respect to the parameters, and get the following update rules P(w|z) = P d TdwP(z|d, w) P w0 P d Tdw0P(z|d, w0) (8) P(d|z) = P w TdwP(z|d, w) P d0 P w Td0wP(z|d 0 , w) (9) P(z) = P d P w TdwP(z|d, w) P z 0 P d0 P w0 Td0w0P(z 0 |d 0 , w0) . (10) 5 Task Implement EM for maximum likelihood estimation and cluster the text data provided in the nips.mat file you downloaded. You can print out the top key words for the topics/clusters by using the show topics.m utility. It takes two parameters: 1) your learned conditional distribution matrix, i.e., P(w|z) and 2) a cell array of words that corresponds to the vocabulary. You can find the cell array wl in the nips.mat file. Try different values of k and see which values produce sensible topics. In assessing your code, we will use another dataset and observe the produces topics.1 Linear Regression [30 pts] In class, we derived a closed form solution (normal equation) for linear regression problem: ˆθ = (XT X) −1XT Y . A probabilistic interpretation of linear regression tells us that we are relying on an assumption that each data point is actually sampled from a linear hyperplane, with some noise. The noise follows a zero-mean Gaussian distribution with constant variance. Specifically, Y i = θ T Xi + i (1) where i ∼ N (0, σ2 I), θ ∈ R d , and {Xi , Y i} is the i-th data point. In other words, we are assuming that each every point is independent to each other and that every data point has same variance. 1https://matlab.mathworks.com/ 2You can also use the environment for developing by registering online with your university email. 3Christopher M. Bishop, Pattern Recognition and Machine Learning, 2006, Springer. 1 (a) Using the normal equation, and the model (Eqn. 1), derive the expectation E[ ˆθ]. Note that here X is fixed, and only Y is random, i.e. “fixed design” as in statistics. [6 pts] (b) Similarly, derive the variance Var[ ˆθ]. [6 pts] (c) Under the white noise assumption above, someone claims that ˆθ follows Gaussian distribution with mean and variance in (a) and (b), respectively. Do you agree with this claim? Why or why not? [8 pts] (d) Weighted linear regression Suppose we keep the independence assumption but remove the same variance assumption. In other words, data points would be still sampled independently, but now they may have different variance σi . Thus, the covariance matrix of Y would be still diagonal, but with different values: Σ = σ 2 1 0 . . . 0 0 σ 2 2 . . . 0 . . . . . . . . . . . . 0 0 . . . σ2 n . (2) Derive the estimator ˆθ (similar to the normal equations) for this problem using matrix-vector notations with Σ. [10 pts] 2 Ridge Regression [15 pts] For linear regression, it is often assumed that y = θ >x + where θ, x ∈ R m by absorbing the constant term, and ∼ N (0, σ2 ) is a Gaussian random variable. Given n i.i.d samples (x 1 , y1 ), …,(x n, yn), we define y = (y 1 , …, yn) > and X = (x 1 , …, x n) >. Thus, we have y ∼ N (Xθ, σ2 I). Show that the ridge regression estimate is the mean of the posterior distribution under a Gaussian prior θ ∼ N (0, τ 2 I). Find the explicit relation between the regularization parameter λ in the ridge regression estimate of the parameter θ, and the variances σ 2 , τ 2 . 3 Bayes Classifier 3.1 Bayes Classifier With General Loss Function In class, we talked about the popular 0-1 loss function in which L(a, b) = 1 for a 6= b and 0 otherwise, which means all wrong predictions cause equal loss. Yet, in many other cases including cancer detection, the asymmetric loss is often preferred (misdiagnosing cancer as no-cancer is much worse). In this problem, we assume to have such an asymmetric loss function where L(a, a) = L(b, b) = 0 and L(a, b) = p, L(b, a) = q, p 6= q. Write down the the Bayes classifier f : X → Y for binary classification Y ∈ {−1, +1}. Simplify the classification rule as much as you can. [20 pts] 3.2 Gaussian Class Conditional distribution (a) Suppose the class conditional distribution is a Gaussian. Based on the general loss function in problem 3.1, write the Bayes classifier as f(X) = sign(h(X)) and simplify h as much as possible. What is the geometric shape of the decision boundary? [10 pts] 2 (b) Repeat (a) but assume the two Gaussians have identical covariance matrices. What is the geometric shape of the decision boundary? [10 pts] (c) Repeat (a) but assume now that the two Gaussians have covariance matrix which is equal to the identity matrix. What is the geometric shape of the decision boundary? [10 pts] 4 Logistic Regression Logistic regression is named after the log-odds of success (the logit of the probability) defined as below: ln P[Y = 1|X = x] P[Y = 0|X = x] where P[Y = 1|X = x] = exp(w0 + w T x) 1 + exp(w0 + wT x) (a) Show that log-odds of success is a linear function of X. [6 pts] (b) Show that the logistic loss L(z) = log (1 + exp(−z)) is a convex function. [9 pts] 5 Programming: Recommendation System [40 pts] Personalized recommendation systems are used in a wide variety of applications such as electronic commerce, social networks, web search, and more. Machine learning techniques play a key role to extract individual preference over items. In this assignment, we explore this popular business application of machine learning, by implementing a simple matrix-factorization-based recommender using gradient descent. Suppose you are an employee in Netflix. You are given a set of ratings (from one star to five stars) from users on many movies they have seen. Using this information, your job is implementing a personalized rating predictor for a given user on unseen movies. That is, a rating predictor can be seen as a function f : U × I → R, where U and I are the set of users and items, respectively. Typically the range of this function is restricted to between 1 and 5 (stars), which is the the allowed range of the input. Now, let’s think about the data representation. Suppose we have m users and n items, and a rating given by a user on a movie. We can represent this information as a form of matrix, namely rating matrix M. Suppose rows of M represent users, while columns do movies. Then, the size of matrix will be m × n. Each cell of the matrix may contain a rating on a movie by a user. In M15,47, for example, it may contain a rating on the item 47 by user 15. If he gave 4 stars, M15,47 = 4. However, as it is almost impossible for everyone to watch large portion of movies in the market, this rating matrix should be very sparse in nature. Typically, only 1% of the cells in the rating matrix are observed in average. All other 99% are missing values, which means the corresponding user did not see (or just did not provide the rating for) the corresponding movie. Our goal with the rating predictor is estimating those missing values, reflecting the user’s preference learned from available ratings. Our approach for this problem is matrix factorization. Specifically, we assume that the rating matrix M is a low-rank matrix. Intuitively, this reflects our assumption that there is only a small number of factors (e.g, genre, director, main actor/actress, released year, etc.) that determine like or dislike. Let’s define r as 3 the number of factors. Then, we learn a user profile U ∈ R m×r and an item profile V ∈ R n×r . (Recall that m and n are the number of users and items, respectively.) We want to approximate a rating by an inner product of two length r vectors, one representing user profile and the other item profile. Mathematically, a rating by user u on movie i is approximated by Mu,i ≈ Xr k=1 Uu,kVi,k. (3) We want to fit each element of U and V by minimizing squared reconstruction error over all training data points. That is, the objective function we minimize is given by E(U, V ) = X (u,i)∈M (Mu,i − U T u Vi) 2 = X (u,i)∈M (Mu,i − Xr k=1 Uu,kVi,k) 2 (4) where Uu is the uth row of U and Vi is the ith row of V . We observe that this looks very similar to the linear regression. Recall that we minimize in linear regression: E(θ) = Xm i=1 (Y i − θ T x i ) 2 = Xm i=1 (Y i − Xr k=1 θkx i k ) 2 (5) where m is the number of training data points. Let’s compare (4) and (5). Mu,i in (4) corresponds to Y i in (5), in that both are the observed labels. U T u Vi in (4) corresponds to θ T x i in (5), in that both are our estimation with our model. The only difference is that both U and V are the parameters to be learned in (4), while only θ is learned in (5). This is where we personalize our estimation: with linear regression, we apply the same θ to any input x i , but with matrix factorization, a different profile Uu are applied depending on who is the user u. As U and V are interrelated in (4), there is no closed form solution, unlike linear regression case. Thus, we need to use gradient descent: Uv,k ← Uv,k − µ ∂E(U, V ) ∂Uv,k , Vj,k ← Vj,k − µ ∂E(U, V ) ∂Vj,k , (6) where µ is a hyper-parameter deciding the update rate. It would be straightforward to take partial derivatives of E(U, V ) in (4) with respect to each element Uv,k and Vj,k. Then, we update each element of U and V using the gradient descent formula in (6). (a) Derive the update formula in (6) by solving the partial derivatives. [10 pts] (b) To avoid overfitting, we usually add regularization terms, which penalize for large values in U and V . Redo part (a) using the regularized objective function below. [5 pts] E(U, V ) = X (u,i)∈M (Mu,i − Xr k=1 Uu,kVi,k) 2 + λ X u,k U 2 u,k + λ X i,k V 2 i,k (λ is a hyper-parameter controlling the degree of penalization.) (c) Implement myRecommender.m by filling the gradient descent part. You are given a skeleton code myRecommender.m. Using the training data rateMatrix, you will implement your own recommendation system of rank lowRank. The only file you need to edit and submit is myRecommender.m. In the gradient descent part, repeat your update formula in (b), observing the average reconstruction error between your estimation and ground truth in training set. You need to set a stopping criteria, based on this reconstruction error as well as the maximum number of iterations. You should play with several different values for µ and λ to make sure that your final prediction is accurate. Formatting information is here: 4 Input • rateMatrix: training data set. Each row represents a user, while each column an item. Observed values are one of {1, 2, 3, 4, 5}, and missing values are 0. • lowRank: the number of factors (dimension) of your model. With higher values, you would expect more accurate prediction. Output • U: the user profile matrix of dimension user count × low rank. • V: the item profile matrix of dimension item count × low rank. Evaluation [15 pts] Upload your myRecommender.m implementation file. (Do not copy and paste your code in your report. Be sure to upload your myRecommender.m file.) To test your code, try to run homework3.m. You may have noticed that the code prints both training and test error, in RMSE (Root Mean Squared Error), defined as follows: X (u,i)∈M (Mu,i − f(u, i))2 where f(u, i) is your estimation, and the summation is over the training set or testing set, respectively. For the grading, we will use another set-aside testing set, which is not released to you. If you observe your test error is less than 1.00 without cheating (that is, training on the test set), you may expect to see the similar performance on the unseen test set as well. Note that we provide homework3.m just to help you evaluate your code easily. You are not expected to alter or submit this to us. In other words, we will not use this file when we grade your submission. The only file we take care of is myRecommender3.m. Grading criteria: • Your code should output U and V as specified. The dimension should match to the specification. • We will test your output on another test dataset, which was not provided to you. The test RMSE on this dataset should be at least 1.05 to get at least partial credit. • We will measure elapsed time for learning. If your implementation takes longer than 3 minutes for rank 5, you should definitely try to make your code faster or adjust parameters. Any code running more than 5 minutes is not eligible for credit. • Your code should not crash. Any crashing code will be not credited. Report [10 pts] In your report, show the performance (RMSE) both on your training set and test set, with varied lowRank. (The default is set to 1, 3, and 5, but you may want to vary it further.) Discuss what you observe with varied low rank. Also, briefly discuss how you decided your hyper-parameters (µ, λ). Note • Do not print anything in your code (e.g, iteration 1 : err=2.4382) in your final submission. • Do not alter input and output format of the skeleton file. (E.g, adding a new parameter without specifying its defalut value) Please make sure that you returned all necessary outputs according to the given skeleton. 5 • Please do not use additional file. This task is simple enough that you can fit in just one file. • Submit your code with the best parameters you found. We will grade without modifying your code. (Applying cross-validation to find best parameters is fine, though you do not required to do.) • Please be sure that your program finishes within a fixed number of iterations. Always think of a case where your stopping criteria is not satisfied forever. This can happen anytime depending on the data, not because your code is incorrect. For this, we recommend setting a maximum number of iteration in addition to other stopping criteria. Grand Prize Similar to the Netflix competition held in 2006 to 2009, the student who achives the lowest RMSE on the secret test set will earn the Grand Prize. We will give extra 10 bonus points to the winner, and share the student’s code to everyone. (Note that the winner should satisfy all other grading criteria perfectly, including answer sanity check and timing requirement. Otherwise, the next student will be considered as the winner.) Typing Bonus As usual, we will give 5 bonus points for typed submissions. Note that all questions should be typed to get this credit.1 Kernels [20 points] (a) Identify which of the followings is a valid kernel. If it is a kernel, please write your answer explicitly as ‘True’ and give mathematical proofs. If it is not a kernel, please write your answer explicitly as ‘False’ and give explanations. [8 pts] Suppose K1 and K2 are valid kernels (symmetric and positive definite) defined on Rm × Rm. 1. K(u, v) = αK1(u, v) + βK2(u, v), α, β ∈ R. 2. K(u, v) = K1(f(u), f(v)) where f : Rm → Rm. coefficients. 1https://matlab.mathworks.com/ 2You can also use the environment for developing by registering online with your university email. 1 3. K(u, v) = ( 1 if ku − vk2 6 1 0 otherwise (1) 4. Suppose K0 is a valid kernel. K(u, v) = K0 (u, v) p K0(u, u)K0(v, v) . (2) (b) Write down kernelized version of Fisher’s Linear Discriminant Analysis using kernel trick. Please provide full steps and all details of the method. [Hint: Use kernel to replace inner products.] [12 pts] 2 Markov Random Field, Conditional Random Field [20 pts] [a-b] A probability distribution on 3 discrete variables a,b,c is defined by P(a, b, c) = 1 Z ψ(a, b, c) = 1 Z φ1(a, b)φ2(b, c), where the table for the two factors are given below. a b φ1(a, b) 0 0 4 0 1 3 1 0 3 1 1 1 b c φ2(b, c) 0 0 3 0 1 2 0 2 1 1 0 4 1 1 1 1 2 3 (a) Compute the slice of the joint factor ψ(a, b, c) corresponding to b = 1. This is the table ψ(a, b = 1, c). [5 pts] (b) Compute P(a = 1, b = 1). [5 pts] (c) Explain the difference between Conditional Random Fields and Hidden Markov Models with respect to the following factors. Please give only a one-line explanation. [10 pts] • Type of model – generative/discriminative • Objective function optimized • Require a normalization constant 3 Hidden Markov Model [50 pts] This problem will let you get familiar with HMM algorithms by doing the calculations by hand. [a-c] There are three coins (1, 2, 3), to throw them randomly, and record the result. S = 1, 2, 3; V = H, T (Head or Tail); A, B, π is given as 2 A: 1 2 3 1 0.9 0.05 0.05 2 0.45 0.1 0.45 3 0.45 0.45 0.1 B: 1 2 3 H 0.5 0.75 0.25 T 0.5 0.25 0.75 π : π 1/3 1/3 1/3 (a) Given the model above, what’s the probability of observation O = H, T, H. [10 pts] (b) Describe how to get the A, B, and π, when they are unknown. [10 pts] (c) In class, we studied discrete HMMs with discrete hidden states and observations. The following problem considers a continuous density HMM, which has discrete hidden states but continuous observations. Let St ∈ 1, 2, …, n denote the hidden state of the HMM at time t, and let Xt ∈ R denote the real-valued scalar observation of the HMM at time t. In a continuous density HMM, the emission probability must be parameterized since the random variable Xt is no longer discrete. It is defined as P(Xt = x|St = i) = N (µi , σ2 i ). Given m sequences of observations (each of length T), derive the EM algorithm for HMM with Gaussian observation model. [14 pts] (d) For each of the following sentences, say whether it is true or false and provide a short explanation (one sentence or so). [16 pts] • The weights of all incoming edges to a state of an HMM must sum to 1. • An edge from state s to state t in an HMM denotes the conditional probability of going to state s given that we are currently at state t. • The ”Markov” property of an HMM implies that we cannot use an HMM to model a process that depends on several time-steps in the past. • The Baum-Welch algorithm is a type of an Expectation Maximization algorithm and as such it is guaranteed to converge to the (globally) optimal solution. 4 Programming [30 pts] In this problem, you will implement algorithm to analyze the behavior of SP500 index over a period of time. For each week, we measure the price movement relative to the previous week and denote it using a binary variable (+1 indicates up and 1 indicates down). The price movements from week 1 (the week of January 5) to week 39 (the week of September 28) are plotted below. Consider a Hidden Markov Model in which xt denotes the economic state (good or bad) of week t and yt denotes the price movement (up or down) of the SP500 index. We assume that x(t+1) = xt with probability 0.8, and P(Yt|Xt)(yt = +1|xt = good) = P(Yt|Xt)(yt = −1|xt = bad) = q. In addition, assume that P(X1)(x1 = bad) = 0.8. Load the sp500.mat, implement the algorithm, briefly describe how you implement this and report the following : (a) Assuming q = 0.7, plot P(Xt|Y )(xt = good|y) for t = 1, 2, …, 39. What is the probability that the economy is in a good state in the week of week 39. [15 pts] (b) Repeat (a) for q = 0.9, and compare the result to that of (a). Explain your comparison in one or two sentences. [15 pts]
CN Y RD NGLSH WTHT VWLS? In this assignment, you will build a finite-state machine to automatically restore vowels to vowelless text. (In English, this may not be a very useful thing to do, but in languages like Arabic and Hebrew where vowels are regularly omitted (cf. abjad1 ), it is extremely useful.) Before you begin • Finish EX1 individually. • (Optional) Read Sections 1 and 2 of “A Primer on Finite-State Software for Natural Language Processing” (https://www.isi.edu/licensed-sw/carmel/carmel-tutorial2.pdf). A brief overview of some of Carmel’s command-line options (run carmel to see complete list): -si expect a string on stdin -l compose stdin onto the left of the named FSA/FST(s) -r compose stdin onto the right of the named FSA/FST(s) -O print only output labels, suppress input labels -I print only input labels, suppress output labels -k n list k sequences rather than the whole FSA/FST -b batch mode: process multiple input lines -WE suppress weights and empty labels in k-best lists • Download https://classes.engr.oregonstate.edu/eecs/fall2019/cs539-001/hw1/hw1-data.tgz: vocab list of English words vocab.small shorter list (for testing purposes only) strings English sentences strings.bad English sentences with bad spelling eval.py script for measuring accuracy 1 Finite-state acceptors 1. I’ve created an example FSA try.fsa for sequences consisting of the three English words: an, at, and age, separated by an underscore between consecutive words, and nothing else. It was done using a simple prefix tree idea. You can test it like this: echo “A T _ A N _ A G E” | carmel -slibOWE try.fsa Based on this example, try to create (by hand) the FSA small.fsa for words in vocab.small. Include a drawing/illustration of this FSA in your PDF report. 2. Write a python program make.py to create an FSA english.fsa that accepts all strings consisting of English words (as defined by vocab) separated by one underscore between consecutive words, and nothing else. The command-line of your python program should be like this: cat vocab | python make.py > english.fsa (FYI my make.py has only 25 lines.) The FSA should be letter-based, not word-based: that is, transitions should be labeled with letters, not whole words. For example, it should accept: 1See https://en.wikipedia.org/wiki/Abjad. 1 T H I S _ I S _ A _ S T R I N G (a) How many states and how many transitions does your FSA have? (Use carmel -c english.fsa.) (b) Verify that your FSA accepts every line in strings and no lines in strings.bad. Show the output of Carmel on the first five lines of each file. You can use commands like: cat strings | carmel -slibOWE english.fsa Hint: you should have either ∼250k or ∼180k states (and ∼360k transitions or less). No need to minimize it. 2 Finite-state transducers 3. Create a FST in Carmel format called remove-vowels.fst that deletes all English vowels, preserving word boundary information. For the purposes of this assignment, vowels are defined to be members of {A, E, I, O, U}. For example, it should perform the following mappings: Y O U _ A R E _ H E R E ↔ Y _ R _ H R R E A D _ A _ B O O K ↔ R D _ _ B K (a) Draw your FST in your PDF report in enough detail for someone else to replicate it. (b) Test your FST in the forward direction on strings with the following command: cat strings | carmel -slibOEWk 1 remove-vowels.fst Show the output on the first five lines, and save the whole output to a file strings.novowels. (c) Test your FST in the backward direction with the following command: echo “B L D N G” | carmel -sriIEWk 10 remove-vowels.fst The -k 10 option asks for up to 10 output strings. Just list them. 4. Now you can use backwards application of your FST to do vowel restoration. (a) Run your vowel restorer on strings.novowels, using the following command: cat strings.novowels | carmel -sribIEWk 1 remove-vowels.fst Show the output on the first five lines, and save the whole output to a file strings.restored. (b) Compute your vowel-restoration accuracy using the command: python eval.py strings strings.restored What was your accuracy? (c) Why is the score so low? Hint: should be around 1.3%. 3 Combining FSAs and FSTs 5. The vowel restorer you built in the previous part had a problem. Can you fix it by combining your FSA and FST? (a) Describe how to combine your FSA and FST to improve vowel restoration. (b) Implement your idea and test it on strings.novowels. What is your new accuracy? (c) Are the results satisfactory? Why doesn’t the machine do what a human would do? Hint: should be around 30%. 6. How might your vowel restorer be further improved? Come up with at least one idea, and for each idea: (a) Describe it in enough detail so that someone else could replicate it. (b) Implement it and try it on strings.novowels. (c) Report your new accuracy on strings.novowels. You will be graded on your final accuracy. Hint: very easy to get above 90%. Submit your code (*.py), your FSAs/FSTs (*.fsa, *.fst), your outputs (strings.*), and report.pdf.In Japanese text, foreign words borrowed in the modern era (such as Western names and technical terms) are transliterated into special symbols called katakana. Katakana is a syllabary, 1 in which most symbols stand for syllable sounds (such as ko or ra). Because spoken Japanese consists largely of consonant-vowel syllables, most katakana symbols stand for two Japanese phonemes. In this assignment, among other things, we’ll decode katakana words of English origin back into English. Refer to the slides for this section and the tutorial on writing systems and transliteration for more information (available from the course website). Download https://classes.engr.oregonstate.edu/eecs/fall2019/cs539-001/hw2/hw2-data.tgz which includes: eword.wfsa a unigram WFSA of English word sequences epron.wfsa a trigram WFSA of English phoneme sequences eword-epron.data an online dictionary of English words and their phoneme sequences eword-epron.wfst a WFST from English words to English phoneme sequences epron-eword.wfst inverse transducer; the result of carmel -v eword-epron.wfst epron-espell.wfst a WFST from English phoneme sequences to English letter sequences epron-jpron.data a database of aligned English/Japanese phoneme sequence pairs jprons.txt a short list of Japanese Katakana sounds to decode epron.probs a human-readable version of epron.wfsa. 1 Shannon Game and Entropy of English (5 pts) 1. Each team member plays the game once.2 Report the entropy from each time (take a screenshot). 2. Explain how the entropy was calculated in this game (with equations). 2 Part-of-Speech Tagging as WFST Composition (10 pts) Recall that in the lectures we did POS tagging for “I hope that this works” and “They can fish” by composing a chain (word sequence), a flower (word/tag lexicon), and a tag bigram. Now you need to 1. build a Carmel WFSA bigram.wfsa for the tag bigram model (similar to the one from the slides, but add PREP and AUX for prepositions and aux. verbs); you can assign probabilities in any reasonable way. 2. build a Carmel WFST lexicon.wfst for the word/tag lexicon (big enough to cover the examples below); again, you can assign probabilities in any reasonable way, but make sure your pipeline is mathematically sound! 3. compose them to solve POS tagging for the above two sentences plus the following examples (show the resulting tag sequences from Carmel output): 1See https://en.wikipedia.org/wiki/Katakana and https://en.wikipedia.org/wiki/Syllabary. 2Try https://classes.engr.oregonstate.edu/eecs/fall2019/cs539-001/shannon.tgz and run appletviewer index.html. 1 (a) A panda eats shoots and leaves (b) They can can a can (c) Time flies like an arrow 4. Why is your composition pipeline mathematically sound? Write the equations. 5. What funny interpretations did you observe? What are possible ways to fix them? (no need to implement them) 3 Pronouncing and Spelling English (25 pts) 1. Pick some English words and pronounce them with eword-epron.wfst. This transducer is essentially a giant lookup table built from eword-epron.data. Turn in five (5) examples. Here is one: echo HELLO | carmel -sliOEQk 5 eword-epron.wfst 2. Now let’s try to pronounce character sequences without whole-word lookup (since we may face unknown words, for example). You can send a character string backwards through epron-espell.wfst. Try several words and turn in your examples; here is a suggested command: echo ‘H E L L O’ | carmel -sriIEQk 5 epron-espell.wfst Did you get some non-sense output? Why? Explain it in terms of probabilitic modeling. 3. Let’s fix the nonsense by adding a language model of likely pronunciations, epron.wfsa. Try several and turn in your examples. echo ‘H E L L O’ | carmel -sriIEQk 5 epron.wfsa epron-espell.wfst 4. Now spell out some phoneme sequences by using epron-espell.wfst in the forward direction. Try several and turn in your examples. echo ‘HH EH L OW’ | carmel -sliOEQk 50 epron-espell.wfst 5. How to improve the above results? Well, you can try to take advantage of the language model eword.wfsa (as a filter). But you need a “bridge” from espell to eword. That’s trivial, isn’t it? Write a simple Python program gen_espell_eword.py to generate espell-eword.wfst from the word list in eword-epron.data. And now you can: echo ‘HH EH L OW’ | carmel -sliOEQk 50 epron-espell.wfst espell-eword.wfst eword.wfsa Try several examples and turn them in. Include gen_espell_eword.py and espell-eword.wfst in the submission also. (this only works for words in both espell-eword.wfst and eword.wfsa.) 6. Well, alternatively, you can go directly from epron to eword: echo ‘HH EH L OW’ | carmel -sliOEQk 50 epron-eword.wfst eword.wfsa Is this better than 5)? Try the same set of examples can compare the results with those from 5). 7. Try to pronounce some words that are not in the data file eword-espell.data. Try several examples and turn them in. echo WHALEBONES | carmel -sliOEQk 5 eword-epron.wfst 2 8. Now try pronouncing letter sequences of those same words. Try several examples and turn them in. echo ‘W H A L E B O N E S’ | carmel -sriIEQk 5 epron.wfsa epron-espell.wfst 9. How about phoneme sequences of new words? What happens when you use eword-epron.wfst versus epron-espell.wfst? Try several examples and turn them in. echo ‘W EY L B OW N Z’ | carmel -sriIEQk 5 eword-epron.wfst echo ‘W EY L B OW N Z’ | carmel -sliOEQk 5 epron-espell.wfst 10. You can hook up these transducers in unexpected ways. What is the following command trying to accomplish? echo ‘BEAR’ | carmel -sliOEQk 10 eword-epron.wfst epron-eword.wfst eword.wfsa 11. Write up a half-page or so of observations from your experiments. What did you learn about these automata and what they can (or can’t) do? Please take time to express your thoughts clearly using complete sentences. 4 Decoding English Words from Japanese Katakana (80 pts) 1. Look at epron-jpron.data. Each pair in this file consists of an English phoneme sequence paired with a Japanese phoneme sequence. For each pair, Japanese phonemes are assigned integers telling which English phonemes map to them. For example: AE K T ER ;; English phoneme sequence for ‘actor’ A K U T A A ;; Same word, loaned into Japanese 1 2 2 3 4 4 ;; e.g., Japanese T maps to the 3rd English sound From the data, we can see that each English phoneme maps to one or more Japanese phonemes. Estimate the channel probabilities p(j | s) where j represents one or more Japanese phoneme(s), and s an English phoneme, using the simplest maximum likelihood estimation (MLE). Write a Python program estimate.py that generates the file epron-jpron.probs like this (each line represents one p(j | s), and you can omit those with probability < 0.01 and you can omit those where one English phoneme maps to more than three (3) Japanese phonemes since they are mostly noise): AE : A # 0.95 AE : Y A # 0.05 T : T # 0.4 T : T O # 0.4 T : TT O # 0.1 T : TT # 0.1 R : A A # 0.8 R : E R # 0.1 R : E R U # 0.1 … Transitions for each English phoneme should be consecutive (see above). Include estimate.py and epron-jpron.probs. 2. Now, based on this, you can create a WFST called epron-jpron.wfst. It should probabilistically map English phoneme sequences onto Japanese ones, behaving something like this in the forward direction: echo ‘L AE M P’ | carmel -sliOEQk 5 epron-jpron.wfst 3 and you’ll get a list like R A M P R U A M P R A M U P R A M P U R A M PP U … Most transducers will consist of 300 transitions or so. 3. Using your knowledge about Japanese syllable structure (see slides), do these results (including their rankings) make sense? If not, why (briefly explain)? 4. How could you improve the results to sound more “Japanese like”? (no need to implement it) 5. Now consider the following Japanese katakana sequences, from a Japanese newspaper (actually these are the phoneme sequences that are easily read off from the katakana characters): H I R A R I K U R I N T O N D O N A R U D O T O R A N P U B I D E O T E E P U H O M A A SH I N P U S O N R A PP U T O PP U SH E E B I N G U K U R I I M U CH A I R U D O SH I I T O SH I I T O B E R U T O SH I N G U R U R U U M U G A A R U H U R E N D O T O R A B E R A A Z U TCH E KK U B E B I I SH I TT A A S U K O TT O R A N D O B A I A R I N K O N TCH E R U T O A PP U R U M A KK U B U KK U P U R O K O N P I U U T A S A I E N S U H I J I K A R U T O R E E N I N G U H I J I K A R U E K U S A S A I S U A I S U K U R I I M U H O TT O M I R U K U T O R I P U R U R U U M U K U R A U N P U R A Z A H O T E R U H E E S U B U KK U R I S A A TCH I S A I E N T I S U T O W O R U H U G A N G U M O TS U A R U T O (For your convenience I have included a jprons.txt for the above sequences.) How would you decode them into English phoneme sequences using carmel (with the WFST you just built and with help of epron.wfsa)? Show the command-line and results (with probs, with -k 5 option). Do they make sense (as English words)? 6. Now redo the above exercise, but this time into English words by assembling eword.wfsa, eword-epron.wfst, and epron-jpron.wfst with carmel. Show the command-line and results (with probs, with -k 5). Do you think the results make more sense this time? Briefly explain. 7. Some desired outputs were not ranked top. Why? How would you improve the results? (No need to implement any of these) 8. Try search for at least five (5) other Japanese Katakana examples not covered in this HW or in slides, and try to decode them back into English using the above approach. Try to be creative. They should preferably be a two word phrase or a compound word, like “shopping center” or ”piano sonata”. You can use Google Translate (from English to Katakana) to verify your results. 9. Are there other ways to decode into English words given all these existing WFSAs and WFSTs (with some tweaking)? What are the differences of these methods (speed, accuracy, etc.)? 10. If you can make some new WFSAs/WFSTs such as espell.wfsa and espell-epron.wfst, what else can you do to decode jprons.txt (not necessarily to English words)? (No need to implement these) Submit a single file: hw2.zip, which includes *.{wfst,wfsa,probs,py} and separate report.pdf.In HW2, we did POS tagging, English pronunciation and spelling, and Katakana-to-English backtransliteration, all using Carmel. In this HW, you will implement the famous Viterbi algorithm used by Carmel for all these decodings; in doing so, you will implicitly implement “composition” as well. In other words, you can view this HW as the “under-the-hood” version of HW2. You will need again all the files we provided for HW2: eword.wfsa a unigram WFSA of English word sequences epron.wfsa a trigram WFSA of English phoneme sequences eword-epron.data an online dictionary of English words and their phoneme sequences eword-epron.wfst a WFST from English words to English phoneme sequences epron-eword.wfst inverse transducer; the result of carmel -v eword-epron.wfst epron-espell.wfst a WFST from English phoneme sequences to English letter sequences epron-jpron.data a database of aligned English/Japanese phoneme sequence pairs jprons.txt a short list of Japanese Katakana sounds to decode epron.probs a human-readable version of epron.wfsa. In addition, you will also need epron-jpron.probs and epron-jpron.wfst from your HW2 submission. But in case you didn’t get everything correctly, we have provided our reference solutions in https://classes.engr.oregonstate.edu/eecs/fall2019/cs539-001/hw3/hw3-data.tgz. (Note that all expected outputs are in the 5-bests, and only in a few cases the 1-bests are confused with similar-sounding words e.g., SHEET vs. SEAT, SAVING vs. SHAVING, MAKE BOOK vs. MAC BOOK, etc.). 1 Part-of-Speech Tagging (10 pts) Redo the POS tagging examples by implementing a bigram Viterbi algorithm (name it tagging.py) that would output the most probable POS tag sequence (and its probability) for an input sentence. Verify your results with Carmel outputs from HW2. After finishing bigram tagging, please work on trigram tagging, which paves the way for Part 2. This question is graded by completeness, not by correctness. 2 Decoding Katakana to English Phonemes (35 pts) Now implement your own Viterbi decoding in Python for HW2 Question 3.5 (jprons to eprons). Your command-line should be: echo -e ‘P I A N O N A I T O’ | ./decode.py epron.probs epron-jpron.probs with the result like: (note the input represents the Japanese katakana PI A NO, not the English word!) P IY AA N OW # 1.489806e-08 N AY T # 8.824983e-06 1 For your convenience, we have converted epron.wfsa into a human-readable epron.probs, in the same format as epron-jpron.probs above, e.g.: T S : # 0.774355 T S : AH # 0.0333625 T S : Z # 2.92651e-07 which correspond to our intuition that sounds T S (words with -ts) are very likely to end a sentence or a word, and sounds T S AH are much less likely (but still noticeable in words like WATSON and BOTSON), but T S Z is (almost) completely unheard of (with a tiny prob here due to smoothing). You only need to print the 1-best solution and its probability. Please 1. Describe your algorithm in English first. 2. Define the subproblem and recurrence relations. 3. Analyse its complexity. 4. Implement it in Python. Your grade will depend on both correctness and efficiency. Note that each English phoneme corresponds to one to three Japanese phonemes (the PIANO example is 1-1 and is too simple, and the NIGHT/KNIGHT example is more interesting), so the Viterbi algorithm from the slides needs to be extended a little bit. Compare your 1-best results with carmel’s. Try your best to match them. Include a side-by-side comparison (hint: diff -y). Hint: if you want to make sure your program is 100% correct with respect to Carmel, you can compare their results by: cat jprons.txt | ./decode.py epron.probs epron-jpron.probs > results.my cat jprons.txt | carmel -bsriIEQk 1 epron.wfsa epron-jpron.wfst 2>/dev/null | awk ‘{for (i=1;i results.carmel diff -b results.my results.carmel You should see no output if you did everything correctly. To generate a side-by-side comparison, replace diff -b by diff -by. FYI, my not-very-optimized implementation is 60 lines of Python, and uses about 4x 2x time compared to Carmel. Your implementation will be graded in terms of both correctness and efficiency. 3 K-Best Output (15 pts) (kbest.py) Now extend your Viterbi algorithm to output k-best sequences (and their corresponding probabilities). Implement it on POS tagging and Katakana-to-English. Verify the results with Carmel using the diff method above (kbest.py should agree with Carmel output for k=1..10). Efficiency is part of the grade. Hint: see Huang and Chiang (2005). Algorithm 2 is enough to receive full credit for this part. 4 Extra Credit: Decode Katakana to English WORDS (30 pts) (Do NOT attempt this problem until you finish everything else. This is the hardest problem in this course.) Now if you are to decode with HW2 approach using eword.wfsa and eword-epron.wfsa and epron-jpron.wfsa. 1. Describe your algorithm in English first. 2. Define the subproblem and recurrence relations. 3. Analyse its complexity. 4. Implement decode_word.py. echo -e ‘P I A N O N A I T O’ | ./decode_word.py eword.probs eword-epron.data epron-jpron.probs PIANO NIGHT # 1.856048e-11 Here eword.probs is just a human-readable format of eword.wfsa. Verify your results with Carmel on jprons.txt. Just 1-best is more than enough! UPDATE: My code is orders of magnitude faster than Carmel, thanks to the trie. (one diff: BABYSITTER is in eword-epron.data but not in eword-epron.wfst)This homework is about unsupervised learning (using the EM algorithm) of the alignment between English and Japanese phonemes in English-Katakana pairs. Recall from HW2/3: AE K T ER ;; English phoneme sequence for ’actor’ A K U T A A ;; Same word, loaned into Japanese 1 2 2 3 4 4 ;; e.g., Japanese T maps to the 3rd English sound But in this homework, you need to learn the alignments yourself. In fact, those alignments we gave you in HWs 2–3 were also learned by EM, and thus are not 100% correct. You will reuse the following files: epron-jpron.data a database of aligned English/Japanese phoneme sequence pairs ignore the aligments – your job is to learn them! eword.wfsa a unigram WFSA of English word sequences eword-epron.wfst a WFST from English words to English phoneme sequences jprons.txt a short list of Japanese Katakana sounds to decode 1 Complete Data, Incomplete Model (10 pts) If we’re given the following manually aligned pairs (English “boat” and “test”): B OW T T EH S T B O O T O T E S U T O 1 2 2 3 3 1 2 3 3 4 4 What is the maximum likelihood estimate for p(T | T) and p(T O | T) ? 2 Complete Model, Incomplete Data (10 pts) Given the following conditional probabilities, p(B | B) = 1 p(O O | OW) = 0.8 p(O | OW) = 0.2 p(T | T) = 0.3 p(T O | T) = 0.5 p(O | T) = 0.1 p(O T O | T) = 0.1 Enumerate (by hand) all legal alignments for the “boat” example above, and compute the probability and normalized probability of each alignment. Which one is the Viterbi alignment? 3 EM, Implementation I: Enumerating All Alignments (70 pts) Now implement the EM algorithm: 1. initialize the conditional probabilities to uniform 2. repeat: 3. E-step: 4. for each English-Katakana pair: 5. enumerate all legal alignments for that E-K pair, along with their respective probabilities 6. renormalize the probabilities to get a distribution over these alignments 1 7. collect the fractional counts for this E-K pair 8. M-step: count-and-divide based on the collected fractional counts from all pairs to get new prob. table 9. until reached maximum iteration or converged You should proceed with the following steps: 1. (15 pts) to start with, you should work on the simplest scenario (“wine” example): W AY N / W A I N. Print the following information in each iteration: 1) the corpus probability, 2) the new prob. table (>= 0.01). 3) the number of nonzero entries in the prob. table (>= 0.01). Note: the corpus probibility should increase in each iteration (we have the proof), and the number of nonzero entries should decrease (as the model gets peakier). For example, you should run echo -e “W AY N W A I N ” | ./em.py 5 and get the following (note that the data should be read in from standard input and 5 is the number of iterations): iteration 0 —– corpus prob= 0.00411522633745 AY|-> A: 0.33 A I: 0.33 I: 0.33 W|-> W: 0.67 W A: 0.33 N|-> N: 0.67 I N: 0.33 nonzeros = 7 iteration 1 —– corpus prob= 0.296296296296 AY|-> A I: 0.50 A: 0.25 I: 0.25 W|-> W: 0.75 W A: 0.25 N|-> N: 0.75 I N: 0.25 nonzeros = 7 … iteration 4 —– corpus prob= 0.912659654395 AY|-> A I: 1.00 W|-> W: 1.00 N|-> N: 1.00 nonzeros = 3 For your debugging convenience you can also print all possible alignments and their unnormalized and normalized probabilities in each iteration. Include the result of 5 iterations in your report. 2. (5 pts) If you add N AY N / N A I N to it you should see it converge faster. Show the results. 3. (5 pts) Now that you have passed “wine”, try the “test” example above (T EH S T / T E S U T O) and you’ll see that EM converges to something wrong. Show me the result of 5 iterations. Now come up with a minimal second example so that when combined with the “test” example, EM will learn the correct alignments. This second pair has to be a real English word, and should be choosen from epron-jpron.data. 4. (8 pts) Now come up with an example dataset where EM does not learn anything and is still ambivalent at the end (i.e., the final probability table is (almost) the same as the initial one). In the lecture we gave B IY / B I I as one such example, but can you come up with something bigger, i.e., longer sequences and multiple, interdependent pairs? 5. (15 pts) If you have passed all previous questions, it’s now time to work on the whole dataset with the following command-line: cat epron-jpron.data | ./em.py 15 >epron-jpron.probs 2>epron-jpron.logs (Ignore the alignment lines in the input – your job is to learn them yourself!) The logs file contains the logging info for all iterations (see above), and print the final prob. table to the .probs file. Theis file should follow the same format as in HW3 (ignore < 0.01 entries). Note: Do not reestimate probs from the final Viterbi alignments – you’ll get HW3 probs that way. Hint: The corpus probs are likely to underflow (beyond the range of Python float). Try fix it. 2 6. (7 pts) Decode jprons.txt from HW3 using your newly learned epron-jpron.probs. Compare with your results from HW3: did you see any differences? 7. (5 pts) Generate Viterbi alignments from epron-jpron.probs: cat epron-jpron.data | ./viterbi_align.py epron-jpron.probs >epron-jpron.viterbi The result file should follow the same format as epron-jpron.data so that we can compare them. 8. (5 pts) Calculate the number of different viterbi alignments between your epron-jpron.viterbi and epron-jpron.data. It should be less than 10 (out of 2684 examples). 4 EM, Implementation II: Forward-Backward (DP) (30 pts) The enumeration of all alignments is not the best idea, since in the worst case, there are exponentially many such alignments per example. 1. (5 pts) BTW how many possible alignments are there for a pair of n English phonemes and m Japanese phonemes (n ≤ m)? What pair(s) in epron-jpron.data give the highest number of alignments? 2. (20 pts) Implement the forward-backward algorithm (replacing lines 5–7 in the pseudocode above) so that you can do dynamic programming to collect fractional counts without enumerating all alignments. Make sure the two approaches match on their results. Include a detailed pseudocode of your implementation. What is the complexity of this forwardbackward algorithm for a (n, m) pair? Try optimize your program (em2.py). How does it compare to the enumeration approach in terms of speed? Part of the grade will depend on efficiency. (5 pts pseudocode/complexity, 10 pts correctness of implementation, 5 pts efficiency). 3. (5 pts) Try to construct an example so that em2.py is substantially faster than em.py and explain why. 5 EM for Decipherment (30 pts) Can you decode this? Hint: letter substitution cipher (i.e., permutation of 27 chars). gjkgcbkycnjpkovryrbgkcrvsczbkyhkahqvykgjvjkcpekrbkjxdjayrpmkyhkmhkyhkyvrcukrpkbjfjvcukihpygb oqykcykujcbykhpjkejihavcyrakvjmrijkjxrbybkrpkjfjvzkiclhvkarfrurtcyrhpkhvkaquyqvjkrpkygjkshvue auqobkrpkvjajpykzjcvbkgcfjkwhqpekohvcbkbhkerwwraquykyhkpjmhyrcyjksrygkygcykicpzkbgqpkgrbkaurjpyb gjkbcrekygjkoqvjcqkiqbykgcfjkygjkerbdqyjkvjbhufjekozkwjovqcvzkrpkhvejvkyhkdjvwhvikygjkajpbqb oqykygjkdqouraryzkbqvvhqperpmkygjkejocaujkajvycrpuzkbjvfjekyhkwhaqbkckbdhyurmgykhpkyghbjkjwwhvyb (Wait: how come there is no space at all, given that space also participated in the permutation??) Using a bigram model trained on Ex2’s train.txt with 50 iterations should be good enough. Hint: as shown in slides, start with a uniform model p(c | e) = 1/27 for all c, e: (the LM does not change!) p(c1 . . . cn) = X e1…en p(c1 . . . cn, e1 . . . en) = X e1…en p(e1 . . . en) · p(c1 . . . cn | e1 . . . en) (1) = X e1…en p(e1 . . . en) · p(c1 | e1)· · · p(cn | en) (2) = X e1…en p(e1 | )· · · p(ei | ei−1)· · · p( | en) · p(c1 | e1)· · · p(cn | en) (3) After each iteration, print some debugging information like these (probs ≤ 0.01 are considered zeros): $ cat cipher.txt | ./decipher2.py train.txt 50 epoch 1 logp(corpus)= -2270.05 entropy= 4.79 nonzeros= 567 … epoch 50 logp(corpus)= -1606.14 entropy= 3.39 nonzeros= 76 notice the initial entropy is similar to 0-gram’s, and the final entropy similar to 2-gram’s (see LM slides). why? Optional: Can you do trigram? (should be better: the stronger your LM is, the better you can decipher).In this assignment you will 1. train a probabilistic context-free grammar (PCFG) from a treebank (need binarization); 2. build a simple CKY parser (which handles unary rules), and report its parsing accuracy on test data. Download https://classes.engr.oregonstate.edu/eecs/fall2019/cs539-001/hw5/hw5-data.tgz: corpora: train.trees training data, one tree per line test.txt test data, one sentence per line test.trees gold-standard trees for the test data (for evaluation) scripts: tree.py Tree class and reading tool evalb.py evaluation script 1 Learning a grammar (50 pts) The file train.trees contains a sequence of trees, one per line, each in the following format: (TOP (S (NP (DT the) (NN boy)) (VP (VB saw) (NP (DT a) (NN girl))))) (TOP (S (NP (PRP I)) (VP (VB need) (S (VP (TO to) (VP (VB arrive) (ADVP (RB early)) (NP (NN today)))))))) (TOP (SBARQ (WHNP (WDT what) (NN time)) (SQ (VBZ is) (NP (PRP it))))) N.B. tree.py provides tools for you to read in these trees from file (building a Tree object from a string). TOP S VP NP NN girl DT a VB saw NP NN boy DT the TOP S VP S VP VP NP NN today ADVP RB early VB arrive TO to VB need NP PRP I TOP SBARQ SQ NP PRP it VBZ is WHNP NN time WDT what Note that these examples are interesting because they demonstrate • unary rules on preterminals (POS tags) like NP -> PRP and NP -> NN; 1 • unary rules on nonterminals (constituents) like S -> VP and TOP -> S; • ternary rules like VP -> VB ADVP NP (so that you need binarization). • NP can have (at least) three roles: subject (the boy), object (a girl), and temporal-adverbial (today). The Penn Treebank does annotate these roles (e.g., NP-SBJ, NP-TMP), but for simplicity reasons we removed them for you. • a sentence can be a declarative sentence (S) or a wh-question (SBARQ), among others. Your job is to learn a probabilistic context-free grammar (PCFG) from these trees. Q1.a Preprocessing: replace all one-count words (those occurring only once, case-sensitive) with : cat train.trees | python replace_onecounts.py > train.trees.unk 2> train.dict where train.trees.unk is the resulting trees with symbols, and train.dict is a file of words which appear more than once (one word per line). Explain why we do this. Q1.b Binarization: we binarize a non-branching tree (recursively) in the following way: A B C D ⇒ A A’ C D B e.g. VP VB NP PP ADVP ⇒ VP VP’ VP’ PP ADVP NP VB cat train.trees.unk | python binarize.py > train.trees.unk.bin What are the properties of this binarization scheme? Why we would use such a scheme? (What are the alternatives, and why we don’t use them?) Is this binarized CFG in Chomsky Normal Form (CNF)? Q1.c Learn a PCFG from trees: cat train.trees.unk.bin | python learn_pcfg.py > grammar.pcfg.bin The output grammar.pcfg.bin should have the following format: TOP TOP -> S # 1.0000 S -> NP VP # 0.8123 S -> VP # 0.1404 VP -> VB NP # 0.3769 VB -> saw # 0.2517 … where the first line (TOP) denotes the start symbol of the grammar, and the number after # in each rule is its probability, with four digits accuracy. We group rules into three categories: binary rules (A -> B C), unary rules (A -> B), and lexical rules (A -> w). How many rules are there in each group? 2 2 CKY Parser (70 pts) Now write a CKY parser that takes a PCFG and a test file as input, and outputs, for each sentence in the test file, the best parse tree in the grammar. For example, if the input file is the boy saw a girl I need to arrive early today the output (according to some PCFG) could be: (TOP (S (NP (DT the) (NN boy)) (VP (VB saw) (NP (DT a) (NN girl))))) (TOP (S (NP (PRP I)) (VP (VB need) (S (VP (TO to) (VP (VB arrive) (ADVP (RB early)) (NP (NN today)))))))) Q2.a Design a toy grammar toy.pcfg and its binarized version toy.pcfg.bin such that the above two trees are indeed the best parses for the two input sentences, respectively. How many strings do these two grammars generate? Q2.b Write a CKY parser. Your code should have the following input-output protocol: cat toy.txt | python cky.py toy.pcfg.bin > toy.parsed Verify that you get the desired output in toy.parsed. Note that your output trees should be debinarized (see examples above). Q2.c Now that you passed the toy testcase, apply your CKY parser to the real test set: cat test.txt | python cky.py grammar.pcfg.bin > test.parsed Your program should handle (any levels of) unary rules correctly, even if there are unary cycles. How many sentences failed to parse? Your CKY code should simply output a line NONE for these cases (so that the number of lines in test.parsed should be equal to that of test.txt). What are main reasons of parsing failures? Q2.d Now modify your parser so that it first replaces words that appear once or less in the training data: cat test.txt | python cky.py grammar.pcfg.bin train.dict > test.parsed.new Note that: • train.dict should be treated as an optional argument: your CKY code should work in either case! (and please submit only one version of cky.py). • your output tree, however, should use the original words rather than the symbol. Like binarization, it should be treated as internal representation only. Now how many sentences fail to parse? Q2.e Evaluate your parsing accuracy by python evalb.py test.parsed test.trees python evalb.py test.parsed.new test.trees Report the labeled precisions, recalls and F-1 scores of the two parsing results. Do their differences make sense?1 Playing with the NLM (0 pts) You can use the NLM to evaluate the probability of a given sequence: >>> NLM.load(‘large’) # load neural model from file; choose ‘base’, ‘large’, or ‘huge’ >>> h = NLM() # initialize a state (and observing ) >>> p = 1 >>> for c in ‘t h e _ e n d _ ‘.split(): >>> print(p, h) # cumulative prob and current state (and the distribution of the next char) >>> p *= h.next_prob(c) # include prob (c | …) >>> h += c # observe another character (changing NLM state internally) 1.000 ““: [t: 0.19, i: 0.10, a: 0.09, m: 0.07, s: 0.06, h: 0.06, c: 0.05, b: 0.04, p: 0.04, f: 0.04, r: 0.03, o: 0.03, d: 0.03, w: 0.03, e: 0.03, n: 0.03, l: 0.02, k: 0.02, g: 0.02, j: 0.01] 0.189 “ t”: [h: 0.82, o: 0.09, r: 0.03, e: 0.02, i: 0.01, w: 0.01] 1 0.156 “ t h”: [e: 0.88, i: 0.07, a: 0.02, r: 0.01] 0.138 “ t h e”: [_: 0.88, y: 0.04, r: 0.03, s: 0.01] 0.121 “ t h e _”: [s: 0.13, c: 0.09, f: 0.08, t: 0.07, p: 0.06, a: 0.05, m: 0.05, r: 0.05, b: 0.05, n: 0.04, w: 0.04, g: 0.04, l: 0.04, e: 0.04, d: 0.03, i: 0.03, h: 0.03, o: 0.03, u: 0.01, v: 0.01] 0.004 “ t h e _ e”: [n: 0.24, a: 0.13, x: 0.11, v: 0.08, p: 0.07, r: 0.05, l: 0.05, m: 0.04, s: 0.04, u: 0.03, f: 0.03, i: 0.02, c: 0.02, d: 0.02, g: 0.01, t: 0.01] 0.001 “ t h e _ e n”: [d: 0.41, g: 0.23, t: 0.18, v: 0.03, c: 0.03, o: 0.02, e: 0.01, r: 0.01] 0.000 “ t h e _ e n d”: [_: 0.84, : 0.05, i: 0.05, e: 0.02, s: 0.02] You can also use the NLM to greedily generate (i.e., always choose the most likely next character): >>> NLM.load(“large”) >>> h = NLM() >>> for i in range(100): >>> c, p = max(h.next_prob().items(), key=lambda x: x[1]) >>> print(c, “%.2f >> h += c You get something like: t 0.19
1. Create a single Makefile that will build all of the below C programs. Place them all in a single directory (no sub-directories). If you don’t have a Makefile, your submission will not be graded. If your Makefile does not build a portion of your assignment, that portion will receive a zero grade. I expect to build all the programs below by using the following two commands: make clean make all If this does not work, it will be bad for your grade on this assignment. 2. Write a C program that calls fork(). Before calling fork(), have the parent process assign the value 100 to int called xx. What value is the value of xx in the child process? Have the parent process assign the value 999 to xx and have the child process assign the value 777 to xx. What happens to xx variable when both the child and parent change the value of xx? Have both processes print the value of xx. 3. Write a C program that opens a file name “JUNK.txt” (with the fopen() function call). Have the parent process print “before fork” into the file. The parent process should call fork() to create a new process. Have the parent process perform a for loop that prints “parent” into the open file 10 times. Have the child process perform a for loop that prints “child” into the open file 10 times. What happens when they are writing to the file concurrently, i.e., at the same time (answer this in the comments in your code)? 4. Write a C program using fork(). The child process should print “hello” to stdout; the parent process should print “goodbye” to stdout. Make sure that the child process always prints first. Can you do this without the parent calling wait() in the parent (and NOT using some big loop in the parent)? 5. Write a C program that calls fork() and then the child process calls each of the following exec() functions to run the program “ls -l -F -h”: execl(), execlp(), execv(), and execvp(). The parent process must wait until the child process is complete before it exits. Describe the differences of the exec functions in your code as comments. Final note The programming projects in this course are intended to give you basic skills. In later programming projects, we will assume that you have mastered the skills introduced in earlier programming projects. If you don’t understand, ask questions.Heap management – beavalloc You know you’ve been wanting to do this, so finally, here is your opportunity. Write you own malloc package. You are going to call it beavalloc() and beavfree(). Those are the basic functions. You also need to implement beavcalloc() and beavrealloc(), but those are implemented using beavalloc() (with some extra code). Your code must not make any use of any of the library malloc() functions. You can use brk() and sbrk() for requesting or returning portions of the heap. In fact, you must use brk() and sbrk(). You will manage the memory that is allocated. You may notice this line in the man page for brk()/sbrk(): Avoid using brk() and sbrk(): the malloc(3) memory allocation package is the portable and comfortable way of allocating memory. Ha!!! As an accomplished C programmer and kernel hacker, you are empowered to use brk() and sbrk() to write your own comfortable heap manager. There are a lot of various examples of malloc() source code out there. I encourage you to not rely on those and learn this on your own. Your implementation of beavalloc() (and the other functions) must be done in a file called beavalloc.c. You must put your main() in a separate file. I’ll provide you with a main() that calls a number tests for your beavealloc() functions. I have provided a file beavalloc.h, read the comments in that file. You must have a Makefile that builds your project. I have placed a sample Makefile, the beavalloc.h file, and sample main.c in the following directory on flip/os[12]: ~chaneyr/Classes/cs444/Labs/Lab2 CS 444/544 Lab 2 Fall 2019 Page 2 of 7 R. Jesse Chaney Do not make changes to the beavalloc.h file. Do what you will with the Makefile, just don’t hurt my feelings. Do not remove any of the flags for gcc (CFLAGS). I expect to build the beavalloc program by using the following two commands: make clean make all Be sure to test your code. The main.c file has a few tests in it. Run all the tests and check to make sure the output is correct. Just because your code does not seg-fault, it does not mean the code correctly operates. The tests can either be run all at once or individually. I recommend you run them individually; it makes them easier to validate. You can use whatever data structure to manage the heap you like. I decided to use a doublylinked list. It worked very well for me. I would say that it is he most space or time efficient, but that is not the objective. This is not an exercise to see if you can develop the fastest or most compact malloc() replacement. Many people have spent years working on that; you don’t need to. This is a simple 1-week assignment; don’t make it more than that. Functions you need to write are: void *beavalloc(size_t size); The basic memory allocator. If you pass NULL or 0, then NULL is returned. If, for some reason, the system cannot allocate the requested memory, set the errno global variable to ENOMEM and return NULL. You must use sbrk() or brk() in requesting more memory for your beavalloc() functions to manage. When calling sbrk() for more memory, always request 1024 bytes. It is best if you use a macro for this value, in case it needs to change. Making kernel calls is expensive, so we want to minimize them. Notice that the parameter type is size_t, which is unsigned. If you happen to pass a negative value, odd things may occur. When beavalloc() is called, only call sbrk() to request more memory form the kernel when there is not enough space in the currently allocated space to accommodate it. You must be able split existing blocks within the allocated space. If an existing block can be split to hold a new request, split it and use it. Correctly splitting blocks is probably the second most challenging portion of the project. You will be using a first-fit algorithm. First-fit algorithm – when receiving a request for memory, scan along the list for the first block that is large enough to satisfy the request. If the chosen block is significantly larger than that requested, then it is split, and the remainder added to the list as another free block. CS 444/544 Lab 2 Fall 2019 Page 3 of 7 R. Jesse Chaney void beavfree(void *ptr); A pointer returned from a previous call to beavalloc() must be passed. If a pointer is passed to a block than is already free, simply return. If NULL is passed, just return. Blocks must be coalesced, when possible, as they are free’ed. Correctly coalescing may be the most challenging portion of this project. void beavalloc_reset(void); Completely reset your heap back to zero bytes allocated. You are going to like being able to do this. Implementation can be done in as few as a single line of C code, though you will probably use more to clean up some variables and reset any stats you keep about heap. The only time you actually give memory back to the OS is when you make this call. A call to beavfree() will only mark the space available for reuse. After you’ve called this function, everything you had in the heap is just __GONE__!!! You should be able to call beavalloc() after calling beavalloc_reset() to restart building the heap again. void beavalloc_set_verbose(uint8_t); I like to have the ability to enable and disable runtime diagnostic messages. A call of beavalloc_set_verbose( TRUE ); will enable runtime diagnostic messages. A call of beavalloc_set_verbose( FALSE ); will disable runtime diagnostic messages. All runtime diagnostic messages must go to stderr, not stdout. void *beavcalloc(size_t nmemb, size_t size); This function should perform exactly as the regular call to calloc() would perform. You can read the man page to see what it does. If nmemb or size is 0, then beavcalloc() returns NULL. Make use of your beavalloc(); don’t reinvent anything for this. Consider the memset() function. void *beavrealloc(void *ptr, size_t size); This function should perform exactly as the regular call to realloc(). You can read the man page to see what it does. If size is NULL, return NULL. Consider the memcpy() or memmove() functions. When this function is called with NULL, it means the pointer has never been allocated. Since this we expect the memory block size to change (likely upward, this is realloc() after all), when the ptr is NULL, actually allocate 2 times the size passed as a parameter. void beavalloc_dump(uint leaks_only); This is the function you must use to display the contents of your heap. It is a large and messy function. I understand that you may need to make a few adjustments to it so that it will work with your data structures. However, make CS 444/544 Lab 2 Fall 2019 Page 4 of 7 R. Jesse Chaney as few as possible and DO NOT remove any of the columns of data shown in the original implementation. A longer description of the output from the beavalloc_dump() is shown in an addendum on page 5. This function can be found in the file beavalloc_dump.c. I recommend you copy it into your beavalloc.c module. Look in the beavalloc.h file for declarations of the above functions. Below is a raw plan on how you can proceed. This plan is basic and not required, but it is pretty much how I worked through the project. 1. Start your Makefile. Initiate revision control. 2. When beavalloc() is called, just call sbrk() for the new space. Place that new space into your data structures to manage the space. Don’t worry about the other calls for now. Check your code into revision control. 3. Make sure beavaalloc_reset() works and deallocates all space previously allocated using sbrk(). This will probably be a lot easier than you think. Consider doing a little brk() dancing for this. A static variable within your beavalloc.c file could make this really easy. Check your code into revision control. 4. When beavefree() is called, set the block as free/unused in your data structures. Check your code into revision control. 5. Split blocks that are large enough when memory is requested. Check your code into revision control. 6. Coalesce blocks when adjacent blocks are free-ed. Check your code into revision control. 7. Implement beavcalloc(). This is pretty simple. Check your code into revision control. 8. Implement beavrealloc(). Slightly more involved that beavcalloc(). Check your code into revision control. 9. Make sure all the tests work. Just because a test does not seg-fault does not mean the test ran correctly. Break out a calculator and make sure things add up. Most computerbased calculators have a “programmer” mode that will allow you to calculate the hex addresses. I know both Mac and Windows have this feature in the calculator. Check your code into revision control. 10. Celebrate. This was my favorite step. I recommend you make use of the assert() statement in your code. You can see some examples of it in the testing code I provide and by looking at the man page. Remember, testing is good. You must use the following gcc command line options (with gcc as your compiler) in your Makefile when compiling your code. -g -Wall CS 444/544 Lab 2 Fall 2019 Page 5 of 7 R. Jesse Chaney -Wshadow -Wunreachable-code -Wredundant-decls -Wmissing-declarations -Wold-style-definition -Wmissing-prototypes -Wdeclaration-after-statement Be sure your code does not generate any errors or warnings when compiled. Hunt down and fix all warnings in your code. You may not use any `std=…` command line options for gcc. Just use the default standard, i.e. c90. For a comparison, my beavalloc.c file contains less than 500 lines of code. I have some comments and some extra things in there taking up space. You don’t have to (completely) write the beavalloc_dump() function. All in all, it is not a huge chunk of code. Final note The labs in this course are intended to give you basic skills. In later labs, we will assume that you have mastered the skills introduced in earlier labs. If you don’t understand, ask questions. Addendum – Format of the beavalloc_dump() output. Below is an example of running my code on test 5 from the suite of tests. The heap map consists of several columns of data. The beavalloc_dump() function has 2 purposes: 1) help grade the assignment, 2) help debug the assignment. The columns in the output are: blk no: I start numbering the blocks managed by beacalloc() from zero, so the initial block number is block 0. For me, this is the start of the doubly-linked list I use to manage the heap. It is also the address of the beginning of the heap. block add: this is the beginning address of a block beavalloc() is managing in the heap. For me, it is the beginning of an instance of the structure in my doubly-linked list. This address is shown as hex. next add: the doubly-linked list I use contains 2 pointers for the previous and next elements in the list. This is the address of the next element. If the next address is (nil), it means that it is the head of the list. This address is shown as hex. prev add: the doubly-linked list I use contains 2 pointers for the previous and next elements in the list. This is the address of the previous element. If the previous address is (nil), it means that it is the end of the list. This address is shown as hex. data add: in my list element data structure, I keep a pointer to where the data for the block begins. This is that pointer; it is a little wasteful to keep this around as a pointer. This address is shown as hex. blk off: this is the offset (in decimal) where the block begins from the beginning of the heap. CS 444/544 Lab 2 Fall 2019 Page 6 of 7 R. Jesse Chaney dat off: this is the offset (in decimal) where the data begins from the beginning of the heap. capacity: this represents the total number of data bytes available in the block (in decimal). Some or even most of the bytes may not be used. See the size column below. size: this represents the number of data bytes (in decimal) requested by the user’s call to beavalloc(). This is the number of bytes the user can use in the block. If the user goes beyond size bytes, then they have actually gone out of bounds of their allocation. The block may have a capacity larger than the size. The additional bytes could be used in a call to beavrealloc() or the block could be split (if large enough). blk size: the total number of bytes used by the block (in decimal). This will be the sum of the capacity and the size of the data structure used to manage the heap. status: this simply represents whether a block is currently in use (allocated to the user) or is free (awaiting reuse). The row of data below the per block information shows (in decimal) the total number of bytes of data capacity that are currently allocated, the total number of bytes the user currently has in use, and the total number of bytes are allocated in the heap. The final line in the output from beavalloc_dump() shows: how many blocks are allocated to the user, how many block are marked as free, what is the address of the beginning of the heap is (Min heap), and what the address of the end of the heap is (Max heap). CS 444/544 Lab 2 Fall 2019 Page 7 of 7 R. Jesse Chaney % ./beavalloc -t 5 beavalloc tests starting running only test 5 *** Begin 5 base: 0x12b3000 5 allocs 3 frees ptr5 : 0x12b4030 ptr4 : 0x12b3c30 ptr3 : 0x12b3830 ptr2 : 0x12b3430 ptr1 : 0x12b3030 heap map blk no block add next add prev add data add blk off dat off capacity size blk size status 0 0x12b3000 0x12b3400 (nil) 0x12b3030 0 48 976 510 1024 in use 1 0x12b3400 0x12b3800 0x12b3000 0x12b3430 1024 1072 976 530 1024 in use 2 0x12b3800 0x12b3c00 0x12b3400 0x12b3830 2048 2096 976 550 1024 in use 3 0x12b3c00 0x12b4000 0x12b3800 0x12b3c30 3072 3120 976 570 1024 in use 4 0x12b4000 (nil) 0x12b3c00 0x12b4030 4096 4144 976 590 1024 in use Total bytes used 4880 2750 5120 Used blocks: 5 Free blocks: 0 Min heap: 0x12b3000 Max heap: 0x12b4400 heap map blk no block add next add prev add data add blk off dat off capacity size blk size status 0 0x12b3000 0x12b3400 (nil) 0x12b3030 0 48 976 510 1024 free * 1 0x12b3400 0x12b3800 0x12b3000 0x12b3430 1024 1072 976 530 1024 in use 2 0x12b3800 0x12b3c00 0x12b3400 0x12b3830 2048 2096 976 550 1024 free * 3 0x12b3c00 0x12b4000 0x12b3800 0x12b3c30 3072 3120 976 570 1024 in use 4 0x12b4000 (nil) 0x12b3c00 0x12b4030 4096 4144 976 590 1024 free * Total bytes used 4880 2750 5120 Used blocks: 2 Free blocks: 3 Min heap: 0x12b3000 Max heap: 0x12b4400 WoooooooHooooooo!!! You survived test 5. *** End 5 Make sure it is correct.Working with inodes – mystat (100 Points) For this part of this assignment, you will write a C program that will display the inode meta data for each file given on the command line. You must call you source file mystat.c and your program will be called mystat. This program will run directly on os2, not within xv6. An example of how my program displays the inode data is shown the Figure 4. You might also want to look at the output from the stat command (the command not a system function, man section 1). Though not as pretty (or in some cases as complete as the replacement you will write), it is the standard for showing inode information. Requirements for your program are: 1. The output of your program should look exactly like mine. 2. Display the file type (regular, directory, symbolic link, …) as a human readable string. If the file is a symbolic link, look 1 step further to find the name of the file to which the symbolic link points. If the file to which the link points does not exist, indicate that. See Figure 4. 3. Display the inode value. 4. Display the device id. 5. Display the mode as both its octal value and its symbolic representation. The symbolic representation will be the rwx string for user, group, and other. See Figure 4 or ‘ls -l’ for how this should look. 6. Show the hard link count. 7. Show both the uid and gid for the file, as both the symbolic values (names) and numeric values. This will be pretty darn easy if you read through the list of suggested function calls. See Figure 4 for how this should look. 8. File size, in bytes. 9. Block count. 10. Show the 3 time values in local time/date. This will be pretty darn easy if you read through the list of suggested function calls. See Figure 4 for how these appear. CS 444/544 Lab 3 Fall 2019 Page 2 of 4 R. Jesse Chaney System and function calls that I believe you will find interesting include: stat()and lstat() (you really want to do “man 2 stat” and read that man entry closely, all of it [yes really, all of it]), readlink(), memset(), getpwuid(), getgrgid(), strcat(), localtime(), and strftime(). Notice that ctime() is NOT in that list and you don’t want to use it. Since you must be able to show the file type if a file is a symbolic link, I encourage you consider lstat() over stat(). My implementation is about 280 lines long, but there is some dead code in my file. I have code commented out to support features not required for your assignment. There is no complex logic for this application, just a lot of long code showing values from the struct stat structure from sys/stat.h. Honestly, the longest portion of your code will likely be devoted to displaying the symbolic representation of the mode. Formatting these strings is a little awkweird. I suggest you create a function. Don’t worry about sticky bits or set uid/gid bits. Do you know what sticky bits are for or how they used to be used? You must to be able to show the following file types: • regular file, • directory, • character device, • block device, • FIFO/pipe, • socket, and • symbolic link (both a good one and a broken one). When formatting the human readable time for the local time, I’d suggest you consider this “%Y-%m-%d %H:%M:%S %z (%Z) %a”, but read through the format options on strftime(). The executable version of my code is in the directory. You are welcome to run it to see the output. I have some examples of both a FIFO/pipe, a socket, symbolic link in my Lab3 directory for you to use in testing (the FUNNY* files, Figure 2). You can find a block device as /dev/sda and a character device as /dev/sg0. See Figure 3. You must have a Makefile that builds the mystat program, without any warnings from the compiler. You must use the following gcc command line options (with gcc as your compiler) in your Makefile when compiling your code: Figure 2: The FUNNY* files. Figure 3: Block and character files. Figure 1: A sample of files to use for your testing. Found in ~chaneyr/Classes/cs444/Labs/Lab3 CS 444/544 Lab 3 Fall 2019 Page 3 of 4 R. Jesse Chaney -g -Wall -Wshadow -Wunreachable-code -Wredundant-decls -Wmissing-declarations -Wold-style-definition -Wmissing-prototypes -Wdeclaration-after-statement When we grade your program, we will issue the following commands to build your executable. make clean make all There must be zero (0) warnings from the compiler. If your program compiles and produces warnings (using the above command line options for gcc), then it is a zero. Final note The labs in this course are intended to give you basic skills. In later labs, we will assume that you have mastered the skills introduced in earlier labs. If you don’t understand, ask questions. For those of you have read this far and are actually reading the entire assignment, thank you. As a reward, if you look at the section 2 man page for stat(), down near the bottom, I think you’ll like it. That is: man 2 stat CS 444/544 Lab 3 Fall 2019 Page 4 of 4 R. Jesse Chaney Figure 4: The inode information of the FUNNY files. Sorry the image is so small. A directory. A pipe. A regular file. A socket. A symbolic link. A broken symbolic link.Part 1 – Some additional programs – (10 points) In class, we added the mult.c and mfork.c programs into the xv6 system (by adding the C code, and editing the Makefile). I need you to make sure those programs are a regular part of your xv6 system. You will use these programs to test the correct functioning of your code and they will be used to test your code when grading your assignment. The C files mult.c and mfork.c can be found in my Lab4 directory, ~chaneyr/Classes/cs444/Labs/Lab4 The purpose of the mult program is to just take a long time to complete. The purpose of the mfork program is to just fork a number of processes in xv6. Add these into the UPROGS macro in your Makefile. Part 2 – Add some tracking information to each process (100 pts) Add four members to the struct proc data structure (found in proc.h). Remember, you are the kernel developer and master level C programmer, so you should be comfortable manipulating the kernel structures. Of course, I put all this stuff within #ifdef blocks using a macro called PROC_TIME. 1. Add a member type struct rtcdate. You’ll find the definition of that structure in the date.h file. I like to call this member begin_date. 2. Add 3 members of type unsigned int. Good names for those 3 members are: a. ticks_total – this will represent the total number of time ticks that the process has run. CS 444 Lab 4 Fall 2019 Page 2 of 7 R. Jesse Chaney b. ticks_begin – this will be used to help calculate the total number of time ticks the process has used. c. sched_times – this will be used to count the number of times the process has been scheduled to run. You can find the definition of struct rtcdate in the date.h file. While you are in the date.h file, make sure it has multiple-include protection. Now that you have the new data members in the struct proc data structure, it’s time to put them to use. When a process is first allocated, set the begin_date to the “current” date/time. How do you know where a process is first allocated? You might look for a function called something like allocproc(). The qemu clock seems to synchronize to UTC time at startup. If you really want it set to the current Pacific time, ask me how (it is a little make magic). Finding the right call to get the date/time is a little awkward. I recommend you look in the lapic.c file for the function cmostime(). However, you probably don’t want to spend too much time looking at that in that file, it’s a bit icky. Calling the cmostime() function is a wee bit awkward, because you must pass an address or pointer. Remember that putting an & in front of a variable/structure when calling a function will pass the address of the variable/structure. Decide to ignore the goto and the label found in that function; we won’t talk about them. Once you set the begin_date member, set ticks_total , ticks_begin, and sched_times to zero in the same area of the code (when a new process is allocated). Now let’s look at the scheduler() code (in proc.c). You’ll notice that the scheduler runs as an infinite loop (see the for(;;) ?). There are 2 places where you want to add a few lines of code into the scheduler() routine. One is just before the newly chosen process is scheduled and the other is just after that process returns back to the scheduler. The first place (before the newly chosen process runs), is pretty easy to find. Look for the place where the state of the process is set to RUNNING. I dropped an #ifdef block just before that. That block does 2 things, it increments the sched_times member for the chosen process, and sets the begin_ticks member to the current number of ticks that have accumulated for the vm. How do you find out how many “ticks” the system has been up? That is an excellent question. My recommendation is to look for a function called uptime(). Unfortunately, because of how the kernel function uptime() is defined and implemented (as sys_uptime()), you cannot directly call it. You have a choice to either 1) replicate the capability of uptime() in the scheduler() code (which I do not like), or 2) make a new function that returns the same thing (which I do like). If you decide to do the second option, I recommend you have sys_uptime() directly call your new function. I went with option 2; I don’t like to duplicate code. Now, about that second place to drop in a few lines of code. What do you think the following 2 lines of code from the scheduler() function do? swtch(&(c->scheduler), p->context); switchkvm(); CS 444 Lab 4 Fall 2019 Page 3 of 7 R. Jesse Chaney ln addition, notice that immediately after those lines of code, the c->proc variable is set to 0 (aka NULL). So, might this be that when a new process is actually scheduled to run by calling swtch() and switchkvm(), and it returns back to scheduler following those calls? Looks like a good place to update the total_ticks member of the process (after return from switchkvm()). Part 3 – Modify ps to show time tracking information (50 pts) Now that you have all this great time tracking information for each process, modify your ps program to display it. You should display the information as follows. One of the fun things you’ll notice is that you may need to put a zero in where a value is represented as a single digit. Notice that the month shown in the start time is 05, not just 5. It’s a simple trick, but worth learning. If you had a fully functioning printf() (or for this example cprintf()), it would be just a change to the format string. However, your printf() is not capable of that (and don’t spend the month or 6 working on the format characters in the xv6 version of printf()). Part 4 – Add a rand() function (50 pts) You will need to use a function the generates random numbers. More specifically, pseudorandom numbers. This does not need to be cryptographically secure random numbers, just something reasonable, such as the standard Unix/Linux rand() function returns. Amazingly, if you happened to read to the bottom of the man page for rand, in section 3 of the man pages, you can see some source code for a version of rand() that works just fine for this purpose. Differing from the man page example, you will call your functions rand() and srand(), NOT myrand() and mysrand(). If you prefer to use some other swanky random number generator, that’s fine but not required. Your implementation of rand() must be done in 2 files: rand.c and rand.h. The rand.c file will contain the implementation of rand() and srand(). The rand.h file will contain the declarations (aka prototypes) of rand() and srand() AND must contain the macro RAND_MAX, yep you need a macro. CS 444 Lab 4 Fall 2019 Page 4 of 7 R. Jesse Chaney Based on the code in the man page, you’d establish RAND_MAX to be 32767 (2^15), but we are going to use 2^31. So, your RAND_MAX macro should be (1 tf = *curproc->tf; The eax register for the new thread is cleared just as it is for a new process, though frankly it does not really apply here. The eip register (in the tf structure) represents the instruction pointer for the new thread. You should assign func to it. This is not touched in the fork() function. We need to assign the esp (extended stack pointer) data member (from the tf structure in np) to the tstack that was passed as a parameter. However, the tstack variable is the opposite end of the stack (as the stack grows low address to high address). So, try something like this: np->tf->esp = ((int) tstack) + PGSIZE; We are going with the assumption that the size of the stack for each thread is a single page (PGSIZE). It would be nice to be able to create threads with different stack size, but that is beyond this assignment. We need to push a value on the stack. Specifically, we need to push the arg_ptr variable value onto the stack for the thread function to pick up. This will take 2 statements. See if these make sense. np->tf->esp -= sizeof(int); *((int *) (np->tf->esp)) = (int) arg_ptr; The first line decrements the value for the beginning of the stack pointer. The thread function will pull the value from the esp register beginning at this location. The second line copies the value from within the arg_ptr variable on the stack. We have to do some magic C casts to make sure everything is happy. We have 2 more manipulations of the stack pointer. We need to push the value of the new thread’s tid onto the thread stack. This is the return value from kthread_create that the thread function can pick up. Assign the new tid value to a local variable (called maybe something wild, like tid). Assign the same value to the tid data member for np. The new tid should come from the parent tid data member (np->parent). Make sure you increment the parent’s nextTid value as well (as a post increment). Now np->tf->esp -= sizeof(int); *((int *) (np->tf->esp)) = tid; You are getting to be a real pro at this. We are almost there for kthread_create. In fork() there is a little loop where the file descriptors from curproc are duplicated (using filedup()) into np. You need to do this for the thread. You need to idup() the curproc->pwd into np->pwd. As fork() does. This is immediately after the loop above. As fork() does, do the safestrcpy(), acquire the ptable lock, set the state of the thread to RUNNABLE, and release the lock. CS 444 Lab 5 Fall 2019 Page 9 of 13 R. Jesse Chaney Return the tid. I liberally sprinkled a number of if statements that use the debugState variable in the function. Some simply check to see if debugState is non-zero while others check to see if debugState is greater than 1 or 2. This allows me to change the number of diagnostic messages from the kernel at runtime. I found use of the kdebug command very useful. Whew… done with kthread_create(). kthread_join() The kthread_join() function is a bit easier than kthread_create(), but it has a few subtle requirements. The kthread_join() has a lot in common with the wait() function. In fact, you will need to make a change to the wait() function as part of developing the kthread_join() function. Anything you do in the kthread_join() function that makes a change to the ptable process array will require a lock on that structure. Make sure that you also unlock it before returning. There is a lot more room for creativity in development of this function. I’m going to walk through how I did it, but there are a lot of opportunities for variation. If the curproc is a parent (of threads), but has a current thread count of 0, just release and return -1. If the given tid is 0, just release and return -1. As I have designed my code, a thread cannot join with thread 0 (the main thread). This is not a requirement, just how I did my code. In the future, I’d like to change this. This is a bit tricky here, I want to make sure that I decrement the correct threadCount data member, so I check to see if curproc->isThread is TRUE. If so, I set curproc = curproc->parent. Remember back in the kthread_create() function, this statement “There is not a parent-child relationship”? We made sure that all threads parent pointer went back to the main thread (thread 0, see Figure 1). This is where that becomes important. The thtst4 program will test this. Acquire a lock on the ptable. If you look at the code for wait(), you’ll see that it has an infinite for loop around a loop through the ptable at this point. I’m going to take a slightly different approach on this (let me know if you find a flaw in it). Look through the ptable with a for loop (I do it in the same way as wait() does, where p is the current entry in the table). If p->parent != curproc || p->tid != tid, just continue in the loop. If we get through the test p->parent != curproc || p->tid != tid, we know we’ve found the right parent and tid combination. I go into the following loop: CS 444 Lab 5 Fall 2019 Page 10 of 13 R. Jesse Chaney while (p->state != ZOMBIE) { release(&ptable.lock); yield(); acquire(&ptable.lock) } Basically, if the thread is not marked as a zombie, release the ptable lock, and yield the processor. Remember when we talked about yielding the time slice for a process? Once the thread is scheduled again, acquire the ptable lock and repeat the test. The thread will be marked as a zombie in the kthread_exit() routine. Once we get through the above loop, we know we have the right thread and that the thread has exited. We need to clean it up. Decrement the threadCount of curproc (which is the main thread). Do the stuff done in wait(), but DO NOT call freevm(). This is the stuff in the if block where p->state == ZOMBIE. The main thread owns the virtual memory for the entire process. Only when the main thread exits is the virtual memory freed. Break out of the loop. If you found the right parent and tid combination, return 0. Otherwise, return -1. Make sure you release the ptable lock before the return. Okay, let’s go make that change to the wait() function. Remember that the kernel is periodically looking for zombie processes. As we write these as kernel threads, they look a lot like processes to the kernel. We do not want the kernel doing cleanup on the threads until we are ready for it. So, there is an if block in the wait() function that looks like this: if (p->parent != curproc) continue; We must change it condition to look like this: if(p->parent != curproc || p->isThread == TRUE) If the thingie the kernel (or other process) is inspecting is a thread, just move along. It does not make sense to call wait() on a thread, you must join with it. 2 functions down (kthread_create() and kthread_join()); 1 more to go. kthread_exit() The code for kthread_exit() is pretty straight forward. Get the curproc (as exit() does). If curproc data member isThread is TRUE, then: Close all the open files (see exit()). CS 444 Lab 5 Fall 2019 Page 11 of 13 R. Jesse Chaney Cleanup the cwd (exacly as exit() does it with begin_op and end_op). In previous development, I have experienced some issues when cleaning up the cwd, but have not with this development. Set killed to FALSE, the thread data member threadExitvalue to the passed exitValue, oncpu to -1, and state to ZOMBIE. Now, acquire the ptable lock and call sched() (not scheduler()). Follow the call to sched() with a panic(“kthread_exit”) call. Obviously, it should never get to the call to panic. That’s it for kthread_exit(). Time to give a big woohoo! Part 6 – Refresh the ps/cps() code – (50 points) We’ve added a couple data members to the proc structure (Remember Part 2? Seems like ages ago.). We want those to show up when we run the ps command. You need to add the following to the output from the cps() function: oncpu, isParent, isThread, and threadCount. The header information should be: “cpu”, “is par”, “is thrd”, and “thrd #”. You only show the oncpu value when it is >= 0. If you’ve followed the instructions from above, this should be easy. Only a RUNNING process/thread should have a value of oncpu that is greater than or equal to 0. Do not show the negative value for oncpu. For isParent (which means it is a main thread) and isThread, just show 0 for false and 1 for true. The threadCount value should be zero except for a main thread in a process. See Figure 2 for how this should look. Figure 2: The ps command with new columns. CS 444 Lab 5 Fall 2019 Page 12 of 13 R. Jesse Chaney The easy way to see this data is to run “thtst2 6 &” (in the background) and then run ps a couple of time to see the output. If you see a “zombie!” after doing this, don’t worry, it is only a flaw in their shell. Part 7 – Update and Validate on 4 CPUs – (100 points) Change the CPUS macro in the Makefile to 4. It is fine for you to do your development with a single cpu in qemu. However, you code will be tested with the value of the CPUS macro in the Makefile set to 4. I highly recommend you test your code this way. In the Makefile, look around line 251. The points for this part are not awarded to simply changing the Makefile, but for all the test commands working correctly with 4 CPUs. How It will be Graded When we grade, we will first run the 6 test commands with a single CPU. We will run 1 test program, then exit qemu before running the next test program. I know this stuff is hard and we really don’t have the 6 months to develop a full test suite, that’s why we will exit qemu between tests. We can run qemu using a single CPU with the following command: make nox CPUS=1 We can run qemu using 4 CPUs with the following command: make nox CPUS=4 The macro on the command line will override the setting within the Makefile. Other Tips I have to be honest, there are a couple places where I struggled when writing this code. One of the biggest is where I called kfree() in the kthread_join() function (you can put it in kthread_exit()). If you look in the code for kfree(), you’ll see that it does a memset() on the page of kernel memory to be freed. Doing the memset() is an excellent idea, for the reason mentioned in the comment. However, somewhere I must have some boundary conditions messed up. When I run thtst2 and do the memset(), I will usually get “unexpected trap 14 from …”. If #ifdef out (or comment out) the call to memset() in kfree(), all is fine. Many web searches later, it would seem that I have overrun a buffer and stomped all over an instruction pointer. But, I cannot see where I’ve gone astray. I would really like to know what I am doing wrong. L If you find out what my error is, have pity on me and let me know. Submit to TEACH When you are done with the Lab5, submit your code to TEACH. Remember how we used the command “make teach” to produce a tar and gzipped file that you can submit into TEACH? Do that and be done. CS 444 Lab 5 Fall 2019 Page 13 of 13 R. Jesse Chaney Final note The labs in this course are intended to give you basic skills. In later labs, we will assume that you have mastered the skills introduced in earlier labs. If you don’t understand, ask questions. Example Output from Test Programs Figure 8: Successful run of thtst1. Figure 9: Failed run of thtst1 Figure 5: Successful run of thtst2. Figure 7: Sample test run of thtst3 Figure 6: Sample output from thtst4 Figure 4: Sample output from thtst5 Figure 3: Sample output from thtst6
1. [Big Data concept] (10) Give one example of Big Data application you know. Use the detailed example to explain each of the five Big V’s. If you are required to design a database system for this application, what are the best data models (relational, XML, RDF, among others) you would use to represent the data and why? 2. [Relational Data Model] (30) As of January 2017, the OpenFlights Airports Database (https://openflights.org/data.html) contains over 10,000 airports, train stations and ferry terminals spanning the globe. Each entry in the Airport table contains the following: ——————————————————————————————————————————————- Airport ID Unique OpenFlights identifier for this airport. Name Name of airport. May or may not contain the City name. City Main city served by airport. May be spelled differently from Name. Country Country or territory where airport is located. See countries.dat to cross-reference to ISO 3166-1 codes. IATA 3-letter IATA code. Null if not assigned/unknown. ICAO 4-letter ICAO code. Latitude Decimal degrees, usually to six significant digits. Negative is South, positive is North. Longitude Decimal degrees, usually to six significant digits. Negative is West, positive is East. Altitude In feet. Timezone Hours offset from UTC. Fractional hours are expressed as decimals, eg. India is 5.5. DST Daylight savings time. One of E (Europe), A (US/Canada), S (South America), O (Australia), Z (New Zealand), N (None) or U (Unknown). See also: Help: Time Tz database time zone Timezone in “tz” (Olson) format, eg. “America/Los_Angeles”. Type Type of the airport. Value “airport” for air terminals, “station” for train stations, “port” for ferry terminals and “unknown” if not known. In airports.csv, only type=airport is included. Source Source of this data. “OurAirports” for data sourced from OurAirports, “Legacy” for old data not matched to OurAirports (mostly DAFIF), “User” for unverified user contributions. In airports.csv, only source=OurAirports is included. ——————————————————————————————————————————————- (a) (5) Consider the following terms: relation schema, relational database schema, domain, attribute, attribute domain, relation instance. Give what these terms are with the above Airport database. Give one small (4-5 tuples) instance of the Airport table. (b) (10) There are three databases in the OpenFlight dataset: Airport, Airline, and Route. Give the schema of these three databases and mark the primary keys, foreign keys and provide examples of functional dependencies you identified over the three tables. [You may draw a diagram to illustrate the schema, PKs, FKs and FDs] (c) [FD inferencing] (10) Recall Armstrong’s axioms. 1. Reflexivity rule: if Y ⊆ X then X → Y 2. Augmentation rule: if X → Y then XZ → YZ 3. Transitivity rule: if X → Y and Y → Z then X → Z (1) Give two examples for using Armstrong’s inference rules to induce new FDs from the set of FDs you designed in question 2 (b). (2) Prove the following inference rules also hold, using FD definition and Armstrong’s Axioms. a. decomposition rule: if X → YZ then: X → Y and X → Z b. Pseudo transitivity: if X → Y and YW → Z then: XW → Z (d) [Normalization] (5) Given a relation R(A1, A2, A3, A4), with three FDs A2, A3 → A4 ; A3, A4 → A1; A1, A2→ A3. Provide the 3NF and BCNF form of the schema and explain why. 3. [Relational Algebra] (20) Consider the following database schema: Movies (Title, Director, Actor); Location (Theater, Address, Phone number); Schedule (Theater, Title, Time). Express the following queries in relational algebra (select σ, project ∏ , Cartesian product X, join (theta-join)) -Q1: which theaters feature “Zootopia”? -Q2: List the names and address of theaters featuring a film directed by Steven Spielberg. -Q3: What are the address and phone number of the Le Champo theater? -Q4: List pairs of actors that acted in the same movie. (* you want to use renaming on Movies and join the Movies with its copy Movie’). 4. [Join Operators] (40) This sets of questions test the understanding of basic database search operators. Consider a join ⋈!.#$%.&. We ignore the cost of output the result, and measure the cost with the number of I/Os. Given the information about relations to be joined below: Relation � contains 20,000 tuples and has 10 tuples per block. Relation � contains 100,000 tuples and has 10 tuples per block. Attribute � is the primary key of �. In total, 52 blocks are available in memory. Assume neither relation has any index. a. (10) Describe a block nested join algorithm, Give the cost of joining � and � with a block nested loops join. b. (15) Describe a sort-merge join algorithm. Give the cost of joining � and � with a sort-merge join. c. (15) Describe a hash-join algorithm. Give the cost of joining � and � with a hash join.1. [XML and RDF] (40) (a) (10) Consider the database instance you gave in Assignment 1 Question 2 (a). Assume now that you don’t have any schema. Give an XML document to represent the tuples as the fact about the airports. (b) (10) Consider the relational schemas you gave in Assignment 1 Question 2 (b). Give an XML schema representation of each relational schema. How do you encode keys? Foreign keys? (c) (20) Consider a set of natural language sentences collected from Webpages. i. A human can like another human. ii. A human can have a sex property of a man or a woman. iii. A man can be the father of anotherhuman. iv. A woman can be the mother of another human. v. A human can be married to another human. vi. A human can have a BirthYear property of type“xs:Year”. vii. If a human is married to another, then they like eachother. viii. If a human is a mother or father, the human is a parent. Write a RDF schema and give a graphical presentation to describe these relationships. 2. [Graph Algorithm] (30) The following questions test your understanding on basic graph algorithms a. (15) Given a directed graph �(�, �, �) with � the node set, � the edge set and � a function that assigns to each edge � ∈ � a label �(�). A label constrained reachability query �(�,�, �) tests if there exists a path from a source node � to a target node �, which consists of edges having a label from a label set �. Give an algorithm (pseudo-code) to answer query �. (hint: A straightforward way is to revise BFS or DFS traversal) b. (15) Consider a network �(�, �) of servers, where each edge � = (�, �) represents a communication channel from a server � to another server �. Each edge has an associated value �(�, �), which is a constant in [0,1]. The value represents the reliability of the channel, i.e., the probability that the channel from server � to server � will not fail. Assume that these probabilities are independent. Give an algorithm (pseudo-code) to find the most reliable path between two given servers. Give the complexity (in Big O notation) of your algorithm. (hint: transform the weight to non-negative numbers and the problem may become very familiar to you). 3. [Approximate Query Processing] (25) This question continues our discussion on using data synopsis for query processing based on data-driven approximation. You are given a vector of numbers: [127, 71, 87, 31, 59, 3, 43, 99, 100, 42, 0, 58, 30, 88, 72, 130], each data point records the frequency of communication of a server in a 5-minute interval. For example, in the first 5 minutes, 127 contacts are observed. In the next 5 minutes which is time interval [5, 10], 71 contacts, … a. Give the Haar decomposition and draw a corresponding error tree for the contacts data vector. b. Give the process and result for reconstructing the frequency during time interval [15, 20] using Haar decomposition (explain the path in a top-down fashion). c. Use Haar decomposition and error tree to compute the total number of communications between time interval [15, 30] (explain the path in a top-down fashion).1. [noSQL] (40) This set of questions get you to engage discussion related with noSQL systems. a. Explain the concept of noSQL b. Describe the 4 types of noSQ systems; for each category, give an example noSQL system (try to give different system from what we introduced in the class). c. Pick one type of noSQL system you give in b, give a real-world application that can make best use of the system, and an application that may not be well-suited for the system. Explain your reasons. 2. [CAP and ACID] (20) This set of questions are related to data consistency a. What is CAP Theory? Consider a toy example of a cluster that contains two servers S1 and S2. Use the example cluster to explain the CAP theory. You can simply use the proof I explained in class. b. Consider the relation Accounts(acctNo, balance), and two SQL statements that conduct a request “transfer $200 from account 123 to 456”. In a DBMS, this usually includes two steps: i. Add $200 to account 456: UPDATE Accounts SET balance=balance+200 WHERE acctNo = 456 ii. Subtract $200 from account 123: UPDATE Accounts SET balance=balance‐200 WHERE acctNo=123 Use this example and necessary scenarios to show when Atomicity, Consistency, Isolation and Durability can be violated. 3. [Column Store] (20) We take a closer look at Column Store a. Explain the major features of column stores in terms of data storage and storage key. b. We introduced three techniques to optimize column-oriented databases: compression, late materialization and block iteration. Please explain how they work. 4. [newSQL] (20) a. Describe the definition of NewSQL systems and design principles. b. For in-memory DBMS, describe the set-associative cache for block placement and LRU block replacement policy.1. [Parallel Data Models] (30) a. What is speedup and scaleup? Give three reasons why we cannot do better than linear speedup. b. Assume a program P running on a single-processor system takes time T to complete. 40% of P can only be executed sequentially on a single processor, and the rest is “embarrassingly parallel” in that it can be easily divided into smaller tasks executing concurrently across multiple processors. What are the best time costs to execute P using 2, 4, 8 machines (expressed by T)? What are the speed-ups respectively? What are the optimal speed-ups given an infinite number of machines? c. Describe and compare the pros and cons of the three architecture for parallel systems. 2. [MapReduce] (40) This set of questions test the understanding and application of MapReduce framework. a. (20) Facebook updates the “common friends” of you and response to hundreds of millions of requests every day. The friendship information is stored as a pair (Person, [List of Friends]) for every user in the social network. Write a MapReduce program to return a dictionary of common friends of the form ((User i, User j), [List of Common Friends of User i and User j]) for all pairs of i and j who are friends. The order of i and j you returned should be the same as the lexicographical order of their names. You need to give the pseudo-code of 1 main function, and 1 Map() and 1 Reduce() function. Specify the key/value pair and their semantics (what are they referring to?). b. (20) Top-10 Keywords. Search engine companies like Google maintains hot webpages in a set � for keyword search. Each record � ∈ � is an article, stored as a sequence of keywords. Write a MapReduce program to report the top 10 most frequent keywords appeared in the webpages in �. Give the pseudo-code of your MR program. Hit: You may need two rounds of MR processes for (b) 3. [Apache Spark] (30) This set of questions relate to Apache Spark a. Explain the definition of RDD and how the lineage retrieval works b. List the reasons why Spark can be faster than MapReduce. c. Explain the definitions of narrow dependencies and wide dependencies. In addition, explain how Spark determines the boundary of each stage in a DAG and why put operators into stages will improve the performance.
Overview and Goal In this course you will be creating an operating system kernel. You’ll be using the BLITZ software tools, which were written for this task. The goals of this project are to make sure that you can use the BLITZ tools and to help you gain familiarity with them. Step 1: Print the Documentation There are a number of documents describing the BLITZ tools. You may obtain the documents by going to the BLITZ homepage: https://www.cs.pdx.edu/~harry/Blitz/index.html From there you can access pdf versions. Print out the following documents: An Overview of the BLITZ System (7 pages) An Overview of the BLITZ Computer Hardware (8 pages) The BLITZ Architecture (71 pages) Example BLITZ Assembly Program (7 pages) BLITZ Instruction Set (4 pages) The BLITZ Emulator (44 pages) An Overview of KPL, A Kernel Programming Language (66 pages) Context-Free Grammar of KPL (7 pages) BLITZ Tools: Help Information (13 pages) The Format of BLITZ Object and Executable Files (12 pages) Step 2: Read the Overview Document Read the first document (“An Overview of the BLITZ System”) before proceeding to Step 3. Project 1 Operating Systems Page 2 Step 3: Choose Your Host Platform You will develop your operating system code on a “host” computer and you will be running the BLITZ tools on that host computer. You should decide now which host computer you will be using. The BLITZ tools run on the follow host platforms: Apple Macintosh, OS X, either PPC-based or Intel-based machines Sun Solaris Unix / Linux Systems Windows, using Cygwin which emulates the Unix POSIX interface (see www.cygwin.com) For the following host platforms, the tools are precompiled and you can simply download the executables: Apple Macintosh, OS X, Intel-based machines Apple Macintosh, OS X, PPC-based machines Sun Solaris For other systems, you can download the tools (which are written in “C” and “C++”. You must then compile them on your computer. The source code for all the BLITZ tools is available, but you should not need to look at it. Nevertheless, it is available for anyone who is interested. The BLITZ Tools Here are the programs that constitute the BLITZ tool set. kpl The KPL compiler asm The BLITZ assembler lddd The BLITZ linker blitz The BLITZ machine emulator (the virtual machine and debugger) diskUtil A utility to manipulate the simulated BLITZ “DISK” file dumpObj A utility to print BLITZ .o and a.out files hexdump A utility to print any file in hex check A utility to run through a file looking for problem ASCII characters endian A utility to determine if this machine is Big or Little Endian Project 1 Operating Systems Page 3 These tools are listed more-or-less in the order they would be used. You will probably only need to use the first 4 or 5 tools and you may pretty much ignore the remaining tools. (The last three tools are only documented by the comments at the beginning of the source code files, which you may read if interested.) Organization of the Course Material The BLITZ system is accessible via the following URL: https://www.cs.pdx.edu/~harry/Blitz/ The Blitz directory contains the following material: Blitz/ BlitzDoc/ …files containing the documents mentioned above… BlitzBin/ MacIntel/ …executables for the Mac/Intel host platform… MacPPC/ …executables for the Mac/PPC host platform… Sun/ …executables for the Sun/Solaris host platform… BlitzSrc/ …source code for the BLITZ tools… OSProject/ p1/ …files related to project 1… p2/ …files related to project 2… …etc… You may access this material through the BLITZ Home page. You should also be able to “ftp” directly to these directories. Step 4A: For Mac Users… Step 1: Create a directory to put the BLITZ tools into. For example, you may wish to create a directory called BlitzTools in your home directory: /Users/YourUserName/BlitzTools Project 1 Operating Systems Page 4 Then copy all the files from www.cs.pdx.edu/~harry/Blitz/BlitzBin/MacIntel to your BlitzTools directory. These are binary files, not text files. (I use an application called “Fetch” (www.fetchsoftworks.com) to do “ftp” file transfers.) People using an older PPC-based Mac should use this directory instead: www.cs.pdx.edu/~harry/Blitz/BlitzBin/MacPPC Step 2: Set the protection bits on these programs to include “executable”, with a command such as: chmod ugo+rx BlitzTools/* Step 4B: For Portland State University Students… See section 4C. Step 4C: For Users in a Shared Environment… This section applies to users who have an account on a shared system, such as a Sun/Solaris system. It is assumed that the BLITZ tools have already been downloaded by someone else and are available in some shared directory. All you need to do is modify your PATH variable so that your shell will search the appropriate directory. For example, students at Portland State University who have an account on the shared Sun/Solaris system can find the executables in this directory: ~harry/public_html/Blitz/BlitzBin/Sun Students at other institutions can find the executables in this directory: _________________________________________________________________________ Step 4D: For Unix/Linux Users… This section applies to users who have a Unix/Linux box and wish to download and re-compile the BLITZ tools for their machine. Project 1 Operating Systems Page 5 Step 1: Create a directory to put the BLITZ source code into. For example, you may wish to create a directory called BlitzSrc in your home directory: ~YourUserName/BlitzSrc Then copy all the files from www.cs.pdx.edu/~harry/Blitz/BlitzSrc to your BlitzSrc directory. Step 2: Compile the programs in ~YourUserName/BlitzSrc There is a “makefile” so you should be able to execute the following commands to compile the tools. cd ~YourUserName/BlitzSrc make This will invoke the “C” and “C++” compilers to produce the following executables: kpl asm lddd blitz diskUtil dumpObj hexdump check endian Step 3: Create a directory for the executables and move them into it: mkdir ~YourUserName/BlitzTools cd ~YourUserName/BlitzSrc mv kpl asm lddd blitz diskUtil dumpObj hexDump check endian ~YourUserName/BlitzTools Step 4E: For Windows Users… Consult the document: www.cs.pdx.edu/~harry/Blitz/OSProject/p1/BlitzOnWindows.pdf www.cs.pdx.edu/~harry/Blitz/OSProject/p1/BlitzOnWindows.htm Project 1 Operating Systems Page 6 Step 5: Modify Your Search Path and Verify the Tools are Working You must add the BlitzTools directory to your shell’s search path so that when you type in the name of a BLITZ tool (such as kpl or blitz), your shell can locate the executable file and execute it. The Unix “shell” program maintains a “shell variable” called PATH which it uses to locate an executable whenever a command name is typed. Details of how to change the PATH variable will vary between the different shells. One approach might be to alter the .aliases file in your home directory. For example, this file may already contain a line that looks something like this: setenv PATH ${PATH}:${HOME}/bin (Between each colon (:) is a directory specification. The above command sets PATH to whatever it was before followed by the bin directory in your home directory.) What you need to do is add the BLITZ tools directory in front of whatever else is in the PATH. Unix / Linux / Mac users who have placed the executables into a subdirectory in their home directory might add the following command to prepend the appropriate directory to the front of the PATH. setenv PATH ${HOME}/BlitzTools:${PATH} Users in a shared Sun/Solaris environment will need to know the shared directory containing the tools. Assume it is: ~instructorUserid/BlitzTools Be sure to get the exact directory name, then add the following command after the last place PATH is set. setenv PATH ~instructorUserid/BlitzTools:${PATH} Portland State University students can add the following command after the last place PATH is set. setenv PATH ~harry/public_html/Blitz/BlitzBin/Sun:${PATH} The “bash” shell is a little different; these people should add something like this to .bashaliases: export PATH=${HOME}/BlitzTools:${PATH} Project 1 Operating Systems Page 7 The shell builds an internal hash table that speeds up the location of programs whenever you type a command. After changing your PATH, you’ll need to restart your shell so that it uses the new PATH when it builds this hash table. You can do this several ways. A Mac user can quit the “Terminal” application and then restart “Terminal”. A Unix / Linux / Solaris user can log out and log back in. In some shells you can simply type the command “source .aliases” instead. Next, verify that whatever you did to the PATH variable worked. At the UNIX/Linux prompt, type the command. kpl You should see the following: ***** ERROR: Missing package name on command line ********** 1 error detected! ********** If you see this, good. If you see anything else, then something is wrong. Step 6: Set up a Directory for Project 1 Create a directory in which to place all files concerned with this class. We recommend a name matching your course number, for example: ~YourUserName/cs333 Create a directory in which to place the files concerned with project 1. We recommend the following name: ~YourUserName/cs333/p1 Copy all files from: https://www.cs.pdx.edu/~harry/Blitz/OSProject/p1/ to your cs333/p1 directory. Project 1 Operating Systems Page 8 The BLITZ Assembly Language In this course you will not have to write any assembly language. However, you will be using some interesting routines which can only be written in assembly. All assembly language routines will be provided to you, but you will need to be able to read them. Take a look at Echo.s and Hello.s to see what BLITZ assembly code looks like. Step 7: Assemble, Link, and Execute the “Hello” Program The p1 directory contains an assembly language program called “Hello.s”. First invoke the assembler (the tool called “asm”) to assemble the program. Type: asm Hello.s This should produce no errors and should create a file called Hello.o. The Hello.s program is completely stand-alone. In other words, it does not need any library functions and does not rely on any operating system. Nevertheless, it must be linked to produce an executable (“a.out” file). The linking is done with the tool called “lddd”. (In UNIX, the linker is called “ld”.) lddd Hello.o –o Hello Normally the executable is called a.out, but the “-o Hello” option will name the executable Hello. Finally, execute this program, using the BLITZ virtual machine. (Sometimes the BLITZ virtual machine is referred to as the “emulator.”) Type: blitz –g Hello The “-g” option is the “auto-go” option and it means begin execution immediately. You should see: Project 1 Operating Systems Page 9 Beginning execution… Hello, world! **** A ‘debug’ instruction was encountered ***** Done! The next instruction to execute will be: 000080: A1FFFFB8 jmp 0xFFFFB8 ! targetAddr = main Entering machine-level debugger… ====================================================== ===== ===== ===== The BLITZ Machine Emulator ===== ===== ===== ===== Copyright 2001-2007, Harry H. Porter III ===== ===== ===== ====================================================== Enter a command at the prompt. Type ‘quit’ to exit or ‘help’ for info about commands. > At the prompt, quit and exit by typing “q” (short for “quit”). You should see this: > q Number of Disk Reads = 0 Number of Disk Writes = 0 Instructions Executed = 1705 Time Spent Sleeping = 0 Total Elapsed Time = 1705 This program terminates by executing the debug machine instruction. This instruction will cause the emulator to stop executing instructions and will throw the emulator into command mode. In command mode, you can enter commands, such as quit. The emulator displays the character “>” as a prompt. After the debug instruction, the Hello program branches back to the beginning. Therefore, if you resume execution (with the go command), it will result in another printout of “Hello, world!”. Step 8: Run the “Echo” Program Type in the following commands: asm Echo.s lddd Echo.o –o Echo blitz Echo Project 1 Operating Systems Page 10 On the last line, we have left out the auto-go “-g” option. Now, the BLITZ emulator will not automatically begin executing; instead it will enter command mode. When it prompts, type the “g” command (short for “go”) to begin execution. Next type some text. Each time the ENTER/RETURN key is pressed, you should see the output echoed. For example: > g Beginning execution… abcd abcd this is a test this is a test q q **** A ‘debug’ instruction was encountered ***** Done! The next instruction to execute will be: cont: 0000A4: A1FFFFAC jmp 0xFFFFAC ! targetAddr = loop > (For clarity, the material entered on the input is underlined.) This program watches for the “q” character and stops when it is typed. If you resume execution with the go command, this program will continue echoing whatever you type. The Echo program is also a stand-alone program, relying on no library functions and no operating system. The KPL Programming Language In this course, you will write code in the “KPL” programming language. Begin studying the document titled “An Overview of KPL: A Kernel Programming Language”. Step 9: Compile and Execute a KPL Program called “HelloWorld” Type the following commands: kpl -unsafe System asm System.s kpl HelloWorld asm HelloWorld.s asm Runtime.s lddd Runtime.o System.o HelloWorld.o -o HelloWorld Project 1 Operating Systems Page 11 There should be no error messages. Take a look at the files HelloWorld.h and HelloWorld.c. These contain the program code. The HelloWorld program makes use of some other code, which is contained in the files System.h and System.c. These must be compiled with the “-unsafe” option. Try leaving this out; you’ll get 17 compiler error messages, such as: System.h:39: ***** ERROR at PTR: Using ‘ptr to void’ is unsafe; you must compile with the ‘unsafe’ option if you wish to do this Using the UNIX compiler convention, this means that the compiler detected an error on line 39 of file System.h. KPL programs are often linked with routines coded in assembly language. Right now, all the assembly code we need is included in a file called Runtime.s. Basically, the assembly code takes care of: Starting up the program Dealing with runtime errors, by printing a message and aborting Printing output (There is no mechanism for input at this stage… This system really needs an OS!) Now execute this program. Type: blitz –g HelloWorld You should see the “Hello, world…” message. What happens if you type “g” at the prompt, to resume instruction execution? The “makefile” The p1 directory contains a file called makefile, which is used with the UNIX make command. Whenever a file in the p1 directory is changed, you can type “make” to re-compile, re-assemble, and relink as necessary to rebuild the executables. Notice that the command kpl HelloWorld will be executed whenever the file System.h is changed. In KPL, files ending in “.h” are called “header files” and files ending in “.c” are called “code files.” Each package (such as HelloWorld) will have both a header file and a code file. The HelloWorld package uses the System package. Whenever the header file of a package that HelloWorld uses is changed, HelloWorld must be recompiled. However, if the code file for System is changed, you do not need to recompile HelloWorld. You only need to relink (i.e., you only need to invoke lddd to produce the executable). Project 1 Operating Systems Page 12 Consult the KPL documentation for more info about the separate compilation of packages. Step 10: Modify the HelloWorld Program Modify the HelloWorld.c program by un-commenting the line –foo (10) In KPL, comments are “–” through end-of-line. Simply remove the hyphens and recompile as necessary, using “make”. The foo function calls bar. Bar does the following things: Increment its argument Print the value Execute a “debug” statement Recursively call itself When you run this program it will print a value and then halt. The keyword debug is a statement that will cause the emulator to halt execution. In later projects, you will probably want to place debug in programs you write when you are debugging, so you can stop execution and look at variables. If you type the go command, the emulator will resume execution. It will print another value and halt again. Type go several times, causing bar to call itself recursively several times. Then try the st command (st is short for “stack”). This will print out the execution stack. Try the fr command (short for “frame”). You should see the values of the local variables in some activation of bar. Try the up and down commands. These move around in the activation stack. You can look at different activations of bar with the fr command. Step 11: Try Some of the Emulator Commands Try the following commands to the emulator. Project 1 Operating Systems Page 13 quit (q) help (h) go (g) step (s) t reset info (i) stack (st) frame (fr) up down Abbreviations are shown in parentheses. The “step” command will execute a single machine-language instruction at a time. You can use it to walk through the execution of an assembly language program, line-by-line. The “t” command will execute a single high-level KPL language statement at a time. Try typing “t” several times to walk through the execution of the HelloWorld program. See what gets printed each time you enter the “t” command. The i command (short for info) prints out the entire state of the (virtual) BLITZ CPU. You can see the contents of all the CPU registers. There are other commands for displaying and modifying the registers. The h command (short for help) lists all the emulator commands. Take a look at what help prints. The reset command re-reads the executable file and fully resets the CPU. This command is useful during debugging. Whenever you wish to re-execute a program (without recompiling anything), you could always quit the emulator and then start it back up. The reset command does the same thing but is faster. Make sure you get familiar with each of the commands listed above; you will be using them later. Feel free to experiment with other commands, too. The “DISK” File The KPL virtual machine (the emulator tool, called “blitz”) simulates a virtual disk. The virtual disk is implemented using a file on the host machine and this file is called “DISK”. The programs in project 1 do not use the disk, so this file is not necessary. However, if the file is missing, the emulator will print a warning. We have included a file called “DISK” to prevent this warning. For more information, try the “format” command in the emulator. Project 1 Operating Systems Page 14 What to Hand In Complete all the above steps. To verify that you did all this, create a transcript of a terminal session showing what happened. In particular, please include the output associated with the following steps in what you hand in. Step 7 Step 8 Step 9 Step 11 We do not need to see the other steps. Hand in a hardcopy print-out showing what happened. If you do not know about creating a script file, check out the UNIX script command by typing man script In LARGE BLOCK LETTERS, write your full name. Note that if you try to use a text editor while running script, a bunch of garbage characters may be put into the file. Please do not do this. After you have created your script file, it is okay to edit it to remove the entire editing session. We really don’t want to see a transcript of your editing session. Alternately, you can start and stop script, creating several script files and then concatenate them. PLEASE STAPLE ALL PAGES! Grading for this Project This project will be graded pass/fail.Overview and Goal In this project you will learn about threads and gain familiarity writing programs involving concurrency control. You will begin by studying the thread package, which implements multi-threading. You will make modifications and additions to the existing code. Then you will use the threads package to solve some traditional concurrency problems. In addition, you will gain familiarity programming in the KPL language. If you have difficulties with this project or want to go into more depth, take a look at the document called “The Thread Scheduler and Concurrency Control Primitives”: https://web.cecs.pdx.edu/~harry/Blitz/BlitzDoc/ThreadScheduler.htm https://web.cecs.pdx.edu/~harry/Blitz/BlitzDoc/ThreadScheduler.pdf Step 1: Download the Files Start by creating a directory for the files you’ll need for this project. You might call it: ~YourUserName/cs333/p2 Then download all the files from: https://www.cs.pdx.edu/~harry/Blitz/OSProject/p2/ into your directory. You should get the following files: makefile DISK System.h System.c Runtime.s Switch.s List.h List.c Thread.h Project 2 Operating Systems Page 2 Thread.c Main.h Main.c Synch.h Synch.c In this project, you will modify and hand in the following files: Main.c Synch.h Synch.c After getting the files, you should be able to compile all the code (as is) with the UNIX make command. The program executable we are building will be called “os”. You can execute the program by typing: % make % blitz –g os In the course of your experimentation, you may modify other files besides Synch.h, Synch.c and Main.c, but the code you are required to write and turn in doesn’t require any changes to the other files. For example, you may wish to uncomment some of the print statements, to see what happens. However, your final versions of Synch.h, Synch.c and Main.c must work with the other files, exactly as they are distributed. Be sure you copy all files. Even though there are similarities with some of the files used for the previous project, there may be some subtle but important differences. Step 2: Study the Existing Code The code provided in this project provides the ability to create and run multiple threads, and to control concurrency through several synchronization methods. Start by looking over the System package. Focus on the material toward the beginning of file System.c, namely the functions: print printInt printHex printChar printBool nl MemoryEqual StrEqual StrCopy StrCmp Project 2 Operating Systems Page 3 Min Max printIntVar printHexVar printBoolVar printCharVar printPtr Get familiar with the printing functions; you’ll be calling them a lot. Some are implemented in assembly code and some are implemented in KPL in the System package. The following functions are used to implement the heap in KPL: KPLSystemInitialize KPLMemoryAlloc KPLMemoryFree Objects can be allocated on the heap and freed with the alloc and free statements. The HEAP implementation is very rudimentary in this implementation. In your kernel, you may allocate objects during start-up but after that, YOU MUST NOT ALLOCATE OBJECTS ON THE HEAP! Why? Because the heap might fill up and then what is a kernel supposed to do? Crash. In this project, you should not allocate anything on the heap. The following functions can be ignored since they concern aspects of the KPL language that we will not be using: KPLUncaughtThrow UncaughtThrowError KPLIsKindOf KPLSystemError The Runtime.s file contains a number of routines coded in assembly language. It contains the program entry point and the interrupt vector in low memory. Take a look at it. Follow what happens when program execution begins at location 0x00000000 (the label “_entry”). The code labeled “_mainEntry” is included in code the compiler produces. The “_mainEntry” code will call the main function, which appears in the file Main.c. In Runtime.s, follow what happens when a timer interrupt occurs. It makes an “up-call” to a routine called _P_Thread_TimerInterruptHandler. This name refers to “a function called TimerInterruptHandler in a package called Thread.” (_P_Thread_TimerInterruptHandler is the name the compiler will give this function.) All the code in this project assumes that no other interrupt types (such as a DiskInterrupt) occur. In Runtime.s, follow what would happen if another sort of interrupt should ever occur. Project 2 Operating Systems Page 4 The KPL language will check for lots of error conditions, such as the use of a null pointer. Try changing the program to make this error. Follow in Runtime.s what happens when this occurs. Next take a look at the List package. Read the header file carefully. This package provides code that implements a linked list. We’ll use linked lists in this project. For example, the threads that are ready to run (and waiting for time on the CPU) will be kept in a linked list called the “ready list”. Threads that become BLOCKED will sit on other linked lists. Also look over the List.c code file. The key class in this project is named Thread, and it is located in the Thread package along with other stuff (see Thread.h, Thread.c). For each thread, there will be a single Thread object. Thread is a subclass of Listable, which means that each Thread object contains a next pointer and can be added to a linked list. The Thread package is central and you should study this code thoroughly. This package contains one class (called Thread) and several functions related to thread scheduling and time-slicing: InitializeScheduler () IdleFunction (arg: int) Run (nextThread: ptr to Thread) PrintReadyList () ThreadStartMain () ThreadFinish () FatalError (errorMessage: ptr to array of char) SetInterruptsTo (newStatus: int) returns int TimerInterruptHandler () FatalError is the simplest function. We will call FatalError whenever we wish to print an error message and abort the program. Typically, we’ll call FatalError after making some check and finding that things are not as expected. FatalError will print the name of the thread invoking it, print the message, and then shut down. It will throw us into the BLITZ emulator command line mode. Normally, the next thing to do might be to type the “st” command (short for “stack”), to see which functions and methods were active. (Of course the information printed out by the emulator will pertain to only the thread that invoked FatalError. The emulator does not know about threads, and it is pretty much impossible to extract information about other threads by examining bytes in memory.) The next function to look at is SetInterruptsTo, which is used to change the “I” interrupt bit in the CPU. We can use it to disable interrupts with code like this: … = SetInterruptsTo (DISABLED) and we can use it to enable interrupts: … = SetInterruptsTo (ENABLED) Project 2 Operating Systems Page 5 This function returns the previous status. This is very useful because we often want to DISABLE interrupts (regardless of what they were before) and then later we want to return the interrupt status to whatever it was before. In our kernel, we’ll often see code like: var oldIntStat: int … oldIntStat = SetInterruptsTo (DISABLED) … oldIntStat = SetInterruptsTo (oldIntStat) Next take a look at the Thread class. Here are the fields of Thread: name: ptr to array of char status: int systemStack: array [SYSTEM_STACK_SIZE] of int regs: array [13] of int stackTop: ptr to void initialFunction: ptr to function (int) initialArgument: int Here are the operations (i.e., methods) you can do on a Thread: Init (n: ptr to array of char) Fork (fun: ptr to function (int), arg: int) Yield () Sleep () CheckOverflow () Print () Each thread is in one of the following states: JUST_CREATED, READY, RUNNING, BLOCKED, and UNUSED, and this is given in the status field. (The UNUSED status is given to a Thread after it has terminated. We’ll need this in later projects.) Each thread has a name. To create a thread, you’ll need a Thread variable. First, use Init to initialize it, providing a name. Each thread needs its own stack and space for this stack is placed directly in the Thread object in the field called systemStack. Currently, this is an array of 1000 words, which should be enough. (It is conceivable our code could overflow this limit; there is a check to make sure we don’t overflow this limited area.) All threads in this project will run in System mode. Therefore the stack is called the “system stack”. In later projects, we’ll see that this stack is used only for kernel routines. User programs will have their own stacks in their virtual address spaces, but we are getting ahead of ourselves. Project 2 Operating Systems Page 6 The Thread object also has space to store the state of the CPU, namely the registers. Whenever a thread switch occurs, the registers will be saved in the Thread object. These fields (regs and stackTop) are used by the assembly code routine named Switch. The Thread object also has space to store a pointer to a function (the initialFunction field) and an argument for this function (the initialArgument field). This pointer will point to the “main” function of this thread; this is the function that will get executed when this thread begins execution. We are storing a pointer to the function because this is a variable: different threads may execute different functions. We are also able to supply an initial argument to this thread, through the initialArgument field. This argument must be an integer. Often there will be several threads executing the same “main” function. The argument is a handy way to let each thread know what its role should be. For example, we might create 10 threads each using the same “main” function, but passing each thread a different integer (say, between 1 and 10) to let it know which thread it is. After initializing a new Thread, we can start it running with the Fork method. This doesn’t immediately begin the thread execution; instead it makes the thread READY to run and places it on the readyList. The readyList is a linked list of Threads. All Threads on the readyList have status READY. ReadyList is a global variable. There is another global variable named currentThread, which points to the currently executing Thread object; i.e., the Thread whose status is RUNNING. The Yield method should only be invoked on the currently running thread. It will cause a switch to some other thread. Follow the code in Yield closely to see what happens when a thread switch occurs. First, interrupts are disabled; we don’t want any interference during a thread switch. The readyList and currentThread are shared variables and, while switching threads, we want to be able to access and update them safely. Then Yield will find the next thread from the readyList. (If there is no other thread, then Yield is effectively a nop.) Then Yield will make the currently running process READY (i.e., no longer RUNNING) and it will add the current thread to the tail end of the readyList. Finally, it will call the Run function to do the thread switch. The Run method will check for stack overflow on the current thread. It will then call Switch to do the actual Switch. Switch may be the most fascinating function you ever encounter! It is located in the assembly code file Switch.s, which you should look at carefully. Switch does not return to the function that called it. Instead, it switches to another thread. Then it returns. Therefore, the return happens to another function in another thread! The only place Switch is called is from the Run function, so Switch returns to some invocation of the Run function in some other thread. That copy (i.e., invocation) of Run will then return to whoever called it. This could have been some other call to Yield, so we’ll return to another Yield which will return to whoever called it. Project 2 Operating Systems Page 7 And this is exactly the desired functionality of Yield! A call to Yield should give up the processor for a while, and eventually return after other threads have had a chance to execute. Run is also called from Sleep, so we might be returning from a call to Sleep after a thread switch. How is everything set up when a thread is first created? How can we “return to a function” when we have not ever called it? Take a look at function ThreadStartMain in file Thread.c and look at function ThreadStartUp in file Switch.s. What happens when a thread is terminated? Take a look at ThreadFinish in file Thread.c. Essentially, the thread is put to sleep with no hope of ever being awakened. (No wonder they call it “Thread Death!”) Next, take a look at what happens when a Timer interrupt occurs while some thread is executing. This is an interrupt, so the CPU begins by interrupting the current routine’s execution and pushing some state onto its (system) stack. Then it disables interrupts and jumps to the assembly code routine called TimerInterruptHandler in Runtime.s, which just calls the TimerInterruptHandler function in Thread.c. In TimerInterruptHandler, we call Yield, which then switches to another thread. Later, we’ll come back here, when this thread gets another chance to run. Then, we’ll return to the assembly language routine which will execute a “reti” instruction. This will restore the state to exactly what it was before and the interrupted routine (whatever it was) will get to continue. Note that this code maintains a variable called currentInterruptStatus. This is because it is rather difficult to query the “I” bit of the CPU. It is easier to just change the variable whenever a change to the interrupt status changes. We see this occurring in the TimerInterruptHandler function. Clearly interrupts will be disabled immediately after the interrupt occurs. And the Yield function will preserve the interrupt status. So when we return from Yield, interrupts will once again be disabled. Before returning to the interrupted thread, we set the currentInterruptStatus to ENABLED. (They must have been enabled before the interrupt occurred—or else it could not have occurred—so after we execute the “reti” instruction, the status will revert to what it was before, namely ENABLED.) Now you are ready to start playing with and modifying the code! Please experiment with the code we have just discussed, as necessary to understand it. Step 3: Run the “SimpleThreadExample” Code Execute and trace through the output of SimpleThreadExample in file Main.c. In TimerInterruptHandler there is a statement printChar (‘_’) Project 2 Operating Systems Page 8 which is commented out. Try uncommenting it. Make sure you understand the output. In TimerInterruptHandler, there is a call to Yield. Why is this there? Try commenting this statement? What happens. Make sure you understand how Yield works here. Step 4: Run the “MoreThreadExamples” Code Trace through the output. Try changing this code to see what happens. Step 5: Implement the “Mutex” Class In this part, you must implement the class Mutex. The class specification for Mutex is given to you in Synch.h: class Mutex superclass Object methods Init () Lock () Unlock () IsHeldByCurrentThread () returns bool endClass You will need to provide code for each of these methods. In Synch.c you’ll see a behavior construct for Mutex. There are methods for Init, Lock, Unlock, and IsHeldByCurrentThread, but these have dummy bodies. You’ll need to write the code for these four methods. You will also need to add a couple of fields to the class specification of Mutex to implement the desired functionality. How can you implement the Mutex class? Take a close look at the Semaphore class; your implementation of Mutex will be quite similar. First consider the IsHeldByCurrentThread method, which may be invoked by any thread. The code of this method will need to know which thread is holding a lock on the mutex; then it can compare that to the currentThread to see if they are the same. So, you might consider adding a field (perhaps called heldBy) to the Mutex class, which will be a pointer to the thread holding the mutex. Of course, you’ll need to set it to the current thread whenever the mutex is locked. You might use a null value in this field to indicate that no thread is holding a lock on the mutex. Project 2 Operating Systems Page 9 When a lock is requested on the mutex, you’ll need to see if any thread already has a lock on this mutex. If so, you’ll need to put the current process to sleep. For putting a thread to sleep, take a look at the method Semaphore.Down. At any one time, there may be zero, one, or many threads waiting to acquire a lock on the mutex; you’ll need to keep a list of these threads so that when an Unlock is executed, you can wake up one of them. As in the case of Semaphores, you should use a FIFO queue, waking up the thread that has been waiting longest. When a mutex lock is released (in the Unlock method), you’ll need to see if there are any threads waiting to acquire a lock on the mutex. You can choose one and move it back onto the readyList. Now the waiting thread will begin running when it gets a turn. The code in Semaphore.Up does something similar. It is also a good idea to add an error check in the Lock method to make sure that the current thread asking to lock the mutex doesn’t already hold a lock on the mutex. If it does, you can simply invoke FatalError. (This would probably indicate a logic error in the code using the mutex. It would lead to a deadlock, with a thread frozen forever, waiting for itself to release the lock.) Likewise, you should also add a check in Unlock to make sure the current thread really does hold the lock and call FatalError if not. You’ll be using your Mutex class later, so these checks will help your debugging in later projects. The function TestMutex in Main.c is provided to exercise your implementation of Mutex. It creates 7 threads that compete vigorously for a single mutex lock. Step 6: Implement the Producer-Consumer Solution Both the Tanenbaum and Silberschatz, et al. textbooks contain a discussion of the Producer-Consumer problem, including a solution. Implement this in KPL using the classes Mutex and Semaphore. Deal with multiple producers and multiple consumers, all sharing a single bounded buffer. The Main package contains some code that will serve as a framework. The buffer is called buffer and contains up to BUFFER_SIZE (e.g., 5) characters. There are 5 producer processes, each modeled by a thread, and 3 consumer processes, each modeled by a thread. Thus, there are 8 threads in addition to the main thread that creates the others. Each producer will loop, adding 5 characters to the buffer. The first producer will add five ‘A’ characters, the second producer will add five ‘B’s, etc. However, since the execution of these threads will be interleaved, the characters will be added in a somewhat random order. Step 7: Implement the Dining Philosopher’s Solution Using a Monitor A starting framework for your solution is provided in Main.c. Each philosopher is modeled with a thread and the code we’ve provided sets up these threads. The synchronization will be controlled by a “monitor” called ForkMonitor. Project 2 Operating Systems Page 10 The code for each thread/philosopher is provided for you. Look over the PhilosophizeAndEat method; you should not need to change this code. The monitor to control synchronization between the threads is implemented with a class called ForkMonitor. The following class specification of ForkMonitor is provided: class ForkMonitor superclass Object fields status: array [5] of int — For each philosopher: HUNGRY, — EATING, or THINKING methods Init () PickupForks (p: int) PutDownForks (p: int) PrintAllStatus () endClass You’ll need to provide the code for the Init, PickupForks and PutDownForks methods. You’ll also need to add additional fields and perhaps even add another method. The code for PrintAllStatus is provided. You should call this method whenever you change the status of any philosopher. This method will print a line of output, so you can see what is happening. How can you proceed? You’ll need a mutex to protect the monitor itself. There are two main methods (PickupForks and PutDownForks) which are called by the philosopher threads. Upon beginning each of these methods, the first thing is to lock the monitor mutex. This will ensure that only one thread at a time is executing within the monitor. Just before each of these methods returns, it must unlock the monitor (by unlocking the monitor’s mutex) so that other threads can enter the monitor code. You’ll also need to use the Condition class, which is provided in the Synch package. (The Condition class uses the class Mutex, so it is assumed that you’ve finished and tested the Mutex class.) The BLITZ emulator has a number of parameters and one of these is how often a timer interrupt occurs. The default value is every 5000 instructions. You might try changing this parameter to see how it affects your programs behavior. To change the simulation parameters, type the sim command into the emulator. This command will give you the option to create a file called .blitzrc After creating this file, you can edit it by hand. The next time you run the emulator, it will use this new value. Also note that too small a value—like 1000—will cause the program to hang. What do you suppose causes this effect? Project 2 Operating Systems Page 11 QUESTION: Both textbooks discuss the semantics of signaling a condition variable. They mention “Hoare semantics.” The comments in the code in Synch.c say that this version implements “Mesastyle” semantics. Is this the same or different from Hoare semantics? An Example of Correct Output The following files contain an example of what correct output should look like: DesiredOutput1.pdf DesiredOutput2.pdf DesiredOutput3.pdf What to Hand In Complete all the above steps. Please submit hardcopy of the following files: Synch.h Synch.c Main.c Also include hardcopy showing the output for steps 5, 6, and 7. Please print both your code and the output using a fixed-width font like this, in this and all future assignments. Much of the output is designed to look nice when printed with a fixed-width font, but is more difficult to read in a standard variable-width font. In LARGE BLOCK LETTERS, write your full name on the first page. PLEASE STAPLE ALL PAGES! Basis for Grading In this course, the code you submit will be graded on both correctness and style. Correctness means that the code must work right and not contain bugs. Style means that the code must be neat, well commented, well organized, and easy to read. In style, your code should match the code we are distributing to you. The indentation should be the same and the degree of commenting should be the same.Overview and Goal In this project you will gain additional familiarity writing programs involving concurrency control. You will implement a couple of classical IPC problems to become more comfortable with Semaphores, Mutexes, Condition Variables, and the Monitor approach. Download New Files Start by creating a directory for the files you’ll need for this project: ~YourUserName/cs333/p3 Then download all the files from: https://www.cs.pdx.edu/~harry/Blitz/OSProject/p3/ Even though some of the files have the same names, be sure to get new copies for each project. In general some files may be modified. Please keep your old files from previous projects separate and don’t modify them once you submit them. This is a good rule for all programming projects in all classes. If there is ever any question about whether code was completed on time, we can always go back and look at the Unix “file modification date” information. For this project, you should get the following files: makefile DISK System.h System.c Runtime.s Switch.s List.h List.c Project 3 Operating Systems Page 2 Thread.h Thread.c Synch.h Synch.c Main.h Main.c The file Synch.c contains our solution for project 2. You may use either our version of this file, or the version of Synch.c you created. We have also included the file Proj2Sol.pdf which contains my solution code for Producer-Consumer and Dining Philosophers. If you had difficulty with it, you can take a look at the solution code. You will modify and submit the following files: Main.h Main.c Task 1: Implement the Sleeping Barber Problem An older edition of the Tanenbaum textbook describes the “Sleeping Barber” problem and gives a solution in “C”. (This material can also be found in a separate file called SleepingBarberProblem.pdf in the p3 directory.) Translate this into a working KPL program. You’ll also need to create some code to print out what happens when you run the program. What will you do for the “cut_hair” and the “get_haircut” methods? You’ll need at least one print statement in each routine, but be careful: they both should run at more-or-less the same time. One barber is okay, but how many customers will you model? Will you just throw a bunch of customers at the barbershop all at once? This might not test your code very well. How long will the haircut last? You can implement waiting with a busy loop, such as for i = 1 to 100 .. want to yield here?… endFor The goals are: Project 3 Operating Systems Page 3 (1) Practice creating a KPL class from scratch (2) Practice testing your code (3) Gain experience with semaphores, mutex locks, and thread synchronization Task 2: The Gaming Parlor Problem Here is the problem scenario: groups of customers come in to a “gaming parlor” to play games. They go to the front desk to obtain one or more dice, which are used by the group while they are playing their game, and then returned to the front desk. The front desk is in charge of lending out the dice and collecting them after each game is finished. The gaming parlor owns only 8 dice, which are available at the front desk before the first group comes in. The customers can play the following games. Listed after each game in parentheses is the number of dice required to play that game. Backgammon (4) Risk (5) Monopoly (2) Pictionary (1) You should model the front desk as a monitor with the following entry methods: Request (numberOfDice: int) Return (numberOfDice: int) Model each group of customers as a thread. When a group is ready to play, it must obtain the necessary number of dice. If the required number of dice is not available, then the group (i.e., the thread) must wait. You might use a condition variable that “more dice have become available.” You should model the following eight different groups. Each group plays one game, as shown below, but each group plays its game 5 times. Each group must return their dice after each game and then reacquire the dice before playing again. A – Backgammon B – Backgammon C – Risk D – Risk E – Monopoly F – Monopoly G – Pictionary H – Pictionary Project 3 Operating Systems Page 4 To simulate playing the game, simply call the Yield method. This problem is a generalization of the problem of resource allocation where (1) there are a number of resources (dice) but each is identical; (2) every requesting process needs a one or more units of the resource, (3) each requesting thread knows how many units it will need before requesting any units and that info is included in the request, (4) all units are returned before any further requests are made. In a monitor, each method is either an “entry” method or a local method. A monitor will always have one Mutex lock, which is associated with entering the monitor. Every monitor will also have zero or more condition variables, as required by the particular application. An entry method must always begin by locking the monitor’s mutex. Before it returns, it must always unlock the mutex. Within the entry method, the code may Signal some of the condition variables and may Wait on condition variables. (In some applications, it may make sense to Broadcast to a condition variable.) However, once inside of an entry method, the code should never touch the monitor’s mutex lock, or try to look inside the condition variables. You should model the front desk as a monitor and you should use condition variables instead of semaphores. If deadlock occurs, the program will freeze up and some requests for dice will go unsatisfied forever. This is a disaster! Regardless of the order in which the groups make their requests, your solution should be structured such that deadlock can never occur. Your solution must not be subject to any race conditions. In other words, regardless of the order in which the groups make their requests and return their dice, each die must never be allocated to more than one group at a time. It should never be the case that groups are allowed to proceed when there are too few dice. Likewise, if a group has returned its dice, other groups which are waiting must be allowed to proceed once enough dice have become available. Starvation can occur if it is possible that one thread can be delayed infinitely by other threads’ requests. If the game parlor problem is extended to assume that each group will continue to play game after game forever, then starvation might be possible, if you are not careful in your solution. If some group X comes in requesting a lot of dice, it will be made to wait until enough dice are available. If it is possible that other groups can come in, request a small number of dice, and have their requests granted, then group X might get delayed forever, since there are never enough dice at the front desk at once to satisfy group X’s needs. In other words, it might be possible for starvation of X to occur. In your solution starvation should not be possible. (Of course, with each group limited to playing only 5 games, all the outstanding dice will always get returned eventually and starvation can never occur, but your solution should also avoid starvation in the presence of a never-ending sequence of Request and Return operations.) To verify that your code is working, please insert print statements to produce output like this… Project 3 Operating Systems Page 5 Initializing Thread Scheduler… A requests 4 ——————————Number of dice now avail = 8 A proceeds with 4 ——————————Number of dice now avail = 4 B requests 4 ——————————Number of dice now avail = 4 B proceeds with 4 ——————————Number of dice now avail = 0 D requests 5 ——————————Number of dice now avail = 0 E requests 2 ——————————Number of dice now avail = 0 A releases and adds back 4 ——————————Number of dice now avail = 4 B releases and adds back 4 ——————————Number of dice now avail = 8 C requests 5 ——————————Number of dice now avail = 8 H requests 1 ——————————Number of dice now avail = 8 B requests 4 ——————————Number of dice now avail = 8 D proceeds with 5 ——————————Number of dice now avail = 3 …etc… This output makes it fairly easy to see what the program is doing and verify that it is correct. Below is a method you should use to produce your output. Feel free to copy this method straight into your program. behavior GameParlor … Other methods… method Print (str: String, count: int) — — This method prints the current thread’s name and the arguments. — It also prints the current number of dice available. — print (currentThread.name) print (” “) print (str) print (” “) printInt (count) nl () print (“——————————Number of dice now avail = “) printInt (numberDiceAvail) nl () endMethod Project 3 Operating Systems Page 6 endBehavior This method would be called in several places… At the beginning of the Request method: self.Print (“requests”, numNeeded) At the end of the Request method: self.Print (“proceeds with”, numNeeded) In the Return method: self.Print (“releases and adds back”, numReturned) What to Hand In Please submit hardcopy of the following files: Main.h Main.c Also hand in hardcopy showing your output for both tasks. For the Sleeping Barber problem, I am not providing any testing material; you’ll have to devise some tests on your own. Please hand in something showing how you have tested your code. In LARGE BLOCK LETTERS, write your full name on the first page. PLEASE STAPLE ALL PAGES! Basis for Grading In this course, the code you submit will be graded on both correctness and style. Correctness means that the code must work right and not contain bugs. Style means that the code must be neat, well commented, well organized, and easy to read. In style, your code should match the code we are distributing to you. The indentation should be the same and the degree of commenting should be the same.Overview and Goal In this project, you will implement three monitors that will be used in our Kernel. These are the ThreadManager, the ProcessManager, and the FrameManager. The code you write will be similar to other code from the previous projects in that these three monitors will orchestrate the allocation and freeing of resources. There is also an additional task—re-implement the Condition and Mutex classes to provide Hoare Semantics—but that code will not be used in the Kernel. Download New Files Start by creating a new directory for this project and then download all the files from: https://www.cs.pdx.edu/~harry/Blitz/OSProject/p4/ Even though some of the files have the same names, be sure to get new copies for each project. In general some files may be modified. Please keep your old files from previous projects separate and don’t modify them once you submit them. This is a good rule for all programming projects in all classes. If there is ever any question about whether code was completed on time, we can always go back and look at the Unix “file modification date” information. For this project, you should get the following files: makefile DISK Runtime.s Switch.s System.h System.c List.h Project 4 Operating Systems Page 2 List.c BitMap.h BitMap.c Main.h Main.c Kernel.h Kernel.c The packages called Thread and Synch have been merged together into one package, now called Kernel. This package contains quite a bit of other material as well, which will be used for later projects. In this and the remaining projects, you will be modifying the Kernel.c and Kernel.h files. Don’t modify the code that is not used in this project; just leave it in the package. The Kernel.c file contains the following stuff, in this order: Thread scheduler functions Semaphore class Mutex class Condition class Thread class ThreadManager class ProcessControlBlock class ProcessManager class FrameManager class AddrSpace class TimerInterruptHandler other interrupt handlers SyscallTrapHandler Handle functions In this project, you can ignore everything after TimerInterruptHandler. The classes called ThreadManager, ProcessManager, and FrameManager are provided in outline, but the bodies of the methods are unimplemented. You will add implementations. Some other methods are marked “unimplemented;” those will be implemented in later projects. The BitMap package contains code you will use; read over it but do not modify it. The makefile has been modified to compile the new code. As before, it produces an executable called os. You may modify the file Main.c while testing, but when you do your final run, please use the Main.c file as it was distributed. In the final version of our kernel, the Main package will perform all initialization and will create the first thread. The current version performs initialization and then calls some testing functions. Project 4 Operating Systems Page 3 Task 1: Threads and the ThreadManager In this task, you will modify the ThreadManager class and provide implementations for the following methods: Init GetANewThread FreeThread In our kernel, we will avoid allocating dynamic memory. In other words, we will not use the heap. All important resources will be created at startup time and then we will carefully monitor their allocation and deallocation. An example of an important resource is Thread objects. Since we will not be able to allocate new objects on the heap while the kernel is running, all the Thread objects must be created ahead of time. Obviously, we can’t predict how many threads we will need, so we will allocate a fixed number of Thread objects (e.g., 10) and re-use them. When a user process starts up, the kernel will need to obtain a new Thread object for it. When a process dies, the Thread object must be returned to a pool of free Thread objects, so it can be recycled for another process. Kernel.h contains the line: const MAX_NUMBER_OF_PROCESSES = 10 Since each process in our OS will have at most one thread, we will also use this number to determine how many Thread objects to place into the free pool initially. To manage the Thread objects, we will use the ThreadManager class. There will be only one instance of this class, called threadManager, and it is created and initialized at startup time in Main.c. Whenever you need a new Thread object, you can invoke threadManger.GetANewThread. This method should suspend and wait if there are currently none available. Whenever a thread terminates, the scheduler will invoke FreeThread. In fact, the Run function has been modified in this project to invoke FreeThread when a thread terminates—thereby adding it to the free list—instead of setting its status to UNUSED. Here is the definition of ThreadManager as initially distributed: Project 4 Operating Systems Page 4 class ThreadManager superclass Object fields threadTable: array [MAX_NUMBER_OF_PROCESSES] of Thread freeList: List [Thread] methods Init () Print () GetANewThread () returns ptr to Thread FreeThread (th: ptr to Thread) endClass When you write the Init method, you’ll need to initialize the array of Threads and you’ll need to initialize each Thread in the array and set its status to UNUSED. (Each Thread will have one of the following as its status: READY, RUNNING, BLOCKED, JUST_CREATED, and UNUSED.) Threads should have the status UNUSED iff they are on the freeList. You’ll also need to initialize the freeList and place all Threads in the threadTable array on the freeList during initialization. You will need to turn the ThreadManager into a “monitor.” To do this, you might consider adding a Mutex lock (perhaps called threadManagerLock) and a condition variable (perhaps called aThreadBecameFree) to the ThreadManager class. The Init method will also need to initialize threadManagerLock and aThreadBecameFree. The GetANewThread and FreeThread methods are both “entry methods,” so they must obtain the monitor lock in the first statement and release it in the last statement. GetANewThread will remove and return a Thread from the freeList. If the freeList is empty, this method will need to wait on the condition of a thread becoming available. The FreeThread method will add a Thread back to the freeList and signal anyone waiting on the condition. The GetANewThread method should also change the Thread status to JUST_CREATED and the FreeThread method should change it back to UNUSED. We have provided code for the Print method to print out the entire table of Threads. Note that the Print method disables interrupts. The Print method is used only while debugging and will not be called in a running OS so this is okay. Within the Print method, we want to get a clean picture of the system state—a “snapshot”—(without worrying about what other threads may be doing) so disabling interrupts seems acceptable. However, the other methods—Init, GetAThread and FreeThread—must NOT disable interrupts, beyond what is done within the implementations of Mutex, Condition, etc. In Main.c we have provided a test routine called RunThreadManagerTests, which creates 20 threads to simultaneously invoke GetAThread and FreeThread. Let’s call these the “testing threads” as opposed to the “resource threads,” which are the objects that the ThreadManager will allocate and monitor. There are 20 testing threads and only 10 resource thread objects. Every thread that terminates will be added back to the freeList (by Run, which calls FreeThread). Since the testing threads were never obtained by a call to GetANewThread, it would be wrong to add Project 4 Operating Systems Page 5 them back to the freeList. Therefore, each testing thread does not actually terminate. Instead it freezes up by waiting on a semaphore that is never signaled. By the way, the testing threads are allocated on the heap, in violation of the principle that the kernel must never allocate anything on the heap, but this is okay, since this is only debugging code, which will not become a part of the kernel. In the kernel, we may have threads that are not part of the threadTable pool (such as the IdleThread), but these threads must never terminate, so there is no possibility that they will be put onto the freeList. Thus, the only things on the freeList should be Threads from threadTable. You will also notice that the Thread class has been changed slightly to add the following fields: class Thread … fields … isUserThread: bool userRegs: array [15] of int — Space for r1..r15 myProcess: ptr to ProcessControlBlock methods … endClass These fields will be used in a later project. The Thread methods are unchanged. Task 2: Processes and the ProcessManager In our kernel, each user-level process will contain only one thread. For each process, there will be a single ProcessContolBlock object containing the per-process information, such as information about open files and the process’s address space. Each ProcessControlBlock object will point to a Thread object and each Thread object will point back to the ProcessControlBlock. There may be other threads, called “kernel threads,” which are not associated with any user-level process. There will only be a small, fixed number of kernel threads and these will be created at kernel start-up time. For now, we will only have a modest number of ProcessControlBlocks, which will make our testing job a little easier, but in a real OS this constant would be larger. const MAX_NUMBER_OF_PROCESSES = 10 All processes will be preallocated in an array called processTable, which will be managed by the ProcessManager object, much like the Thread objects are managed by the ThreadManager object. Each process will be represented with an object of this class: Project 4 Operating Systems Page 6 class ProcessControlBlock superclass Listable fields pid: int parentsPid: int status: int — ACTIVE, ZOMBIE, or FREE myThread: ptr to Thread exitStatus: int addrSpace: AddrSpace fileDescriptor: array [MAX_FILES_PER_PROCESS] of ptr to OpenFile methods Init () Print () PrintShort () endClass Each process will have a process ID (the field named pid). Each process ID will be a unique number, from 1 on up. Processes will be related to other processes in a hierarchical parent-child tree. Each process will know who its parent process is. The field called parentsPid is a integer identifying the parent. One parent may have zero, one, or many child processes. To find the children of process X, we will have to search all processes for processes whose parentsPid matches X’s pid. The ProcessControlBlock objects will be more like C structs than full-blown C++/Java objects: the fields will be accessed from outside the class but the class will not contain many methods of its own. Other than initializing the object and a couple of print methods, there will be no other methods for ProcessControlBlock. We are providing the implementations for the Init, Print and PrintShort methods. Since we will have only a fixed, small number of ProcesControlBlocks, these are resources which must be allocated. This is the purpose of the monitor class called ProcessManager. At start-up time, all ProcessControlBlocks are initially FREE. As user-level processes are created, these objects will be allocated and when the user-level process dies, the corresponding ProcessControlBlock will become FREE once again. In Unix and in our kernel, death is a two stage process. First, an ACTIVE process will execute some system call (e.g., Exit()) when it wants to terminate. Although the thread will be terminated, the ProcessControlBlock cannot be immediately freed, so the process will then become a ZOMBIE. At some later time, when we are done with the ProcessControlBlock it can be FREEd. Once it is FREE, it is added to the freeList and can be reused when a new process is begun. The exitStatus is only valid after a process has terminated (e.g., a call to Exit()). So a ZOMBIE process has a terminated thread and a valid exitStatus. The ZOMBIE state is necessary just to keep the exit status around. The reason we cannot free the ProcessControlBlock is because we need somewhere to store this integer. Project 4 Operating Systems Page 7 For this project, we will ignore the exitStatus. It need not be initialized, since the default initialization (to zero) is fine. Also, we will ignore the ZOMBIE state. Every process will be either ACTIVE or FREE. Each user-level process will have a virtual address space and this is described by the field addrSpace. The code we have supplied for ProcessControlBlock.Init will initialize the addrSpace. Although the addrSpace will not be used in this project, it will be discussed later in this document. The myThread field will point to the process’s Thread, but we will not set it in this project. The fileDescriptors field describes the files that this process has open. It will not be used in this project. Here is the definition of the ProcessManager object. class ProcessManager superclass Object fields processTable: array [MAX_NUM_OF_PROCESSES] of ProcessControlBlock processManagerLock: Mutex aProcessBecameFree: Condition freeList: List [ProcessControlBlock] aProcessDied: Condition methods Init () Print () PrintShort () GetANewProcess () returns ptr to ProcessControlBlock FreeProcess (p: ptr to ProcessControlBlock) TurnIntoZombie (p: ptr to ProcessControlBlock) WaitForZombie (proc: ptr to ProcessControlBlock) returns int endClass There will be only one ProcessManager and this instance (initialized at start-up time) will be called processManager. processManager = new ProcessManager processManager.Init () The Print() and PrintShort() methods for ProcessControlBlocks are provided for you. You are to implement the methods Init, GetANewProcess, and FreeProcess. The methods TurnIntoZombie and WaitForZombie will be implemented in a later project and can be ignored for now. The freeList is a list of all ProcessControlBlocks that are FREE. The status of a ProcessControlBlock should be FREE if and only if it is on the freeList. We assume that several threads may more-or-less simultaneously request a new ProcessControlBlock by calling GetANewProcess. The ProcessManager should be a “monitor,” in order to protect the freeList from concurrent access. The Mutex called processManagerLock is for that purpose. When a Project 4 Operating Systems Page 8 ProcessControlBlock is added to the freeList, the condition aProcessBecameFree can be Signaled to wake up any thread waiting for a ProcessControlBlock. Initializing the ProcessControlManager should initialize the processTable array all the ProcessControlBlocks in that array the processManagerLock the aProcessBecameFree and the aProcessDied condition variables the freeList All ProcessControlBlocks should be initialized and placed on the freeList. The condition called aProcessDied is signaled when a process goes from ACTIVE to ZOMBIE. It will not be used in this project, but should be initialized nonetheless. The GetANewProcess method is similar to the GetANewThread method, except that it must also assign a process ID. In other words, it must set the pid. The ProcessManager will need to manage a single integer for this purpose. (Perhaps you might call it nextPid). Every time a ProcessControlBlock is allocated (i.e., everytime GetANewProcess is called), this integer must be incremented and used to set the process’s pid. GetANewProcess should also set the process’s status to ACTIVE. The FreeProcess method must change the process’s status to FREE and add it to the free list. Both GetANewProcess and FreeProcess are monitor entry methods. Task 3: The Frame Manager The lower portion of the physical memory of the BLITZ computer, starting at location zero, will contain the kernel code. It is not clear exactly how big this will be, but we will allocate 1 MByte for the kernel code. After that will come a portion of memory (called the “frame region”) which will be allocated for various purposes. For example, the disk controller may need a little memory for buffers and each of the user-level processes will need memory for “virtual pages.” The area of memory called the frame region will be viewed as a sequence of “frames”. Each frame will be the same size and we will have a fixed number of frames. For concreteness, here are some constants from Kernel.h. PAGE_SIZE = 8192 — in hex: 0x00002000 PHYSICAL_ADDRESS_OF_FIRST_PAGE_FRAME = 1048576 — in hex: 0x00100000 NUMBER_OF_PHYSICAL_PAGE_FRAMES = 512 — in hex: 0x00000200 This results in a frame region of 4 MB, so our kernel would fit into a 5 MByte memory. Project 4 Operating Systems Page 9 The frame size and the page size are the same, namely 8K. In later projects, each frame will hold a page of memory. For now, we can think of each frame as a resource that must be managed. We will not really do anything with the frames. This is similar to the dice in the gaming parlor and the forks for the philosophers… we were concerned with allocating them to threads, but didn’t really use them in any way. Each frame is a resource, like the dice of the game parlor, or the philosophers’ forks. From time to time, a thread will request some frames; the frameManager will either be able to satisfy the request, or the requesting thread will have to wait until the request can be satisfied. For the purposes of testing our code, we will work with a smaller frame region of only a few frames. This will cause more contention for resources and stress our concurrency control a little more. (For later projects, we can restore this constant to the larger value.) NUMBER_OF_PHYSICAL_PAGE_FRAMES = 27 — For testing only Here is the definition of the FrameManager class: class FrameManager superclass Object fields framesInUse: BitMap numberFreeFrames: int frameManagerLock: Mutex newFramesAvailable: Condition methods Init () Print () GetAFrame () returns int — returns addr of frame GetNewFrames (aPageTable: ptr to AddrSpace, numFramesNeeded: int) ReturnAllFrames (aPageTable: ptr to AddrSpace) endClass There will be exactly one frameManager object, created at kernel start-up time. frameManager = new FrameManager frameManager.Init () With frames (unlike the ProcessControlBlocks) there is no object to represent each resource. So to keep track of which frames are free, we will use the BitMap package. Take a look at it. Basically, the BitMap class gives us a way to deal with long strings of bits. We can do things like (1) set a bit, (2) clear a bit, and (3) test a bit. We will use a long bit string to tell which frames are in use and which are free; this is the framesInUse field. For each frame, there is a bit. If the bit is 1 (i.e., is “set”) then the frame is in use; if the bit is 0 (i.e., is “clear”) then the frame is free. The frameManager should be organized as a “Monitor class.” The frameManagerLock is used to make sure only one method at a time is executing in the FrameManager code. We have provided the code for the Init, Print, and GetAFrame methods; you’ll need to implement GetNewFrames, , and ReturnAllFrames. Project 4 Operating Systems Page 10 The method GetANewFrame allocates one frame (waiting until at least one is available) and returns the address of the frame. (Since there is never a need to return frames one at a time, there is no “ReturnOneFrame” method.) When the frames are gotten, the GetNewFrames method needs to make a note of which frames have been allocated. It does this by storing the address of each frame it allocates (the address of the first byte in each frame) into an AddrSpace object. An AddrSpace object is used to represent a virtual address space and to tell where in physical memory the virtual pages are actually located. For example, for a virtual address space with 10 pages, the AddrSpace object will contain an ordered list of 10 physical memory addresses. These are the addresses of the 10 “frames” holding the 10 pages in the virtual address space. However, the AddrSpace object contains more information. For each page, it also contains information about whether the page has been modified, whether the page is read-only or writable, etc. The information in an AddrSpace object is stored in exactly the format required by the CPU’s memory management hardware. In later projects, this will allow us to use the AddrSpace object as the current page table for a running user-level process. At that time, when we switch to a user-level process, we’ll have to tell the CPU which AddrSpace object to use for its page table. In addition to looking over the code in AddrSpace, you may want to review the BLITZ architecture manual’s discussion of page tables. The code in method GetNewFrames (aPageTable: ptr to AddrSpace, numFramesNeeded: int) needs to do the following: (1) Acquire the frame manager lock. (2) Wait on newFramesAvailable until there are enough free frames to satisfy the request. (3) Do a loop for each of the frames for i = 0 to numFramesNeeded-1 (a) determine which frame is free (find and set a bit in the framesInUse BitMap) (b) figure out the address of the free frame (c) execute the following aPageTable.SetFrameAddr (i, frameAddr) to store the address of the frame which has been allocated (4) Adjust the number of free frames (5) Set aPageTable.numberOfPages to the number of frames allocated. (6) Unlock the frame manager The code in method ReturnAllFrames (aPageTable: ptr to AddrSpace) needs to do more or less the opposite. It can look at aPageTable.numberOfPages to see how many are being returned. It can then go through the page table and see which frames it possessed. For each, it can clear the bit. Project 4 Operating Systems Page 11 for i = 0 to numFramesReturned-1 frameAddr = aPageTable.ExtractFrameAddr (i) bitNumber = …frameAddr… framesInUse.ClearBit(bitNumber ) endFor It will also need to adjust the number of free frames and “notify” any waiting threads that more frames have become available. You’ll need to do a Broadcast, because a Signal will only wake up one thread. The thread that gets awakened may not have enough free frames to complete, but other waiting threads may be able to proceed. A broadcast should be adequate, but perhaps after carefully studying the Game Parlor problem, you will find a more elegant approach which wakes up only a single thread. Also note that there is a possibility of starvation here. It is possible that one large process will be waiting for a lot of frames (e.g., 100 frames). Perhaps there are many small processes which free a few frames here and there, but there are always other small processes that grab those frames. Since there are never more than a few free frames at a time, the big process will get starved. This particular scenario for starvation, where processes are competing for frames) is a very real danger in an OS and a “real” OS would need to ensure that starvation could not happen. However, in our situation, it is acceptable to provide a solution that risks starvation. Do not modify the code for the AddrSpace class. Task 4: Change Condition Variables to Hoare Semantics The code we have given you for the Signal method in the Condition class uses MESA semantics. Change the implementation so that it uses Hoare semantics. With MESA semantics, you tend to see code like this in monitors: NewResourcesHaveBecomeAvail: Condition In method A: … numberAvail = numberAvail + 1 NewResourcesHaveBecomeAvail.Signal() … In method B: … while (numberAvail == 0) NewResourcesHaveBecomeAvail.Wait() endWhile … Project 4 Operating Systems Page 12 The code in method B contains a while loop because there is a possibility that some other thread has snuck in between the Signal and the Wait. When method A increments numberAvail, the condition (“resources are now available for some other thread to use”) has been made true. Method A invokes Signal to wake up some thread that is waiting for resources. The problem is that some other thread may have run between the Signal and the reawakening of the Wait in method B. Other threads may have come in and grabbed all the resources. So when the waiting thread is reawakened, it must always recheck for resources (or more generally, it must check to ensure the condition is still true.) Unfortunately, the condition may have changed back to false (i.e., no resources are available), so the thread will have to wait some more. And with MESA semantics, starvation becomes a bigger problem. What if the thread waits, wakes up, and then goes back to sleep, over and over. Each time a Signal occurs, some other thread just happens to get in first and steal the resource. Then the unlucky thread keeps waking up, testing, and going back into a waiting sleep. This is a very real possibility if you have three threads and they are scheduled in round-robin fashion. Thread A runs and signals thread C. Thread B runs and takes the resource. Thread C runs, finds the condition is false and goes back to sleep. Then thread A runs again, and so on. With Hoare semantics, the thread needing the resources can use code like this: … if (numberAvail == 0) NewResourcesHaveBecomeAvail.Wait() endIf … Once the thread is reawakened, it can be sure that nothing has happened between the Signal and it; it can be sure that no other thread has gotten into the monitor. Starvation is easier to ensure against, too. If a thread needs a resource, it waits. We assume the waiting queue is FIFO, so that the waiting thread will eventually be awakened. And when it wakes, it will proceed forward, without looping. Therefore, each Signal wakes one thread which proceeds and, given enough Signals to awaken any thread who went to sleep first, the thread in question will eventually get awakened. (Of course starvation-related bugs are still possible! For example, it might be that the code fails to Signal the condition enough times, leaving some thread waiting forever. The contribution of the monitor concept is to make it easier to write bug-free code, not to make it impossible to create bugs!) All further design choices are up to you. We are not providing any testing code at all; you’ll have to figure out how to test your code. If you have time, you may go back to the previous tasks to incorporate your Hoare Semantics code in their solutions, but remember to complete the above tasks before starting on the Hoare Semantics task. Project 4 Operating Systems Page 13 Do not modify… Do not modify any files except: Kernel.h Kernel.c Do not create global variables (except for testing purposes). Do not modify the methods we have provided. Policy Concerning Project Deadlines The previous projects were designed to get you up to speed and to let you know how much time the programming portion of this class will take. Some of the tasks were intentionally quite difficult, in order to challenge the ambitious students and to motivate all students to pay close attention to the subtleties of concurrency problems and their solutions. The previous projects were independent, in the sense your solution code did not need to be 100% correct in order to continue with the BLITZ kernel project. Things change with this project. Buggy code in this project is not acceptable and will make it impossible to complete the future projects. In particular, you must complete tasks 1, 2 and 3 in this project before you can begin the next project. You must get your code working and pass the test programs before going on, unlike the previous projects in which successful completion was not a barrier to moving on. However, task 4 (the Hoare Semantics task) is somewhat optional, in the sense that this code will not be needed in future projects. If you are able to complete tasks 1, 2 and 3 on time, then hand in whatever you were able to achieve on task 4 and move on to the next project. The Hoare Semantics task is provided as an extra challenge; if you don’t have time for it, then do not allow yourself to fall behind on the next project. If you are unable to complete tasks 1, 2 and 3, keep working on them until your code works 100% correctly. Students who have difficulty completing the projects on time will start to fall behind. Perhaps such a student can worker a little harder to catch up on the next project and can still get the entire kernel finished on time. Or perhaps the student will remain behind schedule and will be unable to complete one or more of the later projects. Such a student will not be able to get the completed kernel working by the end of the class. We recommend that your instructor base grades, from this project onward, on program “style” only. We recommend they simply not accept any programs that fail to work correctly, as defined by our test suite. If there are problems with the output, the student must finish / fix the code and re-submit it when it is working correctly, regardless of how long it takes. The final course grade will then be determined in part by how many of the projects the student was able to complete before the end of the course. Project 4 Operating Systems Page 14 Your instructor may alter this policy. Of course, your instructor’s policy takes precedence over what is written here. What to Hand In Please submit hardcopy of your output and hardcopy your new versions of the following files: Kernel.h Kernel.c For both files, please take a colored pen and circle exactly those parts you have added or changed. Please submit all of Kernel.h. For Kernel.c, please submit only the pages that have something circled. In other words, please discard all pages that contain ONLY material that we are distributing. Please keep the remaining pages in order. Also, if you have modified any part of a method or function, please hand in the entire function, so there is a little context surrounding your code modifications. As a second option, which will save on wasted paper, you may perform the deletions with an editor, instead of discarding physical pages. You can cut out large sections of code, but please indicate where material has been clipped out. Please replaced deleted material with a line like this XXXXXXXXXXXXXXXXXXX skipping XXXXXXXXXXXXXXXXXXX to indicate which lines were removed. But please remember to circle your material with a colored pen. In the course of testing, you will probably want to modify Main.c; but please hand in output using our version. In other words, after you are done testing, make a new run using our version of the Main package and hand in the output it produces. For the Hoare Semantics task, we are not providing any testing material; you’ll have to devise some tests on your own. Please hand in something showing how you have tested your code. Please print your output in a fixed-width font. We don’t want to see output that looks like this: 2 ProcessControlBlock pid=2, status=ACTIVE, parentsPid=0, exitStatus=0, name=”———-“, addrSpace= addr entry Logical Physical Undefined Bits Dirty Referenced Writeable Valid ========== ========== ========== ========== ============== ===== ========== ========= ===== 0x00093174: 0x00100003 0x00000000 0x00100000 YES YES Project 4 Operating Systems Page 15 We prefer to see something more like this. Even though the long lines wrap around, at least things still line up properly. 2 ProcessControlBlock pid=2, status=ACTIVE, parentsPid=0, exitStatus=0, name=”———- “, addrSpace= addr entry Logical Physical Undefined Bits Dirty Referenced Writeable Valid ========== ========== ========== ========== ============== ===== ========== ========= ===== 0x00093174: 0x00100003 0x00000000 0x00100000 YES YES In LARGE BLOCK LETTERS, write your full name on the first page. PLEASE STAPLE ALL PAGES! Sample Output Here is the output produced by our solution code. You should see something similar, if your program works correctly: ==================== KPL PROGRAM STARTING ==================== Initializing Thread Scheduler… Initializing Process Manager… Initializing Thread Manager… Initializing Frame Manager… ***** THREAD-MANAGER TEST ***** 123.4..1562.7839.1041112.13141.15….16….3295.17.8184..76.192010.111315…14..12..161..9.173188..7.19. 52.20….14410.12.91113…616…15571..18320….122..17..10..9161114586157…….4.123.1.18217.13..20… 9.19.516.1411.2..8..110.15.64.12.1618.9.7.132019..11.3.15..14.812..61.9…2.2018..10165.15.78..9…4.112 013.3..191815….714…17.11.151216.194…18.9.102076..11.12..15…14.84191013..93..2…1511518..8.714.. 9.616.19..1..2015…813.42.1210.9……520.4.2.181116.14915..4.2.1.1117..5.8.1018.15.2.4.14.9.11..1.56.8 .2.15..4.1410..93..18.2013.2.15..7.17…1210914.8.20.2…..17515123.14.9.20..16.819.12.5.3.15.1.16.14.20 .8.3..513.12.15.18..1.8..209.193.5.6.18.7.8….20.14153.511.12.4.7.8.3.14.15.9.11.12…1518.16.3..14..11 17.12..4.5..1710..14.26.1.7..123.16.8.19.17.6.20.2.10..12.165.3.19.1.17..4…6.8.1220216.19..7.13.13.6.1 1.19.16..20.4.2.1.7..13.6.11.17.19..16.14.7..1….18….1013619.16.1711.4.7.20..13.18.16.19.10.6.17..411 .7..13.18.19…17.10.6..7.13..1918.17.10.6…..131817.19.106.13..17…10.13.17..13.. ***** THREAD-MANAGER TEST COMPLETED SUCCESSFULLY ***** ***** PROCESS-MANAGER TEST ***** 123.4..1562783.9.104…1…11.738.12132149.156.5.1016.17..184…19..201.11138.714.12.3….5.26.17164.3.2 0.19..1812..629.14..1015…17.137..16.84.1820.1912…116.15.14…..572.3.89.116….10417513..15.12.1814. ..23..19.716.1204.1511…14..5..9.6131237..162….18.10.1718..5..1119.61620….9184..13.17..14108….612 711.159..2..11317….20.11..16..7146.119515.10.4.168…714..11.92.12.1..181720..14..6319…1219..7.168.. .2513…411.15..129.181.14..19.32.4..6.15.128.16..19.1410.5.20.9.7..46.16.11.3.13.17.14.9.8.7.1.16.2..5. .192010.9.7..164.1.11…31317.5.9.7.6.12…19.13.11.20161.5.7.12.19.3.6..1110..113.5.12..719.11.6.2.3.20 .16.12.5.19.11.6..207.2..9.1416.5.19.15.6.11.20.2.14.9.12.4.5.17..1118.20.2.14.9…12.5154.11.20.6..152. .18.16.14.10.5.20..63.17..9.102.1.14.11.18.13.10…739.2.16.20.14.11.10.8.3..76.16..20.13…8.3…18…. 12194.10.138.3…918..20.1310..124.19.3.1.15.17.13..4.14.10.12…20.8..1913.4.17.10..18.8…13.19.17.4.1 8.15.8.10.17.13.19.18..15.8.18.17.10…15.8.18.17.15…18.17.15.18.17.15.18..15..15. ***** PROCESS-MANAGER TEST COMPLETED SUCCESSFULLY ***** Project 4 Operating Systems Page 16 ***** FRAME-MANAGER TEST ***** 1234.57.68.9110…32.5..74…8.69..101.32.45.7…..6.9.8.310215.4……10967138…2…5107.946…1.25..3 8.9..10.64..79.21..5..710.8.1.3.9.2..684.19..5..6310..48..9..65.1.7.53..2.8..93.10..12.47..1..4.1086..11 0..48.6..35..9.710..9.85.3..4.78.6..310.4.7.3..58.10.6..94.1..37.8..6.92.5..74..12.4.2.3.1..5.26.10.3.1. 7.5.8.10.3.1.9.6.8.10.3.1.4.6.10.8.3..31…134…513..1.83…3101..43..81..8..93.8.4..1.8.34..17.8.3.2.1 .4.8.6.4.2.8.47…83.1.5.34..1.84.3.1..84..28..10..4107…4110..5.410..18.3.1.8.3..41..105..8..4510…71 05..610..75..71…1085..7..410.75…175..9.75.10..58.7..4.52.7.6..7.6.5.4.10.85..10.6.7.47..5..106.7.510 .7..56..26..69..2..652..28.6.1.2..36..52..36..7.29…912..97..16…10.129..76…139..7.91..24..79..101.. 82.6..91.2.6.9..71..93..61.10.9.1..62.8…298..810..48.1..6.810..1.210..8.510.8..110..9.8.10..2810..98.. 710..79.8..7.210.6…27.109.8…1017.2.10.7.8.10.7..47.10.8.1.10..48.7..4.17..84.5.4.1.7..8.45.4..58.4.. 7.5.104..56…56.4..765..10.5.67.4..9.5.6.7.5.6.45.8.6.4.7.6.1.5.9.6..26..105.4.6.10.2..23..3.1.23..6.5. 23…623.10.2.6..54.2..5.3.26.3.7.2.3.1.3.1.9.2…613..41.10…1410..10.6.1.104.1..9.10..219….31019.7. .10.93..10.1.910..32.1.3..109.2.5.1.2.3.10..97.1.10.5.9.4.9.1.2.10.3.9.2.9..110..93.9.1.7..71.7..27.5.7. 4.7..42..7.410.7..48…18.75..87.10..4.8.2.8.46.7…984.5.8..9.5..74.58..10.54..7.8.5.47.8..65.4.7..5.84 .5.1.5.4.9.5.3.5.8..95.6.6..10.65.8.7.6..26…926..72..6.2.54.2.3.6..102..56.10.2…7610…8610.1..2..11 0.4..1210.6..5.101.9..21.6..110.2..410..81.4.10..71..410.8.2.6.1.7..76.10.5.1..7.10.2.71..106.1..102.7.1 0.1.2.7..74.7..57.9..95.9..39..5..931..57..3.6..39.510.7..2.95.8.7…1039.2.9.8.9..2.5.310.5.2.5.7.9.1.. 73.5..54..5.1.43.10.4.2…3.25.4.9..34.25.9..34.5..79..2.3.4.21.4.7.3.5..24.9..3.52.3.4.5.2..82.6.2.5.4. 7.2..94..3.2.9.6.1.6..69.6..56.7.6.4.2..46.9..46.9.6..3.69..49.6..92.6..39.5.9.3.9..9..79.8.8.1.9..86.2. .89.6..29.6..89.8.2.8..96.8..8.6.8..98.4.8.9.8.4.8.6.8.4.8.6..3..3.2.3..93.8.3..63..83.9.3..3.2.3.8.9… 3..3.3.3.3.3.3.3.3. Here is a histogram showing how many times each frame was used: 0: ******************************************************************************************************** ******************************************************************************************************** ******************************************************************************************************** ******************************************************************************************************** ********** 1: ******************************************************************************************************** ******************************************************************************************************** ******************************************************************************************************** ******************************************************************************************************** ********** 2: ******************************************************************************************************** ******************************************************************************************************** ******************************************************************************************************** ******************************************************************************************************** ********** XXXXXXXXXXXXXXXXXXX skipping XXXXXXXXXXXXXXXXXXX 24: ******************************************************************************************************** ******************************************************************************************************** * 25: ******************************************************************************************************** *************************************************************************** 26: ******************************************************************************************************** ***** ***** FRAME-MANAGER TEST COMPLETED SUCCESSFULLY ***** ==================== KPL PROGRAM TERMINATION ==================== Project 4 Operating Systems Page 17 Coding Style For all projects, please follow our coding style. Make your code match our code as closely as possible. The goal is for it to be impossible to tell, just by looking at the indentation and commenting, whether we wrote the code or you wrote the code. (Of course, your code will be circled!) Please use our convention in labeling and commenting functions: —————————– Foo ——————————— function Foo () — — This routine does…. — var x: int x = 1 x = 2 … x = 9 endFunction Methods are labeled like this, with both class and method name: ———- Condition . Wait ———- method Wait (mutex: ptr to Mutex) … Note that indentation is in increments of 2 spaces. Please be careful to indent statements such as if, while, and for properly.Overview and Goal In this project, you will explore user-level processes. You will create a single process, running in its own address space. When this user-level process executes, the CPU will be in “user mode.” The user-level process will make system calls to the kernel, which will cause the CPU to switch into “system mode.” Upon completion, the CPU will switch back to user mode before resuming execution of the user-level process. The user-level process will execute in its own “logical address space.” Its address space will be broken into a number of “pages” and each page will be stored in a frame in memory. The pages will be resident (i.e., stored in frames in physical memory) at all times and will not be swapped out to disk in this project. (Contrast this with “virtual” memory, in which some pages may not be resident in memory.) The kernel will be entirely protected from the user-level program; nothing the user-level program does can crash the kernel. Download New Files The files for this project are available in: https://www.cs.pdx.edu/~harry/Blitz/OSProject/p5/ Please retain your old files from previous projects and don’t modify them once you submit them. You should get the following files: Switch.s Runtime.s System.h System.c Project 5 Operating Systems Page 2 List.h List.c BitMap.h BitMap.c makefile FileStuff.h FileStuff.c Main.h Main.c DISK UserRuntime.s UserSystem.h UserSystem.c MyProgram.h MyProgram.c TestProgram1.h TestProgram1.c TestProgram2.h TestProgram2.c The following files are unchanged from the last project and you should not modify them: Switch.s Runtime.s System.h System.c — except HEAP_SIZE has been modified List.h List.c BitMap.h BitMap.c The following files are not provided; instead you will modify what you created in the last project. Copy these files to your p5 directory, so that you keep the previous p4 versions in your p4 directory, and modify the new copies. Kernel.h Kernel.c Merging New “File Stuff” Code For this project, we are distributing additional code which you should add to the Kernel package. Please add the material in FileStuff.c to the end of file Kernel.c. It should be inserted directly before the final endCode keyword. Project 5 Operating Systems Page 3 Also, please add the material in FileStuff.h to the end of file Kernel.h. It should be inserted directly before the final endHeader keyword. This code adds the following classes: DiskDriver FileManager FileControlBlock OpenFile You will use these classes, but you should not modify them. There will be a single DiskDriver object (called diskDriver) which is created and initialized at start-up time. There will be a single FileManager object (called fileManager) which is created and initialized at start-up time. The new main function contains statements to create and initialize the diskDriver and the fileManager objects. FileControlBlock and OpenFile objects will be handled much like Threads and ProcessControlBlocks. They are a limited resource. A limited supply is created at start-up time and then they are managed by the fileManager. There is a free list of FileControlBlock objects and a free list of OpenFile objects. The fileManager oversees both of these free lists. Threads may make requests and may return resources, by invoking methods in the fileManager. The diskDriver object encapsulates all the hardware specific details of the disk. It provides a method that allows a thread to read a sector from disk into a memory frame and it provides a method that writes a frame from memory to a sector on disk. Other Changes To Your Kernel Code Please make the following changes to your copy of Kernel.h: Change NUMBER_OF_PHYSICAL_PAGE_FRAMES = 27 — for testing only to: NUMBER_OF_PHYSICAL_PAGE_FRAMES = 100 — for testing only Change –diskDriver: DiskDriver –fileManager: FileManager to: diskDriver: DiskDriver fileManager: FileManager Project 5 Operating Systems Page 4 Add a function prototype for the function InitFirstProcess. You can add it after the other function prototypes: Change ProcessFinish (exitStatus: int) to: ProcessFinish (exitStatus: int) InitFirstProcess () Please make the following changes to your copy of Kernel.c: Change the DiskInterruptHandler function from: FatalError (“DISK INTERRUPTS NOT EXPECTED IN PROJECT 4”) to: currentInterruptStatus = DISABLED — print (“DiskInterruptHandler invoked! ”) if diskDriver.semToSignalOnCompletion diskDriver.semToSignalOnCompletion.Up() endIf Task 1: Your first task is to load and execute the user-level program called MyProgram. Since the user-level program must be read from a file on the BLITZ disk, you’ll first need to understand how the BLITZ disk works, how files are stored on the disk, and how the FileManager code works. MyProgram invokes the SystemShutdown syscall, which you’ll need to implement. Task 2: Modify all the syscall handlers so they print the arguments that are passed to them. In the case of integer arguments, this should be straightforward, but the following syscalls take a pointer to an array of char as one of their arguments. Exec Create Open This pointer is in the user-program’s logical address space. You must first move the string from userspace to a buffer in kernel space. Only then can it be safely printed. Project 5 Operating Systems Page 5 Also, some of the syscalls return a result. You must modify the handlers for these syscalls so that the following syscalls return these values. (These are just arbitrary values, to make sure you can return something.) Fork 1000 Join 2000 Exec 3000 Create 4000 Open 5000 Read 6000 Write 7000 Seek 8000 For this task, you should modify only the handler methods (e.g., Handle_Sys_Fork, Handle_Sys_Join, etc.) You should not modify SyscallTrapHandler or the wrapper functions in UserSystem. Task 3: Implement the Exec syscall. The Exec syscall will read a new executable program from disk and copy it into the address space of the process which invoked the Exec. It will then begin execution of the new program. Unless there are errors, there will not be a return from the Exec syscall. The User-Level View First, let’s look at our operating system from the users’ point of view. User-level programs will be able to invoke the following kernel routines: Exit Shutdown Yield Fork Join Exec Create Open Read Write Seek Close (This is the grand plan for our OS; these system calls will not be implemented in this project.) Project 5 Operating Systems Page 6 These syscalls are quite similar to kernel syscalls of the same names in Unix. We describe their precise functionality later. A user-level program will be written in KPL and linked with the following files: UserSystem.h UserSystem.c UserRuntime.s We are providing a sample user-level program in MyProgram.h / .c. The UserSystem package includes a wrapper (or “jacket”) function for each of the system calls. Here are the names of the wrapper functions. There is a one-to-one correspondence between the system calls and the wrapper functions. System call Wrapper function name Exit Sys_Exit Shutdown Sys_Shutdown Yield Sys_Yield Fork Sys_Fork Join Sys_Join Exec Sys_Exec Create Sys_Create Open Sys_Open Read Sys_Read Write Sys_Write Seek Sys_Seek Close Sys_Close (In Unix, the wrapper function often has the same name as the syscall. All wrapper functions have names beginning with Sys_ just to help make the distinction between wrapper and syscall.) Each wrapper function works the same way. It invokes an assembly language routine called DoSyscall, which executes a “syscall” machine instruction. When the kernel call finishes, the DoSyscall function simply returns to the wrapper function, which returns to the user’s code. Arguments may be passed to and from the kernel call. In general, these are integers and pointers to memory. The wrapper function works with DoSyscall to pass the arguments. When the wrapper function calls DoSyscall, it will push the arguments onto the stack. The DoSyscall will take the arguments off the stack and move them into registers. Since it runs as a user-level function, it places them in the “user” registers. (Recall that the BLITZ machine has a set of 16 “system registers” and a set of 16 “user registers.”) Each wrapper function also uses an integer code to indicate which kernel function is involved. Here is the enum giving the different codes. For example, the code for “Fork” is 4. Project 5 Operating Systems Page 7 enum SYSCALL_EXIT = 1, SYSCALL_SHUTDOWN, SYSCALL_YIELD, SYSCALL_FORK, SYSCALL_JOIN, SYSCALL_EXEC, SYSCALL_CREATE, SYSCALL_OPEN, SYSCALL_READ, SYSCALL_WRITE, SYSCALL_SEEK, SYSCALL_CLOSE These code numbers are used both by the user-level program and by the kernel. Consequently, there is an identical copy of this enum in both Kernel.h and UserSystem.h. (You should not change the system call interface, but if one were to change these code numbers, it would be critical that both enums were changed identically.) As an example, here is the code for the wrapper function for “Read.” It simply invokes DoSyscall and returns whatever DoSyscall returns. function Sys_Read (fileDesc: int, buffer: ptr to char, sizeInBytes: int) returns int return DoSyscall (SYSCALL_READ, fileDesc, buffer asInteger, sizeInBytes, 0) endFunction Here is the function prototype for DoSyscall: external DoSyscall (funCode, arg1, arg2, arg3, arg4: int) returns int The DoSyscall routine is set up to deal with up to 4 arguments. Since the Read syscall only needs 3 arguments, the wrapper function must supply an extra zero for the fourth argument. DoSyscall treats all of its arguments as untyped words (i.e., as int), so the wrapper functions must coerce the types of the arguments if they are not int. Whatever DoSyscall returns, the wrapper function will return. DoSyscall is in UserRuntime.s, which will be linked with all user programs. The code is given next. It moves each of the 4 arguments into registers r1, r2, r3, and r4. It then moves the function code into register r5 and executes the syscall instruction. It assumes the kernel will place the result (if any) in r1, so after the syscall instruction, it moves the return value from r1 to the stack, so that the wrapper function can retrieve it. Project 5 Operating Systems Page 8 DoSyscall: load [r15+8],r1 ! Move arg1 into r1 load [r15+12],r2 ! Move arg2 into r2 load [r15+16],r3 ! Move arg3 into r3 load [r15+20],r4 ! Move arg4 into r4 load [r15+4],r5 ! Move funcCode into r5 syscall r5 ! Do the syscall store r1,[r15+4] ! Move result from r1 onto stack ret ! Return Some of the kernel routines require no arguments and/or return no result. As an example, consider the wrapper function for Yield. The compiler knows that DoSyscall returns a result, so it insists that we do something with this value. The wrapper function simply moves it into a variable and ignores it. function Sys_Yield () var ignore: int ignore = DoSyscall (SYSCALL_YIELD, 0, 0, 0, 0) endFunction Here is a list of all the wrapper functions, including their arguments and return types. Sys_Exit (returnStatus: int) Sys_Shutdown () Sys_Yield () Sys_Fork () returns int Sys_Join (processID: int) returns int Sys_Exec (filename: String) returns int Sys_Create (filename: String) returns int Sys_Open (filename: String) returns int Sys_Read (fileDesc: int, buffer: ptr to char, sizeInBytes: int) returns int Sys_Write (fileDesc: int, buffer: ptr to char, sizeInBytes: int) returns int Sys_Seek (fileDesc: int, newCurrentPos: int) returns int Sys_Close (fileDesc: int) In addition to the wrapper functions, the UserSystem package contains a few other routines that support the KPL language. These are more-or-less duplicates of the same routines in the System package. Likewise, some of the material from Runtime.s is duplicated in UserRuntime.s. This duplication is necessary because user-level programs cannot invoke any of the routines that are part of the kernel. For example the functions print, printInt, nl, etc. have been duplicated at the user level so the userlevel program has the ability to print. [Note that, at this point, all printing is done by cheating, using a “trapdoor” in the emulator. Normally, a user-level program would need to invoke syscalls (such as Sys_Write) to perform any output, since user-level programs can’t access the I/O devices directly. However, since we are not yet ready to address questions about output to the serial device, we are including these cheater print functions, which rely on a trapdoor in the emulator.] Every user-level program needs to “use” the UserSystem package and be linked with the UserRuntime.s code. For example: Project 5 Operating Systems Page 9 MyProgram.h header MyProgram uses UserSystem functions main () endHeader MyProgram.c code MyProgram function main () print (“My user-level program is running! ”) Sys_Shutdown () endFunction endCode Here are the commands to prepare a user-level program for execution. The makefile has been modified to include these commands. asm UserRuntime.s comp UserSystem -unsafe asm UserSystem.s comp MyProgram -unsafe asm MyProgram.s lddd UserRuntime.o UserSystem.o MyProgram.o -o MyProgram Note that there is no connection with the kernel. The user-level programs are compiled and linked independently. All communication with the kernel will be through the syscall interface, via the wrapper functions. This is exactly the way Unix works. For user-level programs, library functions and wrapper functions are brought into the “a.out” file at link-time, as needed. This explains why a seemingly small “C” program can produce a rather large “a.out” executable. One small use of printf in a program might pull in, at link-time, more output formatting and buffering routines than you can possibly imagine. When an OS wants to execute a user-level program, it will go to a disk file to find the executable. Then it will read that executable into memory and start up the new process. In order to execute MyProgram, we need to introduce the BLITZ “disk.” The disk is simulated with a Unix file called “DISK.” After the user-level program is compiled, it must be placed on the BLITZ disk with the following Unix commands: diskUtil -i diskUtil -a MyProgram MyProgram The first command creates an empty file system on the disk. The second command copies a file from the Unix file system to the BLITZ disk. It creates a directory entry and moves the data to the proper place on the simulated BLITZ disk. Commands to initialize the BLITZ disk have also been added to the makefile. Project 5 Operating Systems Page 10 Once the kernel is running, it will read the file from the simulated BLITZ disk and copy it into memory. The Syscall Interface In our OS, each process will have exactly one thread. A process may also have several open files and can do I/O via the Read and Write syscalls. The I/O will go to the BLITZ disk. For now, there is no serial (i.e., terminal) device. Next we describe each syscall in more detail. function Sys_Exit (returnStatus: int) This function causes the current process and its thread to terminate. The returnStatus will be saved so that it can be passed to a Sys_Join executed by the parent process. This function never returns. function Sys_Shutdown () This function will cause an immediate shutdown of the kernel. It will not return. function Sys_Yield () This function yields the CPU to another process on the ready list. Once this process is scheduled again, this function will return. From the caller’s perspective, this routine is similar to a “nop.” function Sys_Fork () returns int This function creates a new process which is a copy of the current process. The new process will have a copy of the virtual memory space and all files open in the original process will also be open in the new process. Both processes will then return from this function. In the parent process, the pid of the child will be returned; in the child, zero will be returned. function Sys_Join (processID: int) returns int This function causes the caller to wait until the process with the given pid has terminated, by executing a call to Sys_Exit. The returnStatus passed by that process to Sys_Exit will be returned from this function. If the other process invokes Sys_Exit first, this returnStatus will be saved until either its parent executes a Sys_Join naming that process’s pid or until its parent terminates. function Sys_Exec (filename: String) returns int This function is passed the name of a file. That file is assumed to be an executable file. It is read in to memory, overwriting the entire address space of the current process. Then the OS will Project 5 Operating Systems Page 11 begin executing the new process. Any open files in the current process will remain open and unchanged in the new process. Normally, this function will not return. If there are problems, this function will return -1. function Sys_Create (filename: String) returns int This function creates a new file on the disk. If all is okay, it returns 0, otherwise it returns a nonzero error code. This function does not open the file; so the caller must use Sys_Open before attempting any I/O. function Sys_Open (filename: String) returns int This function opens a file. The file must exist already exist. If all is OK, this function returns a file descriptor, which is a small, non-negative integer. It errors occur, this function returns -1. function Sys_Read (fileDesc: int, buffer: ptr to char, sizeInBytes: int) returns int This function is passed the fileDescriptor of a file (which is assumed to have been successfully opened), a pointer to an area of memory, and a count of the number of bytes to transfer. This function reads that many bytes from the current position in the file and places them in memory. If there are not enough bytes between the current position and the end of the file, then a lesser number of bytes are transferred. The current file position will be advanced by the number of bytes transferred. If the input is coming from the serial device (the terminal), this function will wait for at least one character to be typed before returning, and then will return as many characters as have been typed and buffered since the previous call to this function. This function will return the number of characters moved. If there are errors, it will return -1. function Sys_Write (fileDesc: int, buffer: ptr to char, sizeInBytes: int) returns int This function is passed the fileDescriptor of a file (which is assumed to have been successfully opened), a pointer to an area of memory, and a count of the number of bytes to transfer. This function writes that many bytes from the memory to the current position in the file. If the end of the file is reached, the file’s size will be increased. The current file position will be advanced by the number of bytes transferred, so that future writes will follow the data transferred in this invocation. The output may also be directed to the serial output, i.e., to the terminal. This function will return the number of characters moved. If there are errors, it will return -1. Project 5 Operating Systems Page 12 function Sys_Seek (fileDesc: int, newCurrentPosition: int) returns int This function is passed the fileDescriptor of a file (which is assumed to have been successfully opened), and a new current position. This function sets the current position in the file to the given value and returns the new current position. Setting the current position to zero causes the next read or write to refer to the very first byte in the file. If the file size is N bytes, setting the position to N will cause the next write to append data to the end of the file. The current position is always between 0 and N, where N is the file’s size in bytes. If -1 is supplied as the new current position, the current position will be set to N (the file size in bytes) and N will be returned. It is an error to supply a newCurrentPosition that is less than -1 or greater than N. If so, -1 will be returned. function Sys_Close (fileDesc: int) This function is passed the fileDescriptor of a file, which is assumed to be open. It closes the file, which includes writing out any data buffered by the kernel. Asynchronous Interrupts From time-to-time an asynchronous interrupt will occur. Consider a DiskInterrupt as an example. When this happens, an assembly routine called DiskInterruptHandler in Runtime.s will be jumped to. It begins by saving the system registers (after all, a Disk Interrupt might occur while a kernel routine is executing and we’ll need to return to it). Then DiskInterruptHandler performs an “upcall” to the function named DiskInterruptHandler in Kernel.c. Perhaps it is a little confusing to have an assembly routine and a KPL routine with the same name, but, oh well… The high-level DiskInterruptHandler routine simply signals a semaphore and returns to the assembly DiskInterruptHandler routine, which restores the system registers and returns to whatever code was interrupted. All the time while these routines are running, interrupts are disabled and no other interrupts can occur. Also note that the interrupt handler uses space on the system stack of whichever thread was interrupted. It might be that some unsuspecting user-level code was running. Although the interrupt handler will use the system stack of that thread, the thread will be none-the-wiser. While the interrupt handler is running, it is running as part of some more-or-less randomly selected thread. The interrupt handler is not a thread on its own. Project 5 Operating Systems Page 13 Error Exception Handling When a runtime error is detected by the CPU, the CPU performs exception processing, which is similar to the way it processes an interrupt. Here are the sorts of runtime errors that can occur in the BLITZ architecture: Illegal Instruction Arithmetic Exception Address Exception Page Invalid Exception Page Read-only Exception Privileged Instruction Alignment Exception As an example, consider what happens when an Alignment Exception occurs. (The others are handled the same way.) The CPU will consult the interrupt vector in low memory (see Runtime.s) and will jump to an assembly language routine called AlignmentExceptionHandler. The assembly routine first checks to see if the interrupted code was executing in system mode or not. If it was in system mode, then the assumption is that there is a bug in the kernel, so the assembly routine prints a message and halts execution. However, if the CPU was in user mode, the assumption is that the user-level program has a bug. The OS will need to handle that bug without itself stopping. So the assembly AlignmentExceptionHandler routine makes an upcall to a KPL routine with the same name. The high-level AlignmentExceptionHandler routine simply prints a message and terminates the process. Process termination is performed in a routine called ProcessFinish, which is not yet written. (For now, we’ll assume that user-level programs do not have any bugs.) When ProcessFinish is implemented in a later project, it will need to return the ProcessControlBlock (PCB) to the free pool. It will also need to free any additional resources held by the process, such as OpenFile objects. Of course, any open files will need to be closed first. Finally, ProcessFinish will call ThreadFinish and will not return. (Note that a Thread object cannot be added back to the free thread pool by the thread that is running. Instead, in ThreadFinish the thread is added to a list called threadsTobeDestroyed. Later, after another thread begins executing (in Run) the first thing it will do is add any threads on that list back to the free pool by calling threadManager.FreeThread.) Project 5 Operating Systems Page 14 Syscalls When a user-level thread executes a syscall instruction, the assembly routine SyscallTrapHandler in Runtime.s will be invoked. The assembly routine will then call a KPL routine with the same name. The assembly routine does not need to save registers because the interrupted code was executing in user mode and the handler will be executed in system mode. Recall that just before the syscall, the DoSyscall routine placed the arguments in the (user) registers r1, r2, r3, and r4, with an integer indicating which kernel function is wanted in register r5. The SyscallTrapHandler assembly routine takes the values from the user registers. Since it is running in system mode, it must use a special instruction called “readu” to get values from the user registers. It pushes them on to the system stack so that the high-level routine can access them. Then it calls the high-level SyscallTrapHandler routine. When the high-level routine returns, it takes the returned value from the stack and moves it into user register r1, using an instruction called “writeu,” and then executes a “reti” instruction to return to the interrupted user-level process. Execution will resume back in DoSyscall directly after the “syscall” instruction. The high-level routine called SyscallTrapHandler simply takes a look at the function code and calls the appropriate routine to finish the work. For every kind of syscall, there is a corresponding “handler routine” in the OS. System call Handler function in the kernel Exit Handle_Sys_Exit Shutdown Handle_Sys_Shutdown Yield Handle_Sys_Yield Fork Handle_Sys_Fork Join Handle_Sys_Join Exec Handle_Sys_Exec Create Handle_Sys_Create Open Handle_Sys_Open Read Handle_Sys_Read Write Handle_Sys_Write Seek Handle_Sys_Seek Close Handle_Sys_Close It is these routines that you will need to implement, in this and other projects. Note that interrupts will be disabled when the SyscallTrapHandler routine begins. The first thing the high-level routine does is set the global variable currentInterruptStatus to DISABLED so that it is accurate. In fact, all the interrupt and exception handlers begin by setting currentInterruptStatus to DISABLED for this reason. Also note that after the handler routines return to the interrupted routine, interrupts will be re-enabled. Why? Because the Status Register in the CPU will be restored as part of the operation of the reti instruction, restoring the interrupt (and paging and system mode) status bits to what they were when the Project 5 Operating Systems Page 15 interrupt occurred. (Note that we do not bother to change currentInterruptStatus to ENABLED before returning to user-level code, because any re-entry to the kernel code must be through SyscallTrapHandler, or an interrupt or exception handler, and each of these begins by setting currentInterruptStatus.) Implementing the Shutdown syscall is straightforward. The handler should call FatalError with the following message: Syscall ‘Shutdown’ was invoked by a user thread The BLITZ Disk The BLITZ computer includes a disk which is emulated using a file called DISK on the host computer. In other words, a write to the BLITZ disk will cause data to be written to a Unix file and a read from the BLITZ disk will cause a read from the Unix file. The emulator will simulate the delays involved in reading, by taking account of the current (simulated) disk head position. When the I/O is complete— that is the simulated time when the emulator has calculated the disk I/O will have completed—the emulator causes a DiskInterrupt to occur. To interface with the BLITZ disk, we have supplied a class called DiskDriver, which makes it unnecessary for you to write the code that actually reads and writes disk sectors. You can just use the code in the class DiskDriver. There is only one DiskDriver object; it is created and initialized at startup time. class DiskDriver superclass Object fields … semToSignalOnCompletion: ptr to Semaphore semUsedInSynchMethods: Semaphore diskBusy: Mutex methods Init () SynchReadSector (sectorAddr, numberOfSectors, memoryAddr: int) StartReadSector (sectorAddr, numberOfSectors, memoryAddr: int, whoCares: ptr to Semaphore) SynchWriteSector (sectorAddr, numberOfSectors, memoryAddr: int) StartWriteSector (sectorAddr, numberOfSectors, memoryAddr: int, whoCares: ptr to Semaphore) endClass This class provides a way to read and write sectors synchronously as well as a way to read and write sectors asynchronously. To perform a disk operation without blocking the calling thread, you can call StartReadSector or StartWriteSector. These methods are passed the number of the sector on the disk at which to begin the transfer, the number of sectors to transfer and the location in memory to transfer the data to or from. Project 5 Operating Systems Page 16 These methods are also passed a pointer to a Semaphore; upon completion of the operation (possibly in error!) this semaphore will be signaled with an Up() operation. This is exactly the semaphore that is signaled whenever a DiskInterrupt occurs. So to perform asynchronous I/O, the caller will invoke StartReadSector (or StartWriteSector) giving it a Semaphore. Then the caller can either do other stuff, or wait on the Semaphore. Since it may be a little tricky to manage asynchronous I/O correctly, the DiskDriver class also provides a couple of methods to make it easy to do I/O synchronously. When you call SynchReadSector or SynchWriteSector, the caller will be suspended and will be returned to only after a successful completion of the I/O. These routines will deal with transient errors by retrying the operation until it works. Other errors (such as a bad sectorAddr or bad memoryAddr) will be dealt with by a call to FatalError. In order to implement these methods, the DiskDriver contains a mutex called diskBusy and a semaphore called SemUsedInSynchMethods. Each synch method makes sure the disk is not busy with I/O from some other thread and, if so, waits until it is completed. This is the purpose of the diskBusy mutex. After acquiring the lock, each synch method will call StartReadSector (or StartWriteSector) supplying the semaphore. The synch method will then wait until the disk operation is complete. The calling thread will remain blocked for the duration. The “Stub” File System As a later project, you might want to implement a full file system, more like the one used by Unix systems. For now, you are supplied with a very minimal file system, called the “stub” file system. In Unix, directories are structured in a tree shape and there are lots of complexities concerning how files are stored on the disk. In the stub file system, the disk will contain only one directory, and several files. The directory is limited in size to one sector and is kept in sector 0 of the disk. The exact number of files that can be accommodated depends on how long the file names are. Each file has a name and a file length (in bytes). Each file is stored on disk in a sequence of consecutive sectors. Once a file is placed on the disk, and more files are added after it, it is impossible to increase the size of the file. Each file is allocated an integral number of sectors. (Since the last sector in each file may be only partially full, it would be possible to increase the size of a file up to the next sector boundary. However, it is not worth the effort. Instead, the solution is to design a better file system!) For now, the directory is read-only, so files may not be created and the size of files may not be changed. The classes FileManager, FileControlBlock, and OpenFile are provided for you, to make it easier to use the file system from within the kernel. Project 5 Operating Systems Page 17 The “diskUtil” Tool The BLITZ tool called diskUtil can be used to create a file system on the BLITZ disk, to add files to the disk, to remove files, and to print out the directory. The BLITZ DISK is organized as follows. The disk contains a single directory and this is kept in sector 0. The files are placed sequentially on the disk, one after the other. Each file will take up an integral number of sectors. Each file has an entry in the directory. Each entry contains (1) The starting sector (2) The file length, in bytes (possibly zero) (3) The number of characters in the file name (4) The file name The directory begins with three numbers: (1) Magic Number (0x73747562 = “stub”) (2) Number of files (possibly zero) (3) Number of the next free sector These are followed by the entries for each file. Once created, a BLITZ file may not have its size increased. When a file is removed, the free sectors become unusable; there is no compaction or any attempt to reclaim the lost space. Each time the diskUtil program is run, it performs one of the following functions: Initialize set up a new file system on the BLITZ disk List list the directory on the BLITZ disk Create create a new file of a given size Remove remove a file Add copy a file from Unix to BLITZ Extract copy a file from BLITZ to Unix Write write sectors from a Unix file to the BLITZ disk The following command line options tell which function is to be performed by diskUtil: -h Print help info. -d DiskFileName The file used to emulate the BLITZ disk. If missing, “DISK” will be used. Project 5 Operating Systems Page 18 -i Initialize the file system on the BLITZ “DISK” file. This will effectively remove all files on the BLITZ disk and reclaim all available space. -l List the directory on the BLITZ disk. -c BlitzFileName SizeInBytes Create a file of the given size on the BLITZ disk. The BLITZ disk must not already contain a file with this name. Only the directory will be modified; the actual data in the file will be whatever bytes happened to be on the disk already. -r BlitzFileName Remove the file with the given name from the directory on the BLITZ disk. -a UnixFilename BlitzFileName Copy a file from Unix to the BLITZ disk. If BlitzFileName already exists, it must be large enough to accommodate the new data. -e BlitzFileName UnixFileName Extract a file from the BLITZ disk to Unix. This command will copy the data from the BLITZ disk to a Unix file. The Unix file may or may not already exist; its size will be shortened or lengthened as necessary. -w UnixFileName SectorNumber The UnixFileName must be an existing Unix file. The SectorNumber is an integer. The Unix file data will be written to the BLITZ disk, starting at sector SectorNumber. The directory will not be modified. We are providing a DISK file which should be large enough, but if you want, you may create a new BLITZ disk file of a different size. The new disk file must also be initialized properly; it can be created and initialized with the format command in the BLITZ emulator. For example: % blitz … > format … The name of the disk file is “DISK”. The file “DISK” did not previously exist. (It could not be opened for reading.) Enter the number of tracks (e.g., 1000; type 0 to abort): 3 … Initializing sectors 0 through 47… Successful completion. Project 5 Operating Systems Page 19 Next, we use diskUtil to create a file system, add several files, and print the directory, by typing these commands at the Unix prompt: % diskUtil –i % diskUtil -a temp1 MyFileA % diskUtil -a temp2 MyFileB % diskUtil -a MyProgram MyProgram % diskUtil –l StartingSector SizeInSectors SizeInBytes FileName ============== ============= =========== ===================== 0 1 8192 < directory > 1 1 8192 MyFileA 2 3 17000 MyFileB 5 8 60264 MyProgram The FileManager There is only one FileManager object; it is created and initialized at startup time. We are supplying several methods to help you access files on the “stub” file system; these methods are located in this class. You’ll need to know how to access files in order to create the first user-level process. You’ll need to open the executable file, read the bytes from disk, then close the file. You’ll also need to use the fileManager when you implement the Exec syscall. Some of the following material pertains more to the next project than this project. Read it all now to get familiar with the framework. You may want to review it again during the next project. Associated with the FileManager class, there are two other classes called FileControlBlock and OpenFile. These two classes contain fields, but do not contain many methods of their own (besides Init() and Print() methods). Instead, most of the work associated with the file system is done by the FileManager methods. The FileControlBlock (FCB) objects and the OpenFile objects are limited resources. The FileManager maintains a free list for each of these, as well as code to allocate new FCB objects and new OpenFile objects and maintain the free lists. The FileManager also deals with opening files. This involves finding the file in the file system, that is, determining the file’s location on disk. In the “stub” file system this is pretty simple since there is only one directory and it fits into a single sector. The FileManager—as programmed now—reads the directory sector (sector 0) into a frame as part of the FileManager.Init method. Subsequent attempts to open a file require no disk accesses. (Of course, for a “real” file system, things won’t be so simple!) Project 5 Operating Systems Page 20 FileControlBlock (FCB) and OpenFile The semantics of files in the kernel you are building will be similar to the semantics of files in Unix. Consider the case where one process has opened a file and does a kernel call to read, say, 10 bytes. The kernel must read the appropriate sector, extract the 10 bytes out of that sector, and finally copy those 10 bytes into the process’s virtual memory space. This requires the kernel to maintain a frame of memory to use as a buffer; the sector will be read into this buffer by the OS. If the 10 bytes happen to span the boundary between sectors, the kernel must read both sectors in order to complete the Read syscall. And of course, during the I/O operations other threads must be allowed to run. Now consider what happens when a process wants to write, say, 20 bytes to a file. The kernel will need to bring in the appropriate sector and copy the 20 bytes from the process’s virtual address space to the buffer. Should the kernel write the buffer back to disk immediately? No; it is likely that the process will want to write some more bytes to that very same sector, so it is more efficient to leave the sector in memory. When should the kernel write the sector back to disk? When the process closes the file, the kernel must write it back. Also, other I/O operations on the file may need different sectors, so the kernel should write the sector back to disk when the buffer is needed for another sector. However, if the buffer has not been modified, then there is no need to write it back to the disk. Therefore, we associate a Boolean called bufferIsDirty with each buffer frame. When a buffer is first read in from disk, it is considered to be “clean,” but after any operation modifies the buffer, it should be marked “dirty.” Next consider the case in which two processes have both opened the same file. (Let’s call them processes “A” and “B.”) Any update by process A must be immediately visible to process B. If process A writes to a file and B reads from that same file, even before A has closed the file, then B should see the new data. Since the kernel may not actually write to the disk for a long time after process A does the write, it means that processes A and B must share the buffer. Also, when one process finally closes a file, the buffer must be written back to the disk. The guarantee the kernel makes is that once we return from a call to Sys_Close, the disk has been updated. The program can stop worrying about failures, etc., and can tell the user that it has completed its task. Any changes the program has made—even if the system crashes in the next instance—will be permanent and will not be lost. After a Sys_Close, the kernel must not return to the user-level program until the buffer (or all buffers, if there are more than one) is written to the disk successfully. The purpose of a FileControlBlock (FCB) is to record all the data associated with a single file. This includes the buffer and the bufferIsDirty bit. Here is the definition of FCB: Project 5 Operating Systems Page 21 class FileControlBlock superclass Listable fields fcbID: int numberOfUsers: int — count of OpenFiles pointing here startingSectorOfFile: int — or -1 if FCB not in use sizeOfFileInBytes: int bufferPtr: int — addr of a page frame relativeSectorInBuffer: int — or -1 if none bufferIsDirty: bool — Set to true when buffer is modified methods Init () Print () endClass A small number FCBs are preallocated and kept in a table called fcbTable, which is maintained by the FileManager. The FileManager is responsible for allocating new FileControlBlock objects and for returning unused FileControlBlock objects to a free pool called fcbFreeList. The startingSectorOfFile tells where the file is located on the disk. Since all the sectors in a file are contiguous, the starting address and the length are all we need. The meaning of sizeOfFileInBytes is… well, obvious. [Descriptive variable names like we tend to use are a HUGE help in understanding and reading code!] A single memory frame is allocated for each FCB at kernel startup time and bufferPtr is set to point to that memory region. relativeSectorInBuffer tells which sector of the file is currently in the buffer and is –1 if there is no valid data in the buffer. Next consider a process “A” that has opened a file. All of the “read” and “write” operations that the user-level process executes are relative to a “current position” in the file. Several processes may have the same file open. All processes that have file “F” open will share a single FCB. However, they will each have a different “current position” in the file. To handle the current position, we have the class OpenFile, which is defined as: class OpenFile superclass Listable fields kind: int — FILE, TERMINAL, or PIPE currentPos: int — 0 = first byte of file fcb: ptr to FileControlBlock — null = not open numberOfUsers: int — count of Processes pointing here methods Print () ReadBytes (targetAddr, numBytes: int) returns bool — true=All Okay ReadInt () returns int LoadExecutable (addrSpace: ptr to AddrSpace) returns int — -1 = problems endClass Like the FCBs, there is a preallocated pool of OpenFile objects, which are created at system startup time. The FileManager is responsible for allocating new OpenFile objects and for returning unused OpenFile objects to a free pool called openFileFreeList. Project 5 Operating Systems Page 22 When process “A” opens a file, a new OpenFile object must be allocated and made to point to an FCB describing the file. If there is already an FCB for that file, then the OpenFile should be made to point to it; otherwise, we’ll have to get a new FCB, check the directory, and set up the FCB. When do we return an FCB to the free pool? When there are no more OpenFiles using it. This is the reason we have a field called numberOfUsers in the FCB. This field is a “reference counter.” It tells the number of OpenFile objects that point to the FCB. When a new OpenFile is allocated and made to point to an FCB, the count must be incremented. When an OpenFile is closed, the count should be decremented. When the count becomes zero, the FCB must be returned to the free pool. When a process is terminated, for example due to an error such as an AlignmentException, the kernel must close any and all OpenFiles the process is using. The process may explicitly close an OpenFile with the Close syscall. Once a file is closed, the process should attempt no further I/O on the file and if the process does, the kernel should catch it and treat it as an error (by returning an error code from the Sys_Read or Sys_Write kernel call). Our file I/O will follow the semantics of Unix. When a process is cloned with the Fork syscall, all open files in the parent process must be shared with the child process. Consider what happens when a parent and a child are both writing to the same file, which was originally opened in the parent. Since both processes share the OpenFile object, they will share the current position. If the child writes 5 bytes, the current position will be incremented by 5. Then, if the parent writes 13 bytes, these 13 bytes will follow the 5 bytes written by the child. In order to implement these semantics, it will be possible for several PCBs to point to the same OpenFile object. We need to maintain a reference count for the OpenFiles, just like the reference count for the FCBs. Whenever a process opens a file, we need to allocate a new OpenFile object and set its count to 1. Whenever a process forks, we’ll need to increment the count. When a process closes a file (either by invoking the Close syscall or by dying), we’ll need to decrement the count. If the count goes to zero, we’ll need to return the OpenFile to the free pool and decrement the count associated with the FCB. User-level processes must not be allowed to use pointers into kernel memory and cannot be allowed to touch kernel data structures such as OpenFiles and FCBs. So how does a user process refer to an OpenFile object? Indirectly, through an integer. Here’s how it works. Each Process will have a small array of pointers to OpenFiles called fileDescriptor. class ProcessControlBlock … fields … fileDescriptor: array [MAX_FILES_PER_PROCESS] of ptr to OpenFile methods … endClass When a process invokes the Open syscall, a new OpenFile will be set up. Then the kernel will select an unused position in this array and make it point to the OpenFile. For example, positions 0, 1, and 2 Project 5 Operating Systems Page 23 might be in use, so the kernel may assign a file descriptor of 3 for the newly opened file. The kernel must make fileDescriptor[3] point to the OpenFile and should return “3” as the fileDescriptor to the user-level process. When the user-level process wants to do an I/O operation, such as Read, Write, Seek, or Close, it must supply the fileDescriptor. The kernel must check that (1) this number is a valid index into the array, and (2) the array element points to a valid OpenFile. When closing the file, the kernel will need to decrement the reference count for the OpenFile object and also set fileDescriptor[3] to null. Then, if the user process attempts any future I/O operations with file descriptor 3, the kernel can detect that it is an error. Since user-level file I/O will not be implemented in this project, you will not need to worry about fileDescriptors yet. When a user-level program does a Read or Write syscall—in Unix or in our OS—the data may be transferred from/to either • a file on the disk • an I/O device such as a keyboard or display (these are called “special files” in Unix) • another process, via a “pipe” In all three cases, an OpenFile object will be used. The field called kind tells whether the object corresponds to a FILE, the TERMINAL, or a PIPE. In this project, we will only use OpenFiles to perform the Exec syscall, so the kind will be only FILE (and not TERMINAL or PIPE). To Read in an Executable File To read in an executable file from disk, your code will need to: • Open the file • Invoke LoadExecutable to do the work • Close the file Read through the code for FileManager.Open: method Open (filename: String) returns ptr to OpenFile Open is passed a ptr to array of char; this is the name of the file on the BLITZ disk that you want to read from. It will allocate a new OpenFile object and a new FCB object and set them up. Then it will return a pointer to the OpenFile object, which you’ll use when calling LoadExecutable. If anything goes wrong, Open returns null. The only real danger is getting the filename wrong. In BLITZ, like Unix, executable files have rather complex format. For details, you can read through the document titled “The Format of BLITZ Object and Executable Files.” So that you don’t have to write all this code, we are providing a method called OpenFile.LoadExecutable: method LoadExecutable (addrSpace: ptr to AddrSpace) returns int Project 5 Operating Systems Page 24 Look through LoadExecutable; it will • Create a new address space (by calling frameManager.GetNewFrames) • Read the executable program into the new address space • Determine the starting address (the initial program counter, also called the “entry point”) • Return the entry point If there are any problems with the executable file, this method will return –1. Otherwise it will return the entry point of the executable. This is the address (in the logical address space) at which execution should begin. Normally, this will be 0x00000000. User-Level Processes Each user-level process will have a single thread which will normally execute in User mode, with “paging” turned on and interrupts enabled. Each user-level process will have a logical address space, which will consist of • A Page for “environment” data • Pages for the text segment • Pages for the data segment • Pages for the BSS segment • Pages for the user’s stack These are shown in order, with the stack pages in the highest addresses of the logical address space. The environment page will sit at address 0 and will contain information that the OS wishes to pass to a new user-level process. This includes userID, working directory, etc. We will not use an environment page, so the text pages will begin at address 0. Kernel.h contains this: const NUMBER_OF_ENVIRONMENT_PAGES = 0 USER_STACK_SIZE_IN_PAGES = 1 MAX_PAGES_PER_VIRT_SPACE = 20 The text pages contain the program and any constant values. The data pages will contain the static (global) program variables. The BSS pages will contain space for uninitialized program variables (such as large arrays). The OS will set all bytes in the BSS pages to zero. Most KPL programs do not use a BSS segment, so there will usually be zero BSS pages. Project 5 Operating Systems Page 25 The user-level program will have a stack, which will grow downward. Each logical address space will have a predetermined small number of pages (in our case, this is one page) set aside for its stack. In Unix, if a user process’s stack grows beyond its initial allocation, more stack pages would be added. In our OS, if a user process’s stack grows beyond this, it will begin overwriting the BSS and data pages, and the program will probably get an error of some sort soon thereafter. As an example, a program might use: 0 environment pages 2 text pages 1 data page 0 BSS pages 1 stack page This process’s logical address space will have 4 pages. Each page has PAGE_SIZE bytes (8 Kbytes), so the entire address space will be 32 Kbytes. Any address between 0x00000000 and 0x00007FFF (which is 32K-1 in hex) would be legal for this program. If the program tries to use any other address, a PageInvalid Exception will occur. In Unix, the environment and text pages would be marked read-only and any attempt to update bytes in those pages would cause an exception. In this project, all pages of the logical address space will be read-write, so our OS will not be able to catch that sort of error in the user-level program. Each page in the logical address space will be stored in one frame in memory. The frames do not have to be contiguous and the pages may be stored in pretty much any order. However, all pages will be in memory throughout the process’s lifetime. The page table will keep track of where each page is kept. While the process is executing, “paging” will be turned on so that the memory management unit (MMU) will translate all logical addresses into physical addresses. Our example program will not be able to read or write anything outside of its 4 pages. There may be several processes in the system at any time. Each ProcessControlBlock contains an AddrSpace, which tells how many pages the process’s address space has and which frame in physical memory holds each page. When some process (call it “P”) is ready to be scheduled and given a time-slice, the MMU will be need to be set up so that it points to the page table for process P. You can do this with the method: AddrSpace.SetToThisPageTable () which calls an assembly routine to load the MMU registers. This method must be invoked before paging is turned on. When paging is turned off (i.e., whenever kernel code is being executed), the MMU registers are ignored. Project 5 Operating Systems Page 26 Note that each thread will have two stacks: a user stack and a system stack. We have already seen the system stack; it is used when one kernel function calls another kernel function. The user stack will be used when the thread is running in user mode. The system stack, which is fairly small, normally contains nothing while the user-level program is running. In other words, the system stack is completely empty. After the user-level program begins executing, execution can re-enter the kernel in only through exception processing. That is, the only ways to get back into the kernel are: • an interrupt, • a program exception, or • a syscall In each of these cases, the exact same thing happens: some information is pushed onto the system stack, the mode is changed to system mode, paging is turned off, and a jump is made to a kernel “handler” routine. The BLITZ computer has two sets of registers: one for user-mode code and one for system-mode code. Thus, the user registers do not need to be saved, unless the kernel will switch to another thread. This is done in the Run method, which contains this code: if prevThread.isUserThread SaveUserRegs (&prevThread.userRegs[0]) endIf … Switch (prevThread, nextThread) … if currentThread.isUserThread RestoreUserRegs (¤tThread.userRegs[0]) currentThread.myProcess.addrSpace.SetToThisPageTable () endIf If the kernel handler code wishes to return to the same user-level code that was interrupted, it can merely return to the assembly language handler routine, which will perform a “reti” instruction. The user registers and the MMU registers will (presumably) be unchanged, so when the mode reverts to “user mode” and the paging reverts to “paging enabled,” the user-level program will resume execution with the same values in the user registers and the same logical address space. Creating a User-Level Process The main function calls function InitFirstProcess, which you must implement. The first thing you’ll need to do is get a new thread object by invoking GetANewThread. Since the InitFirstProcess function should return, you cannot use the current thread. Next you’ll need to initialize the thread and invoke Fork to start it running. (You can name this new thread something like “UserProgram,” but the name is only used in the debugging printouts.) Project 5 Operating Systems Page 27 The new thread should execute the StartUserProcess function, which will do the remainder of the work in starting up a user-level process. InitFirstProcess can supply a zero as an argument to StartUserProcess and can return after forking the new thread. The first thing you’ll need to do in StartUserProcess is allocate a new PCB (with GetANewProcess) and connect it with the thread. So initialize the myThread field in the PCB and the myProcess field in the current thread. Next, you’ll need to open the executable file. It is acceptable to “hardcode” the filename (e.g., “TestProgram1”) into the call to Open, although changing the name of the initial process will require a recompile of the kernel. If there are problems with the Open, this is a fatal, unrecoverable error and the kernel startup process will fail. Next, you’ll need to create the logical address space and read the executable into it. The method OpenFile.LoadExecutable will take care of both tasks. If this fails, the kernel cannot start up. LoadExecutable returns the entry point, which you might call initPC. Don’t forget to close the executable file you opened earlier, or else a system resource will be permanently locked up. Next, you’ll need to compute the initial value for the user-level stack, which you might call InitUserStackTop. It should be set to the logical address just past the end of the logical address space, since the initial push onto the user stack will first decrement the top pointer. The logical address space starts at zero. The logical address space contains addrSpace.numberOfPages pages. Each page has size PAGE_SIZE bytes. The StartUserProcess function will end by jumping into the user-level program. This is a one way jump; execution will never return. (Instead, if the user-level program needs to re-enter the kernel, it will execute a syscall). As such, nothing on the system stack will ever be needed again. We want to have a full-sized system stack available for processing any syscalls or interrupts that happen later, so you need to reset the system stack top pointer, effectively clearing the system stack. You might call the new value initSystemStackTop. You’ll need to set it to: & currentThread.systemStack[SYSTEM_STACK_SIZE-1] Project 5 Operating Systems Page 28 Next, you’ll need to turn this thread into a user-level thread. This involves these actions: 1. Disable interrupts 2. Initialize the page table registers for this logical address space 3. Set the isUserThread variable in the current thread to true 4. Set system register r15, the system stack top 5. Set user register r15, the user stack top 6. Clear the System mode bit in the condition code register to switch into user mode 7. Set the Paging bit in the cond. code register, causing the MMU to do virtual memory mapping 8. Set the Interrupts Enabled bit in the cond. code register, so that future interrupts will be handled 9. Jump to the initial entry point in the program Recall that every thread begins life with interrupts enabled, so your StartUserProcess function will be executing with interrupts enabled. The first step is to disable interrupts, since there are possible race conditions with steps (2) and (3). [What is the race problem? Consider what happens if a context switch (i.e., timer interrupt) were to occur between setting the page table registers and setting isUserThread to true. Look at the Run method. The MMU registers would be changed for the other process; then when this thread is once again scheduled, the code in Run will see isUserThread==false so it will not restore the MMU registers. Merely swapping the order of steps (2) and (3) results in a similar race condition.] The first 3 steps can be done in high-level KPL code, but steps (4) through (9) must be done in assembly language. Read through the BecomeUserThread assembly routine in the file Switch.s., which will take care of steps (4) through (9). StartUserProcess should end with a call to this routine: BecomeUserThread (initUserStackTop, initPC, initSystemStackTop) The BecomeUserThread will change the mode bits and perform the jump “atomically.” This must be done atomically since the target jump address is a logical address space. (The way it does this is a little tricky: it pushes some stuff onto the system stack to make it look like syscall or interrupt has occurred, and then executes a “reti” instruction.) BecomeUserThread jumps to the user-level main routine and never returns. Approach to Implementing the Exec Syscall The sequence of steps in InitFirstProcess and StartUserProcess is very similar to what you’ll need when implementing the Exec syscall. You should be able to copy much of this code when implementing Sys_Handle_Exec. Project 5 Operating Systems Page 29 One difference is that during an Exec, you already have a process and a thread, so you will not need to allocate a new ProcesControlBlock, allocate a new Thread object, or do a fork. However, you will have to work with two virtual address spaces. The LoadExecutable method requires an empty AddrSpace object; it will then allocate as many frames as necessary and initialize the new address space. Unfortunately, LoadExecutable may fail and, if so, your kernel must be able to return to the process that invoked Exec (with an error code, of course). So you better not get rid of the old address space until after the new one has been initialized and you can be sure that no more errors can occur. One approach is to create a local variable of type AddrSpace. Don’t allocate it on the heap, just use something like: var newAddrSpace: AddrSpace = new AddrSpace Then, after the new address space has been set up, you can copy it into the ProcessControlBlock, e.g., currentThread.myProcess.addrSpace = newAddrSpace Don’t forget to free the frames in the previous address space first, or else valuable kernel resources will remain forever unavailable and the kernel will eventually freeze up! Another tricky thing is copying the filename string from a virtual address space into the kernel address space where it can be used. The filename argument is a virtual address, but since the kernel is running in Handle_Sys_Exec, paging will be turned off. You’ll need to copy the characters into an array variable, not something newly allocated on the heap. It is okay to put a maximum size on this array and then check that it is not exceeded. In fact, there is a constant in Kernel.h for this purpose: const MAX_STRING_SIZE = 20 (In a real OS, the maximum string size would be much larger or even nonexistent. Here, we use a small size to make testing the limits easier.) Note that the filename pointer is virtual address, which must be translated into a physical address; you can’t just use it, as is. This requires some code to perform the page table lookup in software. Furthermore, since the filename string is in virtual space, it may cross page boundaries. (In fact, the test program contains cases where this happens!) Dealing with the filename is fairly complex, but it turns out that we are giving you a method GetStringFromVirtual (kernelAddr: String, virtAddr, maxSize: int) returns int which will do most of the work. (GetStringFromVirtual calls CopyBytesFromVirtual to do the copying.) The GetStringFromVirtual method can be used like this: Project 5 Operating Systems Page 30 var strBuffer: array [MAX_STRING_SIZE] of char … i = currentThread.myProcess.addrSpace.GetStringFromVirtual ( & strBuffer, filename asInteger, MAX_STRING_SIZE) if i < 0 …error… endIf You might think of allocating a temporary buffer on the heap, but remember that we do not want to allocate anything on the heap after kernel start-up. [ Recall that the “alloc” expression in KPL always allocates bytes on the heap. Once the kernel has booted and is running, you must avoid further allocations. Why? One problem is automatic garbage collection like you see in Java; we can’t use automatic garbage collection since it would produce unpredictable delays and might cause the kernel to miss interrupts or, in the case of a real-time system, miss deadlines. Also, there is the possibility that the heap might fill up, and dealing with a “heap full” error in the kernel is difficult. Another option might be to try to manage the heap without automatic garbage collection, but years of C++ experience has taught everybody that this is very difficult to do correctly. This explains why we have gone to the trouble to create classes like ThreadManager and ProcessManager, instead of simply allocating new Thread and ProcessControlBlock objects. ] AllocateRandomFrames The main function includes a function named AllocateRandomFrames, which is aimed only at catching bugs in the kernel. This function will allocate every other frame in the physical memory and never release them, creating a “checkerboard pattern” in memory. Henceforth, no two pages will ever be allocated to contiguous page frames. Large, multi-byte chunks of data in the user-level process’s address space will occasionally span page boundaries. Since these pages may not be in adjacent frames, your kernel will have to be careful about moving data to and from user space. What may appear to the user-level program as a string of adjacent bytes may in fact be spread all over memory. Some of the user-level syscalls pass pointers to the kernel. For example, Open passes a pointer to a string of characters. Keep in mind that this pointer is a logical address, not a physical address. As such, you cannot simply use the pointer as is. Take a look at these methods AddrSpace: CopyBytesFromVirtual (kernelAddr, virtAddr, numBytes: int) returns int CopyBytesToVirtual (virtAddr, kernelAddr, numBytes: int) returns int GetStringFromVirtual (kernelAddr: String, virtAddr, maxSize: int) returns int An invocation of AllocateRandomFrames has been added just after the FrameManager is initialized. Please leave this in and do not modify the AllocateRandomFrames routine. Project 5 Operating Systems Page 31 What to Hand In Please submit hardcopy of your output and hardcopy your new versions of the following files: Kernel.h Kernel.c For both files, please take a colored pen and circle exactly those parts you have added or changed. Please submit all of Kernel.h. For Kernel.c, please submit only the pages that have something circled. In other words, please discard all pages that contain ONLY material that we are distributing. Please keep the remaining pages in order. Also, if you have modified any part of a method or function, please hand in the entire function, so there is a little context surrounding your code modifications. As a second option, which will save on wasted paper, you may perform the deletions with an editor, instead of discarding physical pages. You can cut out large sections of code, but please indicate where material has been clipped out. Please replaced deleted material with a line like this XXXXXXXXXXXXXXXXXXX skipping XXXXXXXXXXXXXXXXXXX to indicate which lines were removed. But please remember to circle your material with a colored pen. For the output, please run TestProgram1 (which exec’s TestProgram2) and submit what it produces. Coding Style For all projects, please follow our coding style. Make your code match our code as closely as possible. The goal is for it to be impossible to tell, just by looking at the indentation and commenting, whether we wrote the code or you wrote the code. (Of course, your code will be circled!) Please use our convention in labeling and commenting functions: Project 5 Operating Systems Page 32 —————————– Foo ——————————— function Foo () — — This routine does…. — var x: int x = 1 x = 2 … x = 9 endFunction Methods are labeled like this, with both class and method name: ———- Condition . Wait ———- method Wait (mutex: ptr to Mutex) … Note that indentation is in increments of 2 spaces. Please be careful to indent statements such as if, while, and for properly. If you follow these conventions, then even if you throw away other pages, it should be clear which class the method belongs to. Sample Output If your program works correctly, you should see something like this: =================== KPL PROGRAM STARTING =================== Initializing Thread Scheduler… Initializing Process Manager… Initializing Thread Manager… Initializing Frame Manager… AllocateRandomFrames called. NUMBER_OF_PHYSICAL_PAGE_FRAMES = 100 Initializing Disk Driver… Initializing Serial Driver… Serial handler thread running… Initializing File Manager… Loading initial program… User-level program ‘TestProgram1’ is running… ***** Testing Syscall Parameter Passing ***** ***** About to call Sys_Yield… ***** Should print: ***** Handle_Sys_Yield invoked! Handle_Sys_Yield invoked! ***** About to call Sys_Fork… ***** Should print: ***** Handle_Sys_Fork invoked! Handle_Sys_Fork invoked! Project 5 Operating Systems Page 33 ***** About to call Sys_Join… ***** Should print: ***** Handle_Sys_Join invoked! ***** processID = 1111 Handle_Sys_Join invoked! processID = 1111 ***** About to call Sys_Create… ***** Should print: ***** Handle_Sys_Create invoked! ***** virt addr of filename = 0x0000BFF8 ***** filename = MyFileName Handle_Sys_Create invoked! virt addr of filename = 0x0000BFF8 filename = MyFileName ***** About to call Sys_Open… ***** Should print: ***** Handle_Sys_Open invoked! ***** virt addr of filename = 0x0000BFF8 ***** filename = MyFileName Handle_Sys_Open invoked! virt addr of filename = 0x0000BFF8 filename = MyFileName ***** About to call Sys_Read… ***** Should print: ***** Handle_Sys_Read invoked! ***** fileDesc = 2222 ***** virt addr of buffer = 0x0000B0A8 ***** sizeInBytes = 3333 Handle_Sys_Read invoked! fileDesc = 2222 virt addr of buffer = 0x0000B0A8 sizeInBytes = 3333 ***** About to call Sys_Write… ***** Should print: ***** Handle_Sys_Write invoked! ***** fileDesc = 4444 ***** virt addr of buffer = 0x0000B0A8 ***** sizeInBytes = 5555 Handle_Sys_Write invoked! fileDesc = 4444 virt addr of buffer = 0x0000B0A8 sizeInBytes = 5555 ***** About to call Sys_Seek… ***** Should print: ***** Handle_Sys_Seek invoked! ***** fileDesc = 6666 ***** newCurrentPos = 7777 Handle_Sys_Seek invoked! fileDesc = 6666 newCurrentPos = 7777 ***** About to call Sys_Close… ***** Should print: ***** Handle_Sys_Close invoked! ***** fileDesc = 8888 Handle_Sys_Close invoked! fileDesc = 8888 ***** About to call Sys_Exit… ***** Should print: ***** Handle_Sys_Exit invoked! Project 5 Operating Systems Page 34 ***** returnStatus = 9999 Handle_Sys_Exit invoked! returnStatus = 9999 ***** Syscall Test Complete ***** ***** Testing Exec Syscall ***** ***** About to call Sys_Exec with a non-existant file… ***** Should print: ***** Okay Okay ***** About to call Sys_Exec with an overly long file name… ***** Should print: ***** Okay Okay ***** About to perform a successful Exec and jump to TestProgram2… ***** Should print: ***** User-level program ‘TestProgram2’ is running! User-level program ‘TestProgram2’ is running! ***** About to call Sys_Shutdown… ***** Should print: ***** FATAL ERROR in UserProgram: “Syscall ‘Shutdown’ was invoked by a user thread” — TERMINATING! FATAL ERROR in UserProgram: “Syscall ‘Shutdown’ was invoked by a user thread” — TERMINATING! (To find out where execution was when the problem arose, type ‘st’ at the emulator prompt.) =================== KPL PROGRAM TERMINATION ===================
1. Generating Random Networks 1. Create random networks using Erd¨os-R´enyi (ER) model (a) Create undirected random networks with n = 900 nodes, and the probability p for drawing an edge between two arbitrary vertices 0.002, 0.006, 0.012, 0.045, and 0.1. Plot the degree distributions. What distribution (linear/exponential/gaussian/binomial or something else) is observed? Explain why. Also, report the mean and variance of the degree distributions and compare them to the theoretical values. Hint Useful function(s): samplegnp (erdos.renyi.game) , degree , degreedistribution , plot (b) For each p and n = 900, answer the following questions: Are all random realizations of the ER network connected? Numerically estimate the probability that a generated network is connected. For one instance of the networks with that p, find the giant connected component (GCC) if not connected. What is the diameter of the GCC? Hint Useful function(s): isconnected , clusters , diameter (c) It turns out that the normalized GCC size (i.e., the size of the GCC as a fraction of the total network size) is a highly nonlinear function of p, with interesting properties occurring for values where p = O( 1 n ) and p = O( ln n n ). For n = 900, sweep over values of p from 0 to a pmax that makes the network almost surely connected and create 100 random networks for each p. pmax should be roughly determined by yourself. Then scatter plot the normalized GCC sizes vs p. Plot a line of the average normalized GCC sizes for each p along with the scatter plot. i. Empirically estimate the value of p where a giant connected component starts to emerge (define your criterion of “emergence”)? Do they match with theoretical values mentioned or derived in lectures? ii. Empirically estimate the value of p where the giant connected component takes up over 99% of the nodes in almost every experiment. (d) i. Define the average degree of nodes c = n × p = 0.5. Sweep over the number of nodes, n, ranging from 100 to 10000. Plot the expected size of the GCC of ER networks with n nodes and edge-formation probabilities p = c/n, as a function of n. What trend is observed? ii. Repeat the same for c = 1. iii. Repeat the same for values of c = 1.15, 1.25, 1.35, and show the results for these three values in a single plot. iv. What is the relation between the expected GCC size and n in each case? 2. Create networks using preferential attachment model (a) Create an undirected network with n = 1050 nodes, with preferential attachment model, where each new node attaches to m = 1 old nodes. Is such a network always connected? Hint Useful function(s): samplepa (barabasi.game) (b) Use fast greedy method to find the community structure. Measure modularity. Define Assortativity. Compute Assortativity. Hint Useful function(s): clusterfastgreedy , modularity (c) Try to generate a larger network with 10500 nodes using the same model. Compute modularity and assortativity. How is it compared to the smaller network’s modularity? 2 (d) Plot the degree distribution in a log-log scale for both n = 1050, 10500, then estimate the slope of the plot using linear regression. (e) In the two networks generated in 2(a) and 2(c), perform the following: Randomly pick a node i, and then randomly pick a neighbor j of that node. Plot the degree distribution of nodes j that are picked with this process, in the log-log scale. Is the distribution linear in the log-log scale? If so, what is the slope? How does this differ from the node degree distribution? Hint Useful function(s): sample (f) Estimate the expected degree of a node that is added at time step i for 1 ≤ i ≤ 1050. Show the relationship between the age of nodes and their expected degree through an appropriate plot. Note that the newest added node is the youngest. (g) Repeat the previous parts (a-f) for m = 2, and m = 6. Compare the results of each part for different values of m. (h) Again, generate a preferential attachment network with n = 1050, m = 1. Take its degree sequence and create a new network with the same degree sequence, through stub-matching procedure. Plot both networks, mark communities on their plots, and measure their modularity. Compare the two procedures for creating random power-law networks. Hint In case that fastgreedy community detection fails because of self-loops, you may use “walktrap” community detection. Useful function(s): sampledegseq 3. Create a modified preferential attachment model that penalizes the age of a node (a) Each time a new vertex is added, it creates m links to old vertices and the probability that an old vertex is cited depends on its degree (preferential attachment) and age. In particular, the probability that a newly added vertex connects to an old vertex is proportional to: P[i] ∼ (ckα i + a)(dlβ i + b), where ki is the degree of vertex i in the current time step, and li is the age of vertex i. Produce such an undirected network with 1050 nodes and parameters m = 1, α = 1, β = −1, and a = c = d = 1, b = 0. Plot the degree distribution. What is the power law exponent? Hint Useful function(s): samplepaage (b) Use fast greedy method to find the community structure. What is the modularity? 3 2. Random Walk on Networks 1. Random walk on Erd¨os-R´enyi networks (a) Create an undirected random network with 900 nodes, and the probability p for drawing an edge between any pair of nodes equal to 0.015. (b) Let a random walker start from a randomly selected node (no teleportation). We use t to denote the number of steps that the walker has taken. Measure the average distance (defined as the shortest path length) ⟨s(t)⟩ of the walker from his starting point at step t. Also, measure the variance σ 2 (t) = ⟨(s(t) − ⟨s(t)⟩) 2 ⟩ of this distance. Plot ⟨s(t)⟩ v.s. t and σ 2 (t) v.s. t. Here, the average ⟨·⟩ is over random choices of the starting nodes. (c) Measure the degree distribution of the nodes reached at the end of the random walk. How does it compare to the degree distribution of graph? (d) Repeat 1(b) for undirected random networks with 9000 nodes. Compare the results and explain qualitatively. Does the diameter of the network play a role? 2. Random walk on networks with fat-tailed degree distribution (a) Generate an undirected preferential attachment network with 900 nodes, where each new node attaches to m = 1 old nodes. (b) Let a random walker start from a randomly selected node. Measure and plot ⟨s(t)⟩ v.s. t and σ 2 (t) v.s. t. (c) Measure the degree distribution of the nodes reached at the end of the random walk on this network. How does it compare with the degree distribution of the graph? (d) Repeat 2(b) for preferential attachment networks with 90 and 9000 nodes, and m = 1. Compare the results and explain qualitatively. Does the diameter of the network play a role? 3. PageRank The PageRank algorithm, as used by the Google search engine, exploits the linkage structure of the web to compute global “importance” scores that can be used to influence the ranking of search results. Here, we use random walk to simulate PageRank. (a) We are going to create a directed random network with 900 nodes, using the preferential attachment model. Note that in a directed preferential attachment network, the out-degree of every node is m, while the in-degrees follow a power law distribution. One problem of performing random walk in such a network is that, the very first node will have no outbounding edges, and be a “black hole” which a random walker can never “escape” from. To address that, let’s generate another 900-node random network with preferential attachment model, and merge the two networks by adding the edges of the second graph to the first graph with a shuffling of the indices of the nodes. For example, 1 → 2 2 → 3 3 → 4 Initial edge list of the second network 4 → 3 3 → 1 1 → 2 Edge list after node idx shuffling (the edges that should be added to the first network) shuffled to 1,2,3,4 4,3,1,2 4 Create such a network using m = 4. Measure the probability that the walker visits each node. Is this probability related to the degree of the nodes? Hint Useful function(s): asedgelist , sample , permute , addedges (b) In all previous questions, we didn’t have any teleportation. Now, we use a teleportation probability of α = 0.2 (teleport out of a node with prob=0.2 instead of going to its neighbor). By performing random walks on the network created in 3(a), measure the probability that the walker visits each node. How is this probability related to the degree of the node and α ? 4. Personalized PageRank While the use of PageRank has proven very effective, the web’s rapid growth in size and diversity drives an increasing demand for greater flexibility in ranking. Ideally, each user should be able to define their own notion of importance for each individual query. (a) Suppose you have your own notion of importance. Your interest in a node is proportional to the node’s PageRank, because you totally rely upon Google to decide which website to visit (assume that these nodes represent websites). Again, use random walk on network generated in question 3 to simulate this personalized PageRank. Here the teleportation probability to each node is proportional to its PageRank (as opposed to the regular PageRank, where at teleportation, the chance of visiting all nodes are the same and equal to 1 N ). Again, let the teleportation probability be equal to α = 0.2. Compare the results with 3(a). (b) Find two nodes in the network with median PageRanks. Repeat part 4(a) if teleportations land only on those two nodes (with probabilities 1/2, 1/2). How are the PageRank values affected? (c) More or less, 4(b) is what happens in the real world, in that a user browsing the web only teleports to a set of trusted web pages. However, this is against the assumption of normal PageRank, where we assume that people’s interest in all nodes are the same. Can you take into account the effect of this self-reinforcement and adjust the PageRank equation? 5 Final Remarks The following functions from igraph library are useful for this project: • degree, degree.distribution, diameter, clusters, vcount, ecount • random.graph.game, barabasi.game, aging.prefatt.game, degree.sequence.game • page rank For part 2 of the project, you can start off with the Jupyter notebook provided to you.In this project, we will study the various properties of social networks. In the first part of the project, we will study an undirected social network (Facebook). In the second part of the project, we will study a directed social network (Google +). You can complete the Project using R or Python. 1. Facebook network In this project, we will be using the dataset given below: http://snap.stanford.edu/data/egonets-Facebook.html The Facebook network can be created from the edgelist file (facebook combined.txt) 1. Structural properties of the Facebook network Having created the Facebook network, we will study some of the structural properties of the network. To be specific, we will study • Connectivity • Degree distribution QUESTION 1: A first look at the network: QUESTION 1.1: Report the number of nodes and number of edges of the Facebook network. QUESTION 1.2: Is the Facebook network connected? If not, find the giant connected component (GCC) of the network and report the size of the GCC. QUESTION 2: Find the diameter of the network. If the network is not connected, then find the diameter of the GCC. QUESTION 3: Plot the degree distribution of the facebook network and report the average degree. QUESTION 4: Plot the degree distribution of Question 3 in a log-log scale. Try to fit a line to the plot and estimate the slope of the line. 1 2. Personalized network A personalized network of an user vi is defined as the subgraph induced by vi and it’s neighbors. In this part, we will study some of the structural properties of the personalized network of the user whose graph node ID is 1 (node ID in edgelist is 0). From this point onwards, whenever we are refering to a node ID we mean the graph node ID which is 1 + node ID in edgelist. QUESTION 5: Create a personalized network of the user whose ID is 1. How many nodes and edges does this personalized network have? Hint Useful function(s): makeegograph QUESTION 6: What is the diameter of the personalized network? Please state a trivial upper and lower bound for the diameter of the personalized network. QUESTION 7: In the context of the personalized network, what is the meaning of the diameter of the personalized network to be equal to the upper bound you derived in Question 6. What is the meaning of the diameter of the personalized network to be equal to the lower bound you derived in Question 6 (assuming there are more than 3 nodes in the personalized network)? 3. Core node’s personalized network A core node is defined as the nodes that have more than 200 neighbors. For visualization purpose, we have displayed the personalized network of a core node below. An example of a personal network. The core node is shown in black. In this part, we will study various properties of the personalized network of the core nodes. QUESTION 8: How many core nodes are there in the Facebook network. What is the average degree of the core nodes? 3.1. Community structure of core node’s personalized network In this part, we study the community structure of the core node’s personalized network. To be specific, we will study the community structure of the personalized network of the following core nodes: 2 • Node ID 1 • Node ID 108 • Node ID 349 • Node ID 484 • Node ID 1087 QUESTION 9: For each of the above core node’s personalized network, find the community structure using Fast-Greedy, Edge-Betweenness, and Infomap community detection algorithms. Compare the modularity scores of the algorithms. For visualization purpose, display the community structure of the core node’s personalized networks using colors. Nodes belonging to the same community should have the same color and nodes belonging to different communities should have different color. In this question, you should have 15 plots in total. Hint Useful function(s): clusterfastgreedy , clusteredgebetweenness , clusterinfomap 3.2. Community structure with the core node removed In this part, we will explore the effect on the community structure of a core node’s personalized network when the core node itself is removed from the personalized network. QUESTION 10: For each of the core node’s personalized network (use same core nodes as Question 9), remove the core node from the personalized network and find the community structure of the modified personalized network. Use the same community detection algorithm as Question 9. Compare the modularity score of the community structure of the modified personalized network with the modularity score of the community structure of the personalized network of Question 9. For visualization purpose, display the community structure of the modified personalized network using colors. In this question, you should have 15 plots in total. 3.3. Characteristic of nodes in the personalized network In this part, we will explore characteristics of nodes in the personalized network using two measures. These two measures are stated and defined below: • Embeddedness of a node is defined as the number of mutual friends a node shares with the core node. • Dispersion of a node is defined as the sum of distances between every pair of the mutual friends the node shares with the core node. The distances should be calculated in a modified graph where the node (whose dispersion is being computed) and the core node are removed. For further details on the above characteristics, you can read the paper below: http://arxiv.org/abs/1310.6753 QUESTION 11: Write an expression relating the Embeddedness between the core node and a non-core node to the degree of the non-core node in the personalized network of the core node. 3 QUESTION 12: For each of the core node’s personalized network (use the same core nodes as Question 9), plot the distribution histogram of embeddedness and dispersion. In this question, you will have 10 plots. Hint Useful function(s): neighbors , intersection , distances QUESTION 13: For each of the core node’s personalized network, plot the community structure of the personalized network using colors and highlight the node with maximum dispersion. Also, highlight the edges incident to this node. To detect the community structure, use Fast-Greedy algorithm. In this question, you will have 5 plots. QUESTION 14: Repeat Question 13, but now highlight the node with maximum embeddedness and the node with maximum dispersion embeddedness (excluding the nodes having zero embeddedness if there are any). Also, highlight the edges incident to these nodes. Report the id of those nodes. QUESTION 15: Use the plots from Question 13 and 14 to explain the characteristics of a node revealed by each of this measure. 4. Friend recommendation in personalized networks In many social networks, it is desirable to predict future links between pairs of nodes in the network. In the context of this Facebook network it is equivalent to recommending friends to users. In this part of the project, we will explore some neighborhood-based measures for friend recommendation. The network that we will be using for this part is the personalized network of node with ID 415. 4.1. Neighborhood based measure In this project, we will be exploring three different neighborhood-based measures. Before we define these measures, let’s introduce some notation: • Si is the neighbor set of node i in the network • Sj is the neighbor set of node j in the network Then, with the above notation we define the three measures below: • Common neighbor measure between node i and node j is defined as Common Neighbors(i, j) = |Si ∩ Sj | • Jaccard measure between node i and node j is defined as Jaccard(i, j) = |Si ∩ Sj | |Si ∪ Sj | • Adamic-Adar measure between node i and node j is defined as Adamic Adar(i, j) = X k∈Si∩Sj 1 log(|Sk|) 4 4.2. Friend recommendation using neighborhood based measures We can use the neighborhood based measures defined in the previous section to recommend new friends to users in the network. Suppose we want to recommend t new friends to some user i in the network using Jaccard measure. We follow the steps listed below: 1. For each node in the network that is not a neighbor of i, compute the jaccard measure between the node i and the node not in the neighborhood of i Compute Jaccard(i, j) ∀j ∈ S C i 2. Then pick t nodes that have the highest Jaccard measure with node i and recommend these nodes as friends to node i 4.3. Creating the list of users Having defined the friend recommendation procedure, we can now apply it to the personalized network of node ID 415. Before we apply the algorithm, we need to create the list of users who we want to recommend new friends to. We create this list by picking all nodes with degree 24. We will denote this list as Nr. QUESTION 16: What is |Nr|, i.e. the length of the list Nr? 4.4. Average accuracy of friend recommendation algorithm In this part, we will apply the 3 different types of friend recommendation algorithms to recommend friends to the users in the list Nr. We will define an average accuracy measure to compare the performances of the friend recommendation algorithms. Suppose we want to compute the average accuracy of the friend recommendation algorithm. This task is completed in two steps: 1. Compute the average accuracy for each user in the list Nr 2. Compute the average accuracy of the algorithm by averaging across the accuracies of the users in the list Nr Let’s describe the procedure for accomplishing the step 1 of the task. Suppose we want to compute the average accuracy for user i in the list Nr. We can compute it by iterating over the following steps 10 times and then taking the average: 1. Remove each edge of node i at random with probability 0.25. In this context, it is equivalent to deleting some friends of node i. Let’s denote the list of friends deleted as Ri 2. Use one of the three neighborhood based measures to recommend |Ri | new friends to the user i. Let’s denote the list of friends recommended as Pi 3. The accuracy for the user i for this iteration is given by |Pi∩Ri| |Ri| By iterating over the above steps for 10 times and then taking the average gives us the average accuracy of user i. In this manner, we compute the average accuracy for each user in the list 5 Nr. Once we have computed them, then we can take the mean of the average accuracies of the users in the list Nr. The mean value will be the average accuracy of the friend recommendation algorithm. QUESTION 17: Compute the average accuracy of the friend recommendation algorithm that uses: • Common Neighbors measure • Jaccard measure • Adamic Adar measure Based on the average accuracy values, which friend recommendation algorithm is the best? Hint Useful function(s): similarity 2. Google+ network In this part, we will explore the structure of the Google+ network. The dataset for creating the network can be found in the link below: http://snap.stanford.edu/data/egonets-Gplus.html Create directed personal networks for users who have more than 2 circles. The data required to create such personal networks can be found in the file named gplus.tar.gz. QUESTION 18: How many personal networks are there? QUESTION 19: For the 3 personal networks (node ID given below), plot the in-degree and outdegree distribution of these personal networks. Do the personal networks have a similar in and out degree distribution? In this question, you should have 6 plots. • 109327480479767108490 • 115625564993990145546 • 101373961279443806744 1. Community structure of personal networks In this part of the project, we will explore the community structure of the personal networks that we created and explore the connections between communities and user circles. QUESTION 20: For the 3 personal networks picked in Question 19, extract the community structure of each personal network using Walktrap community detection algorithm. Report the modularity scores and plot the communities using colors. Are the modularity scores similar? In this question, you should have 3 plots. Having found the communities, now we will explore the relationship between circles and communities. In order to explore the relationship, we define two measures: 6 • Homogeneity • Completeness Before, we state the expression for homogeneity and completeness, let’s introduce some notation: • C is the set of circles, C = {C1, C2, C3, · · · } • K is the set of communities, K = {K1, K2, K3, · · · } • ai is the number of people in circle Ci • bi is the number of people in community Ki with circle information • N is the total number of people with circle information • Aji is the number of people belonging to community j and circle i Then, with the above notation, we have the following expressions for the entropy H(C) = − X |C| i=1 ai N log( ai N ) (1) H(K) = − X |K| i=1 bi N log( bi N ) (2) and conditional entropy H(C|K) = − X |K| j=1 X |C| i=1 Aji N log(Aji bj ) (3) H(K|C) = − X |C| i=1 X |K| j=1 Aji N log(Aji ai ) (4) Now we can state the expression for homogeneity, h as h = 1 − H(C|K) H(C) (5) and the expression for completeness, c as c = 1 − H(K|C) H(K) (6) QUESTION 21: Based on the expression for h and c, explain the meaning of homogeneity and completeness in words. QUESTION 22: Compute the h and c values for the community structures of the 3 personal network (same nodes as Question 19). Interpret the values and provide a detailed explanation. Are there negative values? Why? 7 3. Cora dataset One of the well-known categories of machine learning problems is “supervised learning”. In supervised learning, we are given some information called “input” features about certain objects. For each object, we are also given an “output” or target variable that we are trying to predict about. Our goal is to learn the mapping between the features and the target variable. Typically, there is a portion of data where both input features and target variables are available. This portion of the dataset is called the training set. There is also typically another portion of the dataset where the target variable is missing and we want to predict it. This portion is called the “test set”. When the target variable can take on a finite number of discrete values, we call the problem at hand a “classification” problem. In this project, we are trying to solve a classification problem in settings where some additional information is provided in the form of “graph structure”. In this project we work with “Cora” dataset. Cora consists of a set of 2708 documents that are Machine Learning related papers. Each documents is labeled with one of the following seven classes: Case Based, Genetic Algorithms, Neural Networks, Probabilistic Methods, Reinforcement Learning, Rule Learning, Theory. For each class, only 20 documents are labeled (a total of 140 for the seven classes). We refer to them as “seed” documents. Each document comes with a set of features about its text content. These features are occurrences of a selection of 1433 words in the vocabulary. We are also given an undirected graph where each node is a document and each edge represents a citation. There are a total of 5429 edges. Our goal is to use the hints from text features as well as from graph connections to classify (assign labels to) these documents. To solve this problem for Cora dataset, we pursue three parallel ideas. Implement each idea and compare. QUESTION 23: Idea 1 Use Graph Convolutional Networks [1]. What hyperparameters do you choose to get the optimal performance? How many layers did you choose? QUESTION 24: Idea 2 Extract structure-based node features using Node2Vec [2]. Briefly describe how Node2Vec finds node features. Choose your desired classifier (one of SVM, Neural Network, or Random Forest) and classify the documents using only Node2Vec (graph structure) features. Now classify the documents using only the 1433-dimensional text features. Which one outperforms? Why do you think this is the case? Combine the Node2Vec and text features and train your classifier on the combined features. What is the best classification accuracy you get (in terms of the percentage of test documents correctly classified)? QUESTION 25: Idea 3 We can find the personalized PageRank of each document in seven different runs, one per class. In each run, select one of the classes and take the 20 seed documents of that class. Then, perform a random walk with the following customized properties: (a) teleportation takes the random walker to one of the seed documents of that class (with a uniform probability of 1/20 per seed document). Vary the teleportation probability in {0, 0.1, 0.2}. (b) the probability of transitioning to neighbors is not uniform among the neighbors. Rather, it is proportional to the cosine similarity between the text features of the current node and the next neighboring node. Particularly, assume we are currently visiting a document x0 which has neighbors x1, x2, x3. 8 Then the probability of transitioning to each neighbor is: pi = exp(x0 · xi) exp(x0 · x1) + exp(x0 · x2) + exp(x0 · x3) ; for i = 1, 2, 3. (7) Repeat part b for every teleportation probability in part a. Run the PageRank only on the GCC. for each seed node, do 1000 random walks. Maintain a class-wise visited frequency count for every unlabeled node. The predicted class for that unlabeled node is the class which lead to maximum visits to that node. Report accuracy and f1 scores. For example if node ’n’ was visited by 7 random walks from class A, 6 random walks from class B… 1 random walk from class G, then the predicted label of node of ’n’ is class A. Submission Please submit the report to Gradescope. Meanwhile, please submit a zip file containing your codes and report to BruinLearn. The zip file should be named as “Project2 UID1 … UIDn.zip” where UIDx are student ID numbers of team members. If you have any questions please feel free to ask them over email or piazza. References [1] Kipf, Thomas N., and Max Welling. “Semi-supervised classification with graph convolutional networks.” arXiv preprint arXiv:1609.02907 (2016).1 Introduction Reinforcement Learning (RL) is the task of learning from interaction to achieve a goal. The learner and the decision maker is called the agent. The thing it interacts with, comprising everything outside the agent, is called the environment. These interact continually, the agent selecting actions and the environment responding to those actions by presenting rewards and new states. In the first part of the project, we will learn the optimal policy of an agent navigating in a 2-D environment. We will implement the Value iteration algorithm to learn the optimal policy. Inverse Reinforcement Learning (IRL) is the task of extracting an expert’s reward function by observing the optimal policy of the expert. In the second part of the project, we will explore the application of IRL in the context of apprenticeship learning. 2 Reinforcement learning (RL) The two main objects in Reinforcement learning are: • Agent • Environment In this project, we will learn the optimal policy of a single agent navigating in a 2-D environment. 2.1 Environment In this project, we assume that the environment of the agent is modeled by a Markov Decision Process (MDP). In a MDP, agents occupy a state of the environment and perform actions to change the state they are in. After taking an action, they are given some representation of the new state and some reward value associated with the new state. An MDP formally is a tuple (S, A,P a ss′, Ra ss′, γ) where: 1 • S is a set of states • A is a set of actions • P a ss′ is a set of transition probabilities, where P a ss′ is the probability of transitioning from state s ∈ S to state s ′ ∈ S after taking action a ∈ A – P a ss′ = P(st+1 = s ′ |st = s, at = a) • Given any current state and action, s and a, together with any next state, s ′ , the expected value of the next reward is Ra ss′ – Ra ss′ = E(rt+1|st = s, at = a, st+1 = s ′ ) • γ ∈ [0, 1) is the discount factor, and it is used to compute the present value of future reward – If γ is close to 1 then the future rewards are discounted less – If γ is close to 0 then the future rewards are discounted more In the next few subsections, we will discuss the parameters that will be used to generate the environment for the project. 2.1.1 State space In this project, we consider the state space to be a 2-D square grid with 100 states. The 2-D square grid along with the numbering of the states is shown in figure 1 Figure 1: 2-D square grid with state numbering 2.1.2 Action set In this project, we consider the action set(A) to contain the 4 following actions: • Move Right 2 • Move Left • Move Up • Move Down The 4 types of actions are displayed in figure 2 Figure 2: 4 types of action From the above figure, we can see that the agent can take 4 actions from the state marked with a dot. 2.1.3 Transition probabilities In this project, we define the transition probabilities in the following manner: 1. If state s ′ and s are not neighboring states in the 2-D grid, then P(st+1 = s ′ |st = s, at = a) = 0 s ′ and s are neighbors in the 2-D grid if you can move to s ′ from s by taking an action a from the action set A. We will consider a state s to be a neighbor of itself. For example, from figure 1 we can observe that states 1 and 11 are neighbors (we can transition from 1 to 11 by moving right) but states 1 and 12 are not neighbors. 2. Each action corresponds to a movement in the intended direction with probability 1 − w, but has a probability of w of moving in a random direction instead due to wind. To illustrate this, let’s consider the states shown in figure 3 3 Figure 3: Inner grid states (Non-boundary states) The transition probabilities for the non-boundary states shown in figure 3 are given below: P(st+1 = 43|st = 44, at =↑) = 1 − w + w 4 P(st+1 = 34|st = 44, at =↑) = w 4 P(st+1 = 54|st = 44, at =↑) = w 4 P(st+1 = 45|st = 44, at =↑) = w 4 From the above calculation it can be observed that if the agent is at a non-boundary state then it has 4 neighbors excluding itself and the probability w is uniformly distributed over the 4 neighbors. Also, if the agent is at a non-boundary state then it transitions to a new state after taking an action (P(st+1 = 44|st = 44, at =↑) = 0) 3. If the agent is at one of the four corner states (0,9,90,99), the agent stays at the current state if it takes an action to move off the grid or is blown off the grid by wind. The actions can be divided into two categories: • Action to move off the grid • Action to stay in the grid To illustrate this, let’s consider the states shown in figure 4 4 Figure 4: Corner states The transition probabilities for taking an action to move off the grid are given below: P(st+1 = 10|st = 0, at =↑) = w 4 P(st+1 = 1|st = 0, at =↑) = w 4 P(st+1 = 0|st = 0, at =↑) = 1 − w + w 4 + w 4 The transition probabilities for taking an action to stay in the grid are given below: P(st+1 = 10|st = 0, at =→) = 1 − w + w 4 P(st+1 = 1|st = 0, at =→) = w 4 P(st+1 = 0|st = 0, at =→) = w 4 + w 4 At a corner state, you can be blown off the grid in two directions. As a result, we have P(st+1 = 0|st = 0, at =→) = w 4 + w 4 since we can be blown off the grid in two directions and in both the cases we stay at the current state. 4. If the agent is at one of the edge states, the agent stays at the current state if it takes an action to move off the grid or is blown off the grid by wind. The actions can be divided into two categories: • Action to move off the grid • Action to stay in the grid To illustrate this, let’s consider the states shown in figure 5 5 Figure 5: Edge states The transition probabilities for taking an action to move off the grid are given below: P(st+1 = 0|st = 1, at =←) = w 4 P(st+1 = 11|st = 1, at =←) = w 4 P(st+1 = 2|st = 1, at =←) = w 4 P(st+1 = 1|st = 1, at =←) = 1 − w + w 4 The transition probabilities for taking an action to stay in the grid are given below: P(st+1 = 0|st = 1, at =↑) = 1 − w + w 4 P(st+1 = 11|st = 1, at =↑) = w 4 P(st+1 = 2|st = 1, at =↑) = w 4 P(st+1 = 1|st = 1, at =↑) = w 4 At an edge state, you can be blown off the grid in one direction. As a result, we have P(st+1 = 1|st = 1, at =↑) = w 4 since we can be blown off the grid in one direction and in that case we stay at the current state. The main difference between a corner state and an edge state is that a corner state has 2 neighbors and an edge state has 3 neighbors. 2.1.4 Reward function To simplify the project, we will assume that the reward function is independent of the current state (s) and the action that you take at the current state (a). To be specific, reward 6 function only depends on the state that you transition to (s ′ ). With this simplification, we have Ra ss′ = R(s ′ ) In this project, we will learn the optimal policy of an agent for two different reward functions: • Reward function 1 • Reward function 2 The two different reward functions are displayed in figures 6 and 7 respectively Figure 6: Reward function 1 7 Figure 7: Reward function 2 Question 1: (10 points) For visualization purpose, generate heat maps of Reward function 1 and Reward function 2. For the heat maps, make sure you display the coloring scale. You will have 2 plots for this question For solving question 1, you might find the following function useful: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.pcolor.html 3 Optimal policy learning using RL algorithms In this part of the project, we will use reinforcement learning (RL) algorithm to find the optimal policy. The main steps in RL algorithm are: • Find optimal state-value or action-value • Use the optimal state-value or action-value to determine the deterministic optimal policy There are a couple of RL algorithms, but we will use the Value iteration algorithm since it was discussed in detail in the lecture. We will skip the derivation of the algorithm here because it was covered in the lecture (for the derivation details please refer to the lecture slides on Reinforcement learning). We will just reproduce the algorithm below for the ease of implementation: 8 1: procedure Value Iteration(P a ss′ , Ra ss′ , S, A, γ): 2: for all s ∈ S do ▷ Initialization 3: V (s) ← 0 4: end for 5: ∆ ← ∞ 6: while ∆ > ϵ do ▷ Estimation 7: ∆ ← 0 8: for all s ∈ S do 9: v ← V (s); 10: V (s) ← max a∈A X s ′∈S P a ss′ [Ra ss′ + γV (s ′ )]; 11: ∆ ← max(∆, |v − V (s)|); 12: end for 13: end while 14: for all s ∈ S do ▷ Computation 15: π(s) ← arg max a∈A X s ′∈S P a ss′ [Ra ss′ + γV (s ′ )]; 16: end for 17: end procedure return π Question 2: (40 points) Create the environment of the agent using the information provided in section 2. To be specific, create the MDP by setting up the state-space, action set, transition probabilities, discount factor, and reward function. For creating the environment, use the following set of parameters: • Number of states = 100 (state space is a 10 by 10 square grid as displayed in figure 1) • Number of actions = 4 (set of possible actions is displayed in figure 2) • w = 0.1 • Discount factor = 0.8 • Reward function 1 After you have created the environment, then write an optimal state-value function that takes as input the environment of the agent and outputs the optimal value of each state in the grid. For the optimal state-value function, you have to implement the Initialization (lines 2-4) and Estimation (lines 5-13) steps of the Value Iteration algorithm. For the estimation step, use ϵ = 0.01. For visualization purpose, you should generate a figure similar to that of figure 1 but with the number of state replaced by the optimal value of that state. In this part of question, you should have 1 plot. Let’s assume that your value iteration algorithm converges in N steps. Plot snapshots of state values in 5 different steps linearly distributed from 1 to N. Report N and your step numbers. What observations do you have from the plots? Question 3: (5 points) Generate a heat map of the optimal state values across the 2- D grid. For generating the heat map, you can use the same function provided in the hint earlier (see the hint after question 1). Question 4: (15 points) Explain the distribution of the optimal state values across the 2-D grid. (Hint: Use the figure generated in question 3 to explain) Question 5: (20 points) Implement the computation step of the value iteration algorithm 9 (lines 14-17) to compute the optimal policy of the agent navigating the 2-D state-space. For visualization purpose, you should generate a figure similar to that of figure 1 but with the number of state replaced by the optimal action at that state. The optimal actions should be displayed using arrows. Does the optimal policy of the agent match your intuition? Please provide a brief explanation. Is it possible for the agent to compute the optimal action to take at each state by observing the optimal values of it’s neighboring states? In this question, you should have 1 plot. Question 6: (10 points) Modify the environment of the agent by replacing Reward function 1 with Reward function 2. Use the optimal state-value function implemented in question 2 to compute the optimal value of each state in the grid. For visualization purpose, you should generate a figure similar to that of figure 1 but with the number of state replaced by the optimal value of that state. In this question, you should have 1 plot. Question 7: (20 points) Generate a heat map of the optimal state values (found in question 6) across the 2-D grid. For generating the heat map, you can use the same function provided in the hint earlier. Explain the distribution of the optimal state values across the 2-D grid. (Hint: Use the figure generated in this question to explain) Question 8: (20 points) Implement the computation step of the value iteration algorithm (lines 14-17) to compute the optimal policy of the agent navigating the 2-D state-space. For visualization purpose, you should generate a figure similar to that of figure 1 but with the number of state replaced by the optimal action at that state. The optimal actions should be displayed using arrows. Does the optimal policy of the agent match your intuition? Please provide a brief explanation. In this question, you should have 1 plot. Question 9:(20 points) Change the hyper parameter w to 0.6 and find the optimal policy map similar to previous question for reward functions. Explain the differences you observe. What do you think about value of new w compared to previous value? Choose the w that you think give rise to better optimal policy and use that w for the next stages of the project. 4 Inverse Reinforcement learning (IRL) Inverse Reinforcement learning (IRL) is the task of learning an expert’s reward function by observing the optimal behavior of the expert. The motivation for IRL comes from apprenticeship learning. In apprenticeship learning, the goal of the agent is to learn a policy by observing the behavior of an expert. This task can be accomplished in two ways: 1. Learn the policy directly from expert behavior 2. Learn the expert’s reward function and use it to generate the optimal policy The second way is preferred because the reward function provides a much more parsimonious description of behavior. Reward function, rather than the policy, is the most succinct, robust, and transferable definition of the task. Therefore, extracting the reward function of an expert would help design more robust agents. In this part of the project, we will use IRL algorithm to extract the reward function. 10 We will use the optimal policy computed in the previous section as the expert behavior and use the algorithm to extract the reward function of the expert. Then, we will use the extracted reward function to compute the optimal policy of the agent. We will compare the optimal policy of the agent to the optimal policy of the expert and use some similarity metric between the two to measure the performance of the IRL algorithm. 4.1 IRL algorithm For finite state spaces, there are a couple of IRL algorithms for extracting the reward function: • Linear Programming (LP) formulation • Maximum Entropy formulation Since we covered the LP formulation of inverse reinforcement learning in lecture—and it is simpler than the original quadratic-program approach—we adopt that version here [?]. A step-by-step derivation can be found in our course slides [?]. The LP formulation of the IRL is given by equation 1 maximize R,ti,ui P|S| i=1(ti − λui) subject to [(Pa1 (i) − Pa(i))(I − γPa1 ) −1R] ≥ ti , ∀a ∈ A a1, ∀i (Pa1 − Pa)(I − γPa1 ) −1R ⪰ 0, ∀a ∈ A a1 −u ⪯ R ⪯ u |Ri | ≤ Rmax, i = 1, 2, · · · , |S| (1) In the LP given by equation 1, R is the reward vector (R(i) = R(si)), Pa is the transition probability matrix corresponding to the action a, λ is the adjustable penalty coefficient, and ti ’s and ui ’s are the extra optimization variables (please note that u(i) = ui). Throughout IRL section we fix the discount factor to γ = 0.9. Use the maximum absolute value of the ground-truth reward as Rmax. For the ease of implementation, we can recast the LP in equation 1 into an equivalent form given by equation 2 using block matrices. maximize x c T x subject to Dx ⪯ b, ∀a ∈ A a1 (2) Question 10: (10 points) Express c, x, D, b in terms of R, Pa, Pa1 , ti , u, λ and Rmax 4.2 Performance measure In this project, we use a very simple measure to evaluate the performance of the IRL algorithm. Before we state the performance measure, let’s introduce some notation: • OA(s): Optimal action of the agent at state s • OE(s): Optimal action of the expert at state s • m(s) = ( 1, OA(s) = OE(s) 0, else 11 Then with the above notation, accuracy is given by equation 3 Accuracy = P s∈S m(s) |S| (3) Since we are using the optimal policy found in the previous section as the expert behavior, so we will use the optimal policy found in the previous section to fill the OE(s) values. Please note that these values will be different depending on whether we used Reward Function 1 or Reward Function 2 to create the environment. To compute OA(s), we will solve the linear program given by equation 2 to extract the reward function of the expert. For solving the linear program you can use the LP solver in python (from cvxopt import solvers and then use solvers.lp). Then, we will use the extracted reward function to compute the optimal policy of the agent using the value iteration algorithm you implemented in the previous section. The optimal policy of the agent found in this manner will be used to fill the OA(s) values. Please note that these values will depend on the adjustable penalty coefficient λ. We will tune λ to maximize the accuracy. Implementation Guide Constants γ = 0.8 Rmax = maxi |Rtrue(si)| CVXOPT options from cvxopt import s o l v e r s s o l v e r s . o p ti o n s . update ({ ’ a b s t o l ’ : 1 e−7, ’ r e l t o l ’ : 1 e −6, ’ f e a s t o l ’ : 1 e −7, ’ s ho w p rog r e s s ’ : Fal s e }) Question 11: (30 points) Sweep λ from 0 to 5 to get 500 evenly spaced values for λ. For each value of λ compute OA(s) by following the process described above. For this problem, use the optimal policy of the agent found in question 5 to fill in the OE(s) values. Then use equation 3 to compute the accuracy of the IRL algorithm for this value of λ. You need to repeat the above process for all 500 values of λ to get 500 data points. Plot λ (x-axis) against Accuracy (y-axis). In this question, you should have 1 plot. Question 12: (5 points) Use the plot in question 11 to compute the value of λ for which accuracy is maximum. For future reference we will denote this value as λ (1) max. Please report λ (1) max Question 13: (15 points) For λ (1) max, generate heat maps of the ground truth reward and the extracted reward. Please note that the ground truth reward is the Reward function 1 and the extracted reward is computed by solving the linear program given by equation 2 with the λ parameter set to λ (1) max. In this question, you should have 2 plots. Question 14: (10 points) Use the extracted reward function computed in question 13, to compute the optimal values of the states in the 2-D grid. For computing the optimal values you need to use the optimal state-value function that you wrote in question 2. For visualization purpose, generate a heat map of the optimal state values across the 2-D grid (similar to the figure generated in question 3). In this question, you should have 1 plot. 12 Question 15: (10 points) Compare the heat maps of Question 3 and Question 14 and provide a brief explanation on their similarities and differences. Question 16: (10 points) Use the extracted reward function found in question 13 to compute the optimal policy of the agent. For computing the optimal policy of the agent you need to use the function that you wrote in question 5. For visualization purpose, you should generate a figure similar to that of figure 1 but with the number of state replaced by the optimal action at that state. The actions should be displayed using arrows. In this question, you should have 1 plot. Question 17: (10 points) Compare the figures of Question 5 and Question 16 and provide a brief explanation on their similarities and differences. Question 18: (30 points) Sweep λ from 0 to 5 to get 500 evenly spaced values for λ. For each value of λ compute OA(s) by following the process described above. For this problem, use the optimal policy of the agent found in question 8 to fill in the OE(s) values. Then use equation 3 to compute the accuracy of the IRL algorithm for this value of λ. You need to repeat the above process for all 500 values of λ to get 500 data points. Plot λ (x-axis) against Accuracy (y-axis). In this question, you should have 1 plot. Question 19: (5 points) Use the plot in question 18 to compute the value of λ for which accuracy is maximum. For future reference we will denote this value as λ (2) max. Please report λ (2) max Question 20: (15 points) For λ (2) max, generate heat maps of the ground truth reward and the extracted reward. Please note that the ground truth reward is the Reward function 2 and the extracted reward is computed by solving the linear program given by equation 2 with the λ parameter set to λ (2) max. In this question, you should have 2 plots. Question 21: (10 points) Use the extracted reward function computed in question 20, to compute the optimal values of the states in the 2-D grid. For computing the optimal values you need to use the optimal state-value function that you wrote in question 2. For visualization purpose, generate a heat map of the optimal state values across the 2-D grid (similar to the figure generated in question 7). In this question, you should have 1 plot. Question 22: (10 points) Compare the heat maps of Question 7 and Question 21 and provide a brief explanation on their similarities and differences. Question 23: (10 points) Use the extracted reward function found in question 20 to compute the optimal policy of the agent. For computing the optimal policy of the agent you need to use the function that you wrote in question 9. For visualization purpose, you should generate a figure similar to that of figure 1 but with the number of state replaced by the optimal action at that state. The actions should be displayed using arrows. In this question, you should have 1 plot. Question 24: (10 points) Compare the figures of Question 9 and Question 23 and provide a brief explanation on their similarities and differences. 13 Question 25: (50 points) From the figure in question 23, you should observe that the optimal policy of the agent has two major discrepancies. Please identify and provide the causes for these two discrepancies. One of the discrepancy can be fixed easily by a slight modification to the value iteration algorithm. Perform this modification and re-run the modified value iteration algorithm to compute the optimal policy of the agent. Also, recompute the maximum accuracy after this modification. Is there a change in maximum accuracy? The second discrepancy is harder to fix and is a limitation of the simple IRL algorithm. 5 Submission Please submit your report to Gradescope. Also submit a zip file containing your codes and report to bruinlearn. The zip file should be named as “Project3 UID1 … UIDn.zip” where UIDx are student ID numbers of team members. ReferencesIntroduction In this project we will explore graph theory theorems and algorithms, by applying them on real data. In the first part of the project, we consider a particular graph modeling correlations between stock price time series. In the second part, we analyse traffic data on a dataset provided by Uber. 1. Stock Market In this part of the project, we study data from stock market. The data is available on this Dropbox Link. The goal of this part is to study correlation structures among fluctuation patterns of stock prices using tools from graph theory. The intuition is that investors will have similar strategies of investment for stocks that are effected by the same economic factors. For example, the stocks belonging to the transportation sector may have different absolute prices, but if for example fuel prices change or are expected to change significantly in the near future, then you would expect the investors to buy or sell all stocks similarly and maximize their returns. Towards that goal, we construct different graphs based on similarities among the time series of returns on different stocks at different time scales (day vs a week). Then, we study properties of such graphs. The data is obtained from Yahoo Finance website for 3 years. You’re provided with a number of csv tables, each containing several fields: Date, Open, High, Low, Close, Volume, and Adj Close price. The files are named according to Ticker Symbol of each stock. You may find the market sector for each company in Name sector.csv. We recommend doing this part of the project (Q1 – Q8) in R. 1. Return correlation In this part of the project, we will compute the correlation among log-normalized stock-return time series data. Before giving the expression for correlation, we introduce the following notation: • pi(t) is the closing price of stock i at the t th day • qi(t) is the return of stock i over a period of [t − 1, t] qi(t) = pi(t) − pi(t − 1) pi(t − 1) 1 • ri(t) is the log-normalized return stock i over a period of [t − 1, t] ri(t) = log(1 + qi(t)) Then with the above notation, we define the correlation between the log-normalized stock-return time series data of stocks i and j as ρij = ⟨ri(t)rj (t)⟩ − ⟨ri(t)⟩⟨rj (t)⟩ p (⟨ri(t) 2⟩ − ⟨ri(t)⟩ 2 )(⟨rj (t) 2⟩ − ⟨rj (t)⟩ 2 ) where ⟨·⟩ is a temporal average on the investigated time regime (for our data set it is over 3 years). QUESTION 1: What are upper and lower bounds on ρij? Provide a justification for using lognormalized return (ri(t)) instead of regular return (qi(t)). 2. Constructing correlation graphs In this part,we construct a correlation graph using the correlation coefficient computed in the previous section. The correlation graph has the stocks as the nodes and the edge weights are given by the following expression wij = q 2(1 − ρij ) Compute the edge weights using the above expression and construct the correlation graph. QUESTION 2: Plot a histogram showing the un-normalized distribution of edge weights. 3. Minimum spanning tree (MST) In this part of the project, we will extract the MST of the correlation graph and interpret it. QUESTION 3: Extract the MST of the correlation graph. Each stock can be categorized into a sector, which can be found in Name sector.csv file. Plot the MST and color-code the nodes based on sectors. Do you see any pattern in the MST? The structures that you find in MST are called Vine clusters. Provide a detailed explanation about the pattern you observe. QUESTION 4: Run a community detection algorithm (for example walktrap) on the MST obtained above. Plot the communities formed. Compute the homogeneity and completeness of the clustering. (you can use the ’clevr’ library in r to compute homogeneity and completeness). 4. Sector clustering in MST’s In this part, we want to predict the market sector of an unknown stock. We will explore two methods for performing the task. In order to evaluate the performance of the methods we define the following metric α = 1 |V | X vi∈V P(vi ∈ Si) 2 where Si is the sector of node i. Define P(vi ∈ Si) = |Qi | |Ni | where Qi is the set of neighbors of node i that belong to the same sector as node i and Ni is the set of neighbors of node i. Compare α with the case where P(vi ∈ Si) = |Si | |V | QUESTION 5: Report the value of α for the above two cases and provide an interpretation for the difference. 5. Correlation graphs for weekly data In the previous parts, we constructed the correlation graph based on daily data. In this part of the project, we will construct a correlation graph based on WEEKLY data. To create the graph, sample the stock data weekly on Mondays and then calculate ρij using the sampled data. If there is a holiday on a Monday, we ignore that week. Create the correlation graph based on weekly data. QUESTION 6: Repeat questions 2,3,4,5 on the WEEKLY data. 6. Correlation graphs for MONTHLY data In this part of the project, we will construct a correlation graph based on MONTHLY data. To create the graph, sample the stock data Monthly on 15th and then calculate ρij using the sampled data. If there is a holiday on the 15th, we ignore that month. Create the correlation graph based on MONTHLY data. QUESTION 7: Repeat questions 2,3,4,5 on the MONTHLY data. QUESTION 8: Compare and analyze all the results of daily data vs weekly data vs monthly data. What trends do you find? What changes? What remains similar? Give reason for your observations. Which granularity gives the best results when predicting the sector of an unknown stock and why? 2. Let’s Help Santa! Companies like Google and Uber have a vast amount of statistics about transportation dynamics. Santa has decided to use network theory to facilitate his gift delivery for the next Christmas. When we learned about his decision, we designed this part of the project to help him. We will send him your results for this part! 1. Download the Data Normally we use the last winter data but this year the latest available data is Winter 2019. Go to “Uber Movement” website and download data of Travel Times by Month (All Days), 3 2019 Quarter 4, for Los Angeles area1 . The dataset contains pairwise traveling time statistics between most pairs of points in the Los Angeles area. Points on the map are represented by unique IDs. To understand the correspondence between map IDs and areas, download Geo Boundaries file from the same website2 . This file contains latitudes and longitudes of the corners of the polygons circumscribing each area. To be specific, if an area is represented by a polygon with 5 corners, then you have a 5 × 2 matrix of the latitudes and longitudes, each row of which represents latitude and longitude of one corner. We recommend doing this part of the project (Q9 – Q18) in Python. 2. Build Your Graph Read the dataset at hand, and build a graph in which nodes correspond to locations, and undirected weighted edges correspond to the mean traveling times between each pair of locations (only December). Add the centroid coordinates of each polygon region (a 2-D vector) as an attribute to the corresponding vertex. The graph will contain some isolated nodes (extra nodes existing in the Geo Boundaries JSON file) and a few small connected components. Remove such nodes and just keep the largest connected component of the graph. In addition, merge duplicate edges by averaging their weights 3 . We will refer to this cleaned graph as G afterwards. QUESTION 9: Report the number of nodes and edges in G. 3. Traveling Salesman Problem QUESTION 10: Build a minimum spanning tree (MST) of graph G. Report the street addresses near the two endpoints (the centroid locations) of a few edges. Are the results intuitive? QUESTION 11: Determine what percentage of triangles in the graph (sets of 3 points on the map) satisfy the triangle inequality. You do not need to inspect all triangles, you can just estimate by random sampling of 1000 triangles. Now, we want to find an approximation solution for the traveling salesman problem (TSP) on G. Apply the 1-approximate algorithm described in the class4 . Inspect the sequence of street addresses visited on the map and see if the results are intuitive. QUESTION 12: Find an upper bound on the empirical performance of the approximate algorithm: ρ = Approximate TSP Cost Optimal TSP Cost QUESTION 13: Plot the trajectory that Santa has to travel! 1 If you download the dataset correctly, it should be named as los angeles-censustracts-2019-4-All-MonthlyAggregate.csv 2The file should be named los angeles censustracts.json 3Duplicate edges may exist when the dataset provides you with the statistic of a road in both directions. We remove duplicate edges for the sake of simplicity. 4You can find the algorithm in: Papadimitriou and Steiglitz, “Combinatorial optimization: algorithms and complexity”, Chapter 17, page 414 4 4. Analysing Traffic Flow Next December, there is going to be a large group of visitors travelling between a location near Malibu to a location near Long Beach. We would like to analyse the maximum traffic that can flow between the two locations. 5. Estimate the Roads We want to estimate the map of roads without using actual road datasets. Educate yourself about Delaunay triangulation algorithm and then apply it to the nodes coordinates5 . QUESTION 14: Plot the road mesh that you obtain and explain the result. Create a graph G∆ whose nodes are different locations and its edges are produced by triangulation. 6. Calculate Road Traffic Flows QUESTION 15: Using simple math, calculate the traffic flow for each road in terms of cars/hour. Report your derivation. Hint: Consider the following assumptions: • Each degree of latitude and longitude ≈ 69 miles • Car length ≈ 5 m = 0.003 mile • Cars maintain a safety distance of 2 seconds to the next car • Each road has 2 lanes in each direction Assuming no traffic jam, consider the calculated traffic flow as the max capacity of each road. 7. Calculate Max Flow Consider the following locations in terms of latitude and longitude: • Source coordinates (in Malibu): [34.04, -118.56] • Destination coordinates (in Long Beach): [33.77, -118.18] QUESTION 16: Calculate the maximum number of cars that can commute per hour from Malibu to Long Beach. Also calculate the number of edge-disjoint paths between the two spots. Does the number of edge-disjoint paths match what you see on your road map? 5You can use scipy.spatial.Delaunay in Python, or RTriangle package in R. 5 8. Prune Your Graph In G∆, there are a number of unreal roads that could be removed. For instance, you might notice some unreal links along the concavities of the beach, as well as in the hills of Topanga. Apply a threshold on the travel time of the roads in G∆ to remove the fake edges. Call the resulting graph G˜∆. QUESTION 17: Plot G˜∆ on actual coordinates. Do you think the thresholding method worked? QUESTION 18: Now, repeat question 13 for G˜∆ and report the results. Do you see any changes? Why? Submission Please submit your report to Gradescope. In addition, please submit a zip file containing your codes and report to BruinLearn. The zip file should be named as “Project4 UID1 … UIDn.zip” where UIDx are student ID numbers of team members. If you have any questions you can post on piazza.
