Tutorial: Moodle VPL -- Testing the Size of A Program's Executable

From DftWiki

Jump to: navigation, search

--D. Thiebaut (talk) 19:58, 12 June 2014 (EDT)


Contents

MoodleVPLLogo.png



Moodle VPL Tutorials



This tutorial builds on the first one in the series, which should be done first, as the present tutorial skips some steps. This tutorial setups a VPL activity that assigns a grade to an assembly program depending on how small it is, 100 for the smallest size possible (assumed to be the instructor's solution), and 0 for a program larger than the minimum size plus 10 times some user-defined increment. The activity can be easily adapted to another programming language.


Setup


  • Moodle Version 2.7 + (Build: 20140529)
  • VPL Version 3.0.1
  • For details on how Moodle and VPL were installed, go to this page.


VPL Activity Description


Test Assembly Program Size

Due date: Monday, 20 June 2016, 8:00 PM
Requested files: printStars.asm (Download)
Type of work: Individual work
Grade settings: Maximum grade: 100
Run: Yes Evaluate: Yes
Automatic grade: Yes
Rewards the smallest program that correctly outputs the given pattern


Execution files


vpl_run.sh


#! /bin/bash

cat > vpl_execution << 'EOF'
#! /bin/bash

file="printStars"  # whatever executable is to be tested
bestSize=1065    # the size of the executable version of the solution program
                         # in bytes.
 
# assemble and link the executable
nasm -f elf -F stabs ${file}.asm
ld -melf_i386 -o $file ${file}.o

# run the executable in a subshell to catch any segmentation fault error
# that may result.
#{ ./$file &> /dev/null ; } &> log
 
./$file &> /dev/null
 
# if an error occured, automatically return 0
if [ $? -eq 139 ]; then
     echo "0";
     exit 0;
fi
 
# print out the size of the file
size=$(stat -c%s "$file")
 
# compare size to best size recorded and output
# grade based on size.
 
increment=20   # size increment that takes away 10 more points
                       # from the grade
 
# possible grades
grades=( 100 90 80 70 60 50 40 30 20 10 0 )
 
# find bin corresponding to size of executable, based on
# $bestSize, and bins of width $increment
for i in {0..10} ; do
     bestSize=$(($bestSize+$increment))
     if [ $size -le $bestSize ] ; then
         echo ${grades[$i]}
         exit 0
     fi  
done
echo "0"
EOF

 
chmod +x vpl_execution


vpl_debug.sh

no file needed

vpl_evaluate.sh


#!/bin/bash
cat >vpl_execution << 'EOF'
#!/bin/bash
# ====================================================
# assemble, link, and measure program footprint
# ====================================================
file="printStars"  # whatever executable is to be tested
bestSize=1065    # the size of the executable version of the solution program
                         # in bytes.
 
# assemble and link the executable
nasm -f elf -F stabs ${file}.asm
ld -melf_i386 -o $file ${file}.o
 
# run the executable in a subshell to catch any segmentation fault error
# that may result.
#{ ./$file &> /dev/null ; } &> log
 
./$file &> /dev/null
 
# if an error occured, automatically return 0
if [ $? -eq 139 ]; then
     echo "<|--"
     echo "Your program crashed."
     echo "--|>"
     echo "Grade :=>>0"
     exit 0;
fi
 
# print out the size of the file
size=$(stat -c%s "$file")

echo "Comment :=>> Your program is $size bytes long"
echo "Comment :=>> The solution program is $bestSize bytes long"
 
# compare size to best size recorded and output
# grade based on size.
 
increment=20   # size increment that takes away 10 more points
               # from the grade
 
# possible grades
grades=( 100 90 80 70 60 50 40 30 20 10 0 )
 
# find bin corresponding to size of executable, based on
# $bestSize, and bins of width $increment
for i in {0..10} ; do
    bestSize=$(($bestSize+$increment))
    if [ $size -le $bestSize ] ; then
        echo "Grade :=>> ${grades[$i]}"
        exit 0
    fi  
done
echo "Grade :=>>0"
 
EOF

 
chmod +x vpl_execution



vpl_evaluate.cases

No file needed


Some Explanations


vpl_run.sh

  • The script creates a vpl_execution script that will behave as follows:
    • First it sets up the name of the program to test (without extension), and its ideal size in bytes. In this example the program name is printStars. The closer the student program comes to this ideal size, the higher the grade.
    • The vpl_execution script will assemble the user program (called printStars.asm),
    • it will then link it and create an executable called printStars.
    • It will run the executable, and right away will test for a possible crash, in which case 0 is printed and the script stops.
    • Otherwise it computes the size of the executable, and compares it repeatedly to the best-size set at the beginning of the script plus some fixed increment, then the best size plus twice the fixed increment, then the best-size plus three times the fixed increment, and so on. The first time through the loop the possible grade is 100, then becomes 90, then 80, etc. When the student program is found to be less than the increasing size, his/her program gets the current grade.
    • The vpl_execution script outputs the grade, and not the size!


vpl_evaluate.sh


This script is very similar to the vpl_run.sh, except that it outputs the grade information in the format required by vpl.
Running vpl_evaluate.sh and then vpl_execution for a program which deserves a grade of 30/100 will generate the following output:

./vpl_evaluate.sh
./vpl_execution 
Comment :=>> Your program is 1221 bytes long
Comment :=>> The solution program is 1065 bytes long
Grade :=>> 30


Testing


Error-Free Execution


  • Go to the VPL Administration block, click on Testing activity, then Edit and enter the following program in the tab:


                        section .data
        stars           db      "*********************", 10
                        db      "*********************", 10
                        db      "*********************", 10
                        db      "*********************", 10
                        db      "*********************", 10
                        db      "*********************", 10

                                section .text
                                global _start
_start:

;;; print stars
                                mov     eax, 4
                                mov     ebx, 1
                                mov     ecx, stars
                                mov     edx, 132
                                int     0x80

;;; ; exit

                                mov     ebx, 0
                                mov     eax, 1
                                int     0x80


Verify that you get a grade of 100/100:

MoodleVPLTestAssemblyProgramSize.png



You can duplicate some of the strings of stars to increase the size of the program without modifying its behavior. You should see the grade getting lower and lower.

Buggy Program


  • Modify the program by placing a comma (;) in front of the last int instruction of the program. This prevents the program from exiting properly by calling the OS, and forces it to execute random instructions that follow the program in memory. A sure way to make it crash.
              ; int 0x80
  • Save the program.
  • Evaluate the program. Verify that you get a grade of 0.


MoodleVPLTestAssemblyProgramSizeCrash.png


This concludes this tutorial