CS350 Container

.gitignore

+assignments

README.md

+# CS350 Container
+This is the CS350 Docker container that assists students who wish to have an
+isolated local environment for their os161 and userspace
+assignments. This repo currently expects that you are running in a UNIX
+environments that support docker (MacOS, Linux, or Windows WSL)
+
+If you do not have access to this type of environment then you can use the 
+[older instructions](https://student.cs.uwaterloo.ca/~cs350/common/Install161.html) for the provided student environments.
+
+
+## Windows
+* Before you can install docker, you must have [WSL](https://docs.microsoft.com/en-us/windows/wsl/install) installed.
+* You can then [install docker](https://docs.docker.com/desktop/windows/install) for WSL
+
+
+## Requirements
+* Docker [installed](https://docs.docker.com/engine/install/)
+* Your own working copy of os161 (if you do not have one see the next section)
+
+
+## Starting Fresh with OS161
+If you are starting fresh and do not currently have a working copy of os161, you can retrieve your own by copying the
+`os161.tar.gz` file within `os161-container/dependencies` directory.
+```
+cp os161-container/dependencies/os161.tar.gz ~/
+cd ~/
+tar xzvf os161.tar.gz
+```
+You should now have a folder called os161-1.99 in your home directory. Make this a git repo!
+
+## Setup
+Once the above requirements are met, you run:
+```
+sudo ./install.sh
+```
+
+From within the project directory. This script will pull the assignments repo and build it into a local copy of the os161-runner image.
+
+
+## Running OS161
+To start using the container you use the run.sh script, which takes one argument -- the directory of your working code repo.
+
+```
+sudo ./run.sh ~/os161-1.99
+```
+Once executed you will be within the context of the container itself (its like your own mini environment), you will have preloaded two
+ease of use functions (found at /helpers.sh within the container), `build_kernel ASST#`, and `test_kernel ASST#`. Where the `#` signifies the
+assignment number.
+
+Within the container you may run within the shell (these functions are preloaded in your shell):
+```
+build_kernel ASST0
+test_kernel ASST0
+```
+To build and run within the context of ASST0. Finally you may evaluate your assignment based off public test cases (there could be hidden ones not found within
+the repo)
+
+```
+/evalaute /logs/test_public.log /assignments/ASST0
+```
+
+
+## Running other assignments
+You may point the `run.sh` to any directory as all this does is makes sure that the directory is shared between the container and you. Allowing you to
+continue to code and work on the files within the directoy and have the container running and seeing those changes. Further instructions will
+be given to the scripts used to run and evaluate userspace code.
+
+When mounting other directories that are not os161 note that the `build_kernel` and `test_kernel` functionality will not work.
+
+
+## Layout
+There are some noted directories within the container
+* `/kernel` holds YOUR assignment code
+* `/os161-1.99` holds unchanged os161 kernel code
+* `/assignments` holds testing and evaluation scripts used for each assignment
+
+Within the `/assignments` directory there are subdirectories for each assignment, to see the test cases we pass and use, looks to the `public` file,
+to see how we run and build your assignment, see the `run.sh` file. Finally, to see how we automatically evaluate the output of your assignment,
+look at `verify.py`
+
+If the assignments repo has changed, either due to assignments being changed or bugs found (found at `os161-container/assignments`) you will need to pull this repo 
+and rebuild.
+
+```
+cd os161-container/assignments
+git pull origin master
+cd ../..
+sudo docker build -t os161-runner os161-container
+```

install.sh

+#!/usr/bin/env bash
+git clone https://git.uwaterloo.ca/krhancoc/cs350-assignments.git os161-container/assignments
+
+docker build -t os161-runner ./os161-container
+

os161-container/Dockerfile

+FROM ubuntu:18.04
+
+
+# Update gcc - ubuntu is used cause seems alpine gets upset
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y gcc make wget g++ python3 libncurses-dev python3-pip && \
+    rm -rf /var/lib/apt/lists/*
+
+# Setup directories
+RUN mkdir -p /os161/tools/bin; \
+    mkdir -p /os161/tools/share/man/man1; \
+    mkdir -p /os161/tools/share/mk;
+
+
+COPY dependencies/os161-binutils.tar.gz /binutils.tar.gz
+RUN mkdir /binutils-src; \
+    tar -xf /binutils.tar.gz -C /binutils-src --strip-components 1
+WORKDIR /binutils-src
+
+RUN find . -name '*.info' | xargs touch; \
+    ./configure --nfp --disable-werror --target=mips-harvard-os161 --prefix=/os161/tools; \
+    make -j 4 && make install; \
+    rm -rf /binutils-src; 
+
+
+# Build GCC
+ENV PATH="${PATH}:/os161/tools/bin"
+
+COPY dependencies/os161-gcc.tar.gz /gcc.tar.gz
+RUN mkdir /gcc-src; \
+    tar -xf /gcc.tar.gz -C /gcc-src --strip-components 1
+WORKDIR /gcc-src
+
+RUN CFLAGS="-std=gnu89" ./configure -nfp --disable-shared --disable-threads \
+	--disable-libmudflap --disable-libssp --target=mips-harvard-os161 --prefix=/os161/tools; \
+    make -j 4 && make install; \
+    rm -rf /gcc.tar.gz /gcc-src;
+
+# Make bmake
+WORKDIR /
+COPY dependencies/os161-mk.tar.gz /mk.tar.gz
+COPY dependencies/os161-bmake.tar.gz /bmake.tar.gz
+RUN tar -xzvf /bmake.tar.gz
+
+WORKDIR /bmake
+RUN tar -xvzf ../mk.tar.gz && chmod u+x mk/install-mk; \
+    ./boot-strap --prefix=/os161/tools -m /mk; \
+    cp Linux/bmake /os161/tools/bin/ && mkdir -p /os161/tools/share/man/cat1; \
+    cp bmake.cat1 /os161/tools/share/man/cat1/bmake.1; \
+    sh mk/install-mk /os161/tools/share/mk; \
+    rm -rf /bmake.tar.gz /bmake /mk.tar.gz;
+
+# Build sys161
+COPY dependencies/sys161.tar.gz /sys161.tar.gz
+RUN mkdir /sys161-src
+RUN tar -xf /sys161.tar.gz -C /sys161-src --strip-components 1; \
+    cd /sys161-src && ./configure --prefix=/os161/tools mipseb && make -j 4 && make install; \
+    rm /sys161.tar.gz
+
+# Clean up
+WORKDIR /
+COPY dependencies/os161.tar.gz /os161.tar.gz
+RUN tar -xvzf /os161.tar.gz && rm /os161.tar.gz
+
+COPY dependencies/os161-gdb.tar.gz /os161-gdb.tar.gz
+RUN mkdir /os161-gdb/ && tar -xf /os161-gdb.tar.gz -C /os161-gdb --strip-components 1
+
+WORKDIR /os161-gdb
+RUN CFLAGS="-std=gnu89" ./configure --target=mips-harvard-os161 --prefix=/os161/tools --disable-werror && \
+    make && make install
+
+WORKDIR /
+RUN rm -rf /os161-gdb /os161-gdb.tar.gz
+
+RUN cd /os161/tools/bin && sh -c 'for i in mips-*; do ln -s /os161/tools/bin/$i /os161/tools/bin/cs350-`echo $i | cut -d- -f4-`; done'
+
+RUN python3 -m pip install -U prettytable
+COPY evaluate.py /evalaute
+COPY helpers.sh /helpers.sh
+RUN echo "source /helpers.sh" >> ~/.bashrc
+COPY run_assignment.sh /run_assignment.sh
+COPY dependencies/sys161.conf /sys161.conf
+COPY assignments /assignments
+
+# In the future I would like to run the build_run_kernel.sh right away - currently we use run.sh to start and exec.
+# There is currently an issue where when you build_run_kernel.sh as the entrypoint, when we finally try to run the 
+# tests, sys161 will start but almost immediately exit. To hack around this
+# we sleep instead and call docker exec to run it.

os161-container/dependencies/sys161.conf

+# Sample sys161.conf file
+#
+# This file tells System/161 what devices to use.
+#
+# There are 32 LAMEbus slots on the System/161 motherboard. There may
+# be only one bus controller card, and it must go in slot 31. Other
+# than that, you can put in whatever devices you want.
+#
+# The syntax is simple: one slot per line; the slot number goes first,
+# then the expansion card name, then any arguments. Some of the devices
+# have required arguments.
+#
+# The devices are:
+#
+#   mainboard The multiprocessor LAMEbus controller card. Must go in
+#             slot 31, and only in slot 31. Required argument
+#             "ramsize=NUMBER" specifies the amount of physical RAM in
+#             the system. This amount must be a multiple of the
+#             hardware page size (which is probably 4096 or 8192.) The
+#             maximum amount of RAM allowed is 16M; this restriction
+#             is meant as a sanity check and can be altered by
+#             recompiling System/161. The argument "cpus=NUMBER"
+#             selects the number of CPUs; the default is 1 and the
+#             maximum 32.
+#
+#   oldmainboard  The uniprocessor LAMEbus controller card, fully
+#             backwards compatible with OS/161 1.x. In general,
+#             uniprocessor kernels should nonetheless work on the
+#             multiprocessor mainboard; therefore this device will
+#             probably be removed in the future. Configuration is the
+#             same as the multiprocessor mainboard, except that the
+#             "cpus" argument is not accepted. The OS/161 1.x name
+#             "busctl" is an alias for "oldmainboard".
+#
+#   trace     The System/161 trace controller device. This can be used
+#             by software for various debugging purposes. You can have
+#             more than one trace card, but they all manipulate the 
+#             same internal state. No arguments.
+#
+#   timer     Countdown timer. The timer card also contains a real-time 
+#             clock and a small speaker for beeping. Most configurations
+#             will include at least one timer. No arguments. 
+#
+#   serial    Serial port. This is connected to the standard input and
+#             standard output of the System/161 process, and serves as
+#             the system console. Most configurations need this. There
+#             is no support at present for more than one serial port.
+#             No arguments.
+#
+#   screen    Full-screen memory-mapped text video card. This is 
+#             connected to the standard input and standard output of
+#             the System/161 process, and serves as the system console.
+#             There is no support at present for more than one screen.
+#             Likewise, at present you may not use "screen" and "serial"
+#             together. No arguments. NOTE: not presently implemented.
+#
+#   random    (Pseudo-)random number generator. This accesses the 
+#             randomizer state within System/161; thus, while you can
+#             add multiple random cards, they all return values from the
+#             same pseudorandom sequence. The random seed is set by 
+#             using either the "seed=NUMBER" argument, which sets the
+#             random seed to a specified value, or the "autoseed" 
+#             argument, which sets the random seed based on the host
+#             system clock. If neither argument is given or no random
+#             device is used, the seed is set to 0. Note that the seed
+#             affects various randomized behavior of the system as well
+#             as the values provided by the random device.
+#
+#   disk      Fixed disk. The options are as follows:
+#                 rpm=NUMBER         Set spin rate of disk.
+#                 sectors=NUMBER     Set disk size. Each sector is 512 bytes.
+#                 file=PATH          Specify file to use as storage for disk.
+#                 paranoid           Set paranoid mode.
+#
+#             The "file=PATH" argument must be supplied. The size must be
+#             at least 128 sectors (64k), and the RPM setting must be a
+#             multiple of 60.
+#
+#             The "paranoid" argument, if given, causes fsync() to be 
+#             called on every disk write to make sure the data written
+#             reaches actual stable storage. This will make things very 
+#             slow.
+#
+#             You can have as many disks as you want (until you run out
+#             of slots) but each should have a distinct file to use for
+#             storage. Most common setups will use two separate disks,
+#             one for filesystem storage and one for swapping.
+#
+#   nic       Network card. This allows communication among multiple
+#             simultaneously-running copies of System/161. The arguments
+#             are:
+#                 hub=PATH           Give the path to the hub socket.
+#                 hwaddr=NUMBER      Specify the hardware-level card address.
+#
+#             The hub socket path should be the argument supplied to the
+#             hub161 program. The default is ".sockets/hub".
+#
+#             The hardware address should be unique among the systems 
+#             connected to the same hub. It should be an integer between 
+#             1 and 65534. Values 0 and 65535 are reserved for special
+#             purposes. This argument is required.
+#
+#             NOTE: disable (comment out) nic devices if you aren't 
+#             actively using them, to avoid unnecessary overhead.
+#
+#   emufs     Emulator filesystem. This provides access *within* 
+#             System/161 to the filesystem that System/161 is running
+#             in. There is one optional argument, "dir=PATH". The path
+#             specified is used as the root of the filesystem provided
+#             by emufs. (Note that it is possible to access the real 
+#             parent of this root and thus any other directory; this
+#             argument does not restrict access.) The default path is
+#             ".", meaning System/161's own current directory.
+#
+
+#
+# Here is a suggested default configuration: 512k RAM, two 5M disks.
+#
+
+0	serial
+#0	screen
+
+1	emufs
+
+2	disk	rpm=7200	sectors=10240	file=DISK1.img
+3	disk	rpm=7200	sectors=10240	file=DISK2.img
+
+#27	nic hwaddr=1
+
+28	random	autoseed
+29	timer
+30	trace
+31	mainboard  ramsize=524288  cpus=1
+#31	mainboard  ramsize=524288  cpus=2
+#31	mainboard  ramsize=524288  cpus=4

os161-container/evaluate.py

+#!/usr/bin/env python3
+""" 
+Evalaute
+
+This script is expected to be run within the context of the os161 docker container
+
+Expected directories are:
+/logs
+"""
+
+import argparse
+import os
+import importlib.util
+from pprint import pprint as pprint
+
+from prettytable import PrettyTable
+
+HEADER = ["Section Name", "Marks Recieved", "Out Of", "Comments"]
+
+
+class Report:
+    def __init__(self):
+        self._marks = []
+
+    def add(self, name, marks, out_of, comments):
+        self._marks.append([name, marks, out_of, comments])
+
+    def __str__(self):
+        total = 0
+        table = PrettyTable(HEADER)
+        recieved = 0
+        for x in self._marks:
+            total += x[2]
+            recieved += x[1]
+            table.add_row(x)
+        total = "Total Marks = {} / {}".format(recieved, total)
+        tb = str(table) + "\n" + total
+
+        return tb
+
+    def to_csv(self, file):
+        with open(file, "w") as f:
+            header = ",".join(HEADER) + "\n"
+            f.write(header)
+            for l in self._marks:
+                line = ",".join([str(m) for m in l]) + "\n"
+                f.write(line)
+
+
+class Test:
+    def __init__(self, log):
+        log = [l for l in log if len(l) > 0]
+        self.name = log[0].split("=")[1]
+        self._log = "\n".join(log[4:])
+
+    def results(self):
+        return self._log
+
+
+def check_all(strs, res):
+    return any([f in res for f in strs])
+
+
+class Namespace:
+    def __init__(self, **kwargs):
+        self.__dict__.update(kwargs)
+
+
+def main(args):
+    tests = []
+    with open(args.log, "r") as f:
+        log = f.read()
+        log.strip()
+        # Remove one cause topline is a split so would get empty line before
+        tmp = log.split(">SPLIT<")[1:]
+        for t in tmp:
+            t.strip()
+            tests.append(Test(t.split("\n")))
+    # We load in the verify.py from the assignment folder
+    spec = importlib.util.spec_from_file_location(
+        "verify", os.path.join(args.vdir, "verify.py")
+    )
+    verify = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(verify)
+
+    helpers = Namespace(check_all=check_all)
+
+    # Generate the report for the student
+    report = Report()
+    for eval_func, out_of in verify.RUBRIC.items():
+        name, mark, comments = eval_func(tests, helpers)
+        report.add(name, mark, out_of, comments)
+    print(str(report))
+    report.to_csv("/logs/report.csv")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(prog="Evaluate script for CS350 test logs")
+    parser.add_argument("log", help="Log file of the test")
+    parser.add_argument(
+        "vdir", help="Path to test Directory holding tests and verify.py"
+    )
+    args = parser.parse_args()
+    main(args)

os161-container/helpers.sh

+#! /bin/bash
+export LOGS=/logs
+export OS161_DIR="/os161-1.99"
+export PATH=$PATH:/os161/tools/bin
+
+unpack_kernel() {
+	mkdir /kernel 
+	tar -xf /assignment.tgz -C /kernel --strip-components 1
+	cp -r /kernel/kern/* $OS161_DIR/kern/
+}
+
+build_k_helper() {
+	cd $OS161_DIR
+	mkdir /os-compile
+	set -e
+	# Configure - step 1
+	echo "[CS350] $1 Configuring Kernel..."
+	touch $LOGS/configure.log
+	./configure --ostree=/os-compile --toolprefix=mips-harvard-os161- >> $LOGS/configure.log
+	cd kern/conf
+	./config $1 >> $LOGS/configure.log
+
+	# Depends - step 2
+	cd ../compile/$1
+	echo "[CS350] $1 Making Kernel..."
+	touch $LOGS/depend.log
+	bmake depend >> $LOGS/depend.log 2>> $LOG/depend.log
+
+	# Make and install - step 3
+	touch $LOGS/make.log
+	bmake WERROR= >> $LOGS/make.log 2>> $LOG/make.log
+	bmake install >> $LOGS/make.log 2>> $LOG/make.log
+
+	cd /os-compile/
+	cp /sys161.conf .
+}
+
+build_kernel() {
+	if [ "$#" -ne 1 ]; then
+		echo "Usage: build_kernel <ASST0, ASST1 ...>"
+	fi
+	
+	build_k_helper $1 &
+	wait
+}
+
+test_kernel() {
+	if [ "$#" -ne 1 ]; then
+		echo "Usage: test_kernel <ASST0, ASST1 ...>"
+	fi
+
+	cd /os-compile
+	echo "[CS350] Tests $1"
+	echo "============="
+	# Run public tests - step 4
+	for f in /assignments/$1/public*
+	do
+		LOG=$LOGS/test_`basename $f`.log
+		TOTAL=$(wc -l < $f)
+		DONE=0
+		touch $LOG
+		for p in `cat $f` 
+		do
+			echo ">SPLIT<" >> $LOG
+			echo ">TEST=$p<" >> $LOG
+			sys161 kernel "$p;q" >> $LOG 2> /dev/null
+			DONE=$((DONE+1))
+			echo "[CS350] Running Tests [$DONE/$TOTAL]"
+		done
+	done
+	cd -
+}
+
+

os161-container/run_assignment.sh

+#! /bin/bash
+# 
+# We assume boths a logs and tests directory is available and is run within the context
+# of the os161 docker container
+
+set -e
+
+source /helpers.sh
+
+/assignments/$1/run.sh $1
+
+/evalaute $LOGS/test_public.log /assignments/$1

run.sh

+#! /bin/sh
+rm -rf logs
+
+set -e
+if [ "$#" -ne 1 ];then
+    echo "Usage: run.sh <os161_dir>"
+    exit 1
+fi
+
+mkdir -p logs
+
+KERNFILE=$1
+docker run -it --rm \
+	-v `realpath $KERNFILE`:/kernel \
+	-v `realpath os161-container/assignments`:/assignments \
+	-v `realpath logs`:/logs \
+	--entrypoint /bin/bash \
+	os161-runner