Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • dzlin/cs350-container
  • jahyong/cs350-container
  • koberai/cs350-container
  • pkashigi/cs350-container
  • krhancoc/cs350-container
5 results
Show changes
Commits on Source (37)
Hide whitespace changes
Inline Side-by-side
README.md
View file @ 23a948e3
# CS350 Container
This is the CS350 Docker container that assists students who wish to have an
isolated local environment for their os161 and userspace
assignments. This repo currently expects that you are running in a UNIX
environments that support docker (MacOS, Linux, or Windows WSL)
This is CS350 Docker container repository. It includes the CS350 Docker container and the tools to build, run, debug and test OS/161 and Linux based programming assignments for CS350. This repository currently expects that you are running in a UNIX or UNIX-like environment that supports docker, such as, MacOS, Linux or Windows WSL.
If you do not have access to this type of environment then you can use the
[older instructions](https://student.cs.uwaterloo.ca/~cs350/common/Install161.html) for the provided student environments.
If you have a Windows system without WSL, then start [here](https://git.uwaterloo.ca/krhancoc/cs350-container/-/edit/master/README.md#windows).
If you do not have access to a UNIX or UNIX-like environment then you can use the
[older instructions](https://student.cs.uwaterloo.ca/~cs350/common/Install161.html) to build, run, debug and test in University of Waterloo's linux student environment.
## Windows
* Before you can install docker, you must have [WSL](https://docs.microsoft.com/en-us/windows/wsl/install) installed.
* You can then [install docker](https://docs.docker.com/desktop/windows/install) for WSL
## Windows Option A
This is an emulated environment and allows for bash like commands, but you are running windows underneath. You will need to use
Git Bash for all commands stated in this README and to run any of the provided scripts.
* Install [Git for Windows](https://gitforwindows.org/)
* Install [docker](https://docs.docker.com/desktop/windows/install) for Windows.
## Windows Option B
This setup will let you have a full linux environment on your windows machine, WSL2 is just a linux VM running on your computer!
* Install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).
* Install [docker](https://docs.docker.com/desktop/windows/install) for WSL.
## Requirements
* Docker [installed](https://docs.docker.com/engine/install/)
* Your own working copy of os161 (if you do not have one see the next section)
## UNIX and UNIX-like Environments - Prerequiste to install Docker
* Install [docker](https://docs.docker.com/engine/install/)
## Starting Fresh with OS161
If you are starting fresh and do not currently have a working copy of os161, you can retrieve your own by copying the
`os161.tar.gz` file within `os161-container/dependencies` directory.
## Prerequiste - repository for OS/161 source code
* This repository assumes that you have a copy of the OS/161 source code on the local machine.
* You **should** have a remote repository to manage your OS/161 source code and move between the local and linux.student environments.
* If you do not currently have a working copy of OS/161, you can retrieve your own by copying the
`os161.tar.gz` file within `os161-container/dependencies` directory from the cs350-container repository.
```
cp os161-container/dependencies/os161.tar.gz ~/
cd ~/
tar xzvf os161.tar.gz
```
You should now have a folder called os161-1.99 in your home directory. Make this a git repo!
You should now have a folder called `os161-1.99` in your home directory. Make this a git repository!
## Setup
Once the above requirements are met, you run:
## Prerequiste complete - install CS350-container
Once the above requirements are met, make sure docker desktop is running (if on WSL or MacOS), then you can clone this repository and install the cs350-container by typing:
```
sudo ./install.sh
```
In a terminal from within the project directory. This script will pull the assignments reposiroty and build it into a local copy of the os161-runner image.
From within the project directory. This script will pull the assignments repo and build it into a local copy of the os161-runner image.
## Running OS161
To start using the container you use the run.sh script, which takes one argument -- the directory of your working code repo.
## Running OS/161 in the cs350-container
To start using the container you use the run.sh script, which takes one argument -- the directory of your working code repository.
```
sudo ./run.sh ~/os161-1.99
```
Once executed you will be within the context of the container itself (its like your own mini environment), you will have preloaded two
ease of use functions (found at /helpers.sh within the container), `build_kernel ASST#`, and `test_kernel ASST#`. Where the `#` signifies the
assignment number.
Within the container you may run within the shell (these functions are preloaded in your shell):
Once executed you will be within the context of the container itself (its like your own mini environment), there are two functions (found at /helpers.sh within the container), `build_kernel ASST#`, and `test_kernel ASST#`. Where the `#` represents the
assignment number. You can run these commands from the shell container. These functions are preloaded in your shell. For example, to build and run ASST0:
```
# build_kernel ASST0
# test_kernel ASST0
```
To build and run within the context of ASST0. Finally you may evaluate your assignment based off public test cases (there could be hidden ones not found within
the repo)
To fully evaluate youre kernel with the automated testing scripts you may use
```
# /evalaute /logs/test_public.log /assignments/ASST0
# ./run_assignment.sh ASST#
```
This will run `build_kernel` and `test_kernel` as well as evaluate the output produced by `test_kernel` (The raw output for build and test kernel can be found in /logs).
## Running GDB
To run GDB, in a window with the container running and having built the kernel using the `build_kernel ASST0`:
If you would like to run the kernel normally (no scripts), once built, you can run the kernel
```
# cd /os-compile
# sys161 kernel
```
There also exists a `clean_kernel ASST#` command which will try and clean up old builds, this may be necessary if running into issues. If this
fails that is okay (means there was no build to begin with), just run `build_kernel ASST#` as normal.
## Running GDB
* Make sure you have built the kernel using `build_kernel ASST0`.
* To run GDB, open a terminal with the cs350-container running, and enter the following commands:
```
# cd /os-compile/
# sys161 -w kernel
```
Now in another terminal, move to this project directory and use the `./connect.sh` script to connect to the running container:
* Now in another terminal, move to the project directory and use the `./connect.sh` script to connect to the previosuly running container:
```
# cd /os-compile/
# cs350-gdb kernel
# cd os-compile/
# cs350-gdb kernel
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
...
(gdb) dir /kernel/kern/compile/ASST0
(gdb) dir /os161-1.99/kern/compile/ASST0
(gdb) target remote unix:.sockets/gdb
...
```
You will need to use the command `dir /kernel/kern/compile/ASST#` where `#` is the assignment number. This will make sure to load the symbols
of the kernel. Symbols are the names we have given functions and variables, this allows gdb to convey what you are looking at (this is because
the kernel has been compiled WITH these symbols in mind), otherwise gdb will show just bare assembly!
of the kernel. Symbols are the names we have given functions and variables. This allows gdb to correctly convey what you are looking at, since the kernel has been compiled WITH these symbols, otherwise gdb will show just bare assembly!
Note: If `./connect.sh` does not work it may be because you have multiple running containers of os161-runner. Use `docker ps` to figure this out.
This script is not a smart script just an ease of use script.
Note: If run.sh fails it means there is still a os161-runner container running. Use `docker kill os161` to kill it.
## Running other assignments
You may point the `run.sh` to any directory as all this does is makes sure that the directory is shared between the container and you. Allowing you to
continue to code and work on the files within the directoy and have the container running and seeing those changes. Further instructions will
be given to the scripts used to run and evaluate userspace code.
## Building and installing OS/161 userspace programs
User-level programs are built and installed from the directory /os161-1.99/.
To build and install all of the user-level programs, in the container
When mounting other directories that are not os161 note that the `build_kernel` and `test_kernel` functionality will not work.
You can type
```
cd /os161-1.99/
bmake
bmake install
```
User-level programs are installed under $OS161TOP/root/ in the bin, sbin, testbin and uw-testbin directories. This is done automatically
by the build_kernel command.
## Running Linux userspace assignments
To run the Linux userspace programs in the container, that are part of the progrmaming assignment, you must start the container using the `run.sh` script, which takes one argument -- the directory that holds your code for that userspace programming assignment, that is, the `c` files for the programming assignments. For example,
```
sudo ./run.sh ~/cs350-teststudent/a0
```
Once you are inside the container, you can run
```
./run_assignment.sh ASSTUSER#
```
where `#` represents the assignment number.
It is important to note that: when mounting other directories that are not OS/161, the `build_kernel` and `test_kernel` functionality will not work.
## Layout
## [Layout](Layout)
There are some noted directories within the container
* `/kernel` holds YOUR assignment code
* `/os161-1.99` holds unchanged os161 kernel code
* `/os161-1.99` holds unchanged OS/161 kernel code
* `/os-compile` holds the compiled version of the code
* `/assignments` holds testing and evaluation scripts used for each assignment
Within the `/assignments` directory there are subdirectories for each assignment, to see the test cases we pass and use, looks to the `public` file,
to see how we run and build your assignment, see the `run.sh` file. Finally, to see how we automatically evaluate the output of your assignment,
look at `verify.py`
Within the `/assignments` directory there are subdirectories for each assignment, which you can review:
* the `public` file - shows the test cases we pass and use,
* the `run.sh` file - how we run and build your assignment,
* the `verify.py` script - how we evaluate the output of your assignment.
If the assignments repo has changed, either due to assignments being changed or bugs found (found at `os161-container/assignments`) you will need to pull this repo
and rebuild.
If the assignments repo has changed, either due to assignments being changed or bugs found (found at `os161-container/assignments`) you will need to just run `sudo ./install.sh` again.
```
cd os161-container/assignments
git pull origin master
cd ../..
sudo docker build -t os161-runner os161-container
```
## Common Issues
**Commands do not work and also there is randomly ^M, strings within the output:**
This is due to windows placing return characters in the strings that get
passed to the scripts. Please use git bash or a terminal that does UNIX
string passing.
connect.sh
View file @ 23a948e3
#/usr/bin/env bash
ID=`docker ps -aqf "ancestor=os161-runner"`
docker exec -it $ID /bin/bash
docker exec -it os161 /bin/bash
install.sh
View file @ 23a948e3
#!/usr/bin/env bash
git clone https://git.uwaterloo.ca/krhancoc/cs350-assignments.git os161-container/assignments
echo "Updating our own directory"
git pull origin master
echo "Cloning/Updating assignment directory"
git clone https://git.uwaterloo.ca/krhancoc/cs350-assignments.git os161-container/assignments > /dev/null 2> /dev/null
cd os161-container/assignments > /dev/null
git pull origin master > /dev/null 2> /dev/null
cd - > /dev/null
echo "Building CS350 container"
docker build -t os161-runner ./os161-container
os161-container/Dockerfile
View file @ 23a948e3
......@@ -10,12 +10,16 @@ RUN mkdir -p /os161/tools/bin; \
mkdir -p /os161/tools/share/man/man1; \
mkdir -p /os161/tools/share/mk;
# Install a newer version of config.guess with ARM support. Pin it to a specific commit for stability.
ADD https://git.savannah.gnu.org/cgit/config.git/plain/config.guess?id=20403c5701973a4cbd7e0b4bbeb627fcd424a0f1 /tmp/config.guess
RUN chmod +x /tmp/config.guess
COPY dependencies/os161-binutils.tar.gz /binutils.tar.gz
RUN mkdir /binutils-src; \
tar -xf /binutils.tar.gz -C /binutils-src --strip-components 1
WORKDIR /binutils-src
RUN cp -f /tmp/config.guess .
RUN find . -name '*.info' | xargs touch; \
./configure --nfp --disable-werror --target=mips-harvard-os161 --prefix=/os161/tools; \
make -j 4 && make install; \
......@@ -30,6 +34,7 @@ RUN mkdir /gcc-src; \
tar -xf /gcc.tar.gz -C /gcc-src --strip-components 1
WORKDIR /gcc-src
RUN cp -f /tmp/config.guess .
RUN CFLAGS="-std=gnu89" ./configure -nfp --disable-shared --disable-threads \
--disable-libmudflap --disable-libssp --target=mips-harvard-os161 --prefix=/os161/tools; \
make -j 4 && make install; \
......@@ -65,16 +70,20 @@ COPY dependencies/os161-gdb.tar.gz /os161-gdb.tar.gz
RUN mkdir /os161-gdb/ && tar -xf /os161-gdb.tar.gz -C /os161-gdb --strip-components 1
WORKDIR /os161-gdb
RUN cp -f /tmp/config.guess .
RUN CFLAGS="-std=gnu89" ./configure --target=mips-harvard-os161 --prefix=/os161/tools --disable-werror && \
make && make install
WORKDIR /
RUN rm -rf /os161-gdb /os161-gdb.tar.gz
RUN rm -f /tmp/config.guess
RUN cd /os161/tools/bin && sh -c 'for i in mips-*; do ln -s /os161/tools/bin/$i /os161/tools/bin/cs350-`echo $i | cut -d- -f4-`; done'
RUN python3 -m pip install -U prettytable
RUN apt update && apt install sudo
COPY evaluate.py /evalaute
RUN useradd -ms /usr/sbin/nologin user
COPY helpers.sh /helpers.sh
RUN echo "source /helpers.sh" >> ~/.bashrc
COPY run_assignment.sh /run_assignment.sh
......
os161-container/evaluate.py
View file @ 23a948e3
......@@ -50,8 +50,9 @@ class Report:
class Test:
def __init__(self, log):
log = [l for l in log if len(l) > 0]
self.name = log[0].split("=")[1]
self._log = "\n".join(log[4:])
self.name = log[0].split("=")[1].strip("<").strip(">")
# Remove the TEST=#### line
self._log = "\n".join(log[1:])
def results(self):
return self._log
......@@ -68,9 +69,15 @@ class Namespace:
def main(args):
tests = []
with open(args.log, "r") as f:
encoding_error = False
with open(args.log, "rb") as f:
log = f.read()
log.strip()
try:
log.decode(encoding='utf-8', errors='strict')
except:
encoding_error = True
log = log.decode(encoding='utf-8', errors='ignore')
log = "\n".join([ l.strip() for l in log.split("\n") ])
# Remove one cause topline is a split so would get empty line before
tmp = log.split(">SPLIT<")[1:]
for t in tmp:
......@@ -89,6 +96,8 @@ def main(args):
report = Report()
for eval_func, out_of in verify.RUBRIC.items():
name, mark, comments = eval_func(tests, helpers)
if encoding_error:
comments = "We could not decode your log into ASCII, this could lead to undefined behaviour everywhere, you have a memory corruption most likely in your code"
report.add(name, mark, out_of, comments)
print(str(report))
report.to_csv("/logs/report.csv")
......
os161-container/helpers.sh
View file @ 23a948e3
......@@ -8,14 +8,28 @@ unpack_kernel() {
tar -xf /assignment.tgz -C /kernel --strip-components 1
}
_clean_kernel_h() {
set -e
cd /os161-1.99/kern/compile/$1
bmake clean
cd -
}
clean_kernel() {
_clean_kernel_h $1 &
wait
}
build_k_helper() {
cd $OS161_DIR
mkdir /os-compile
mkdir /os-compile 2> /dev/null
cp -r /kernel/kern/* $OS161_DIR/kern/
rm -rf $OS161_DIR/kern/compile/* 2> /dev/null
set -e
# Configure - step 1
echo "[CS350] $1 Configuring Kernel..."
touch $LOGS/configure.log
truncate -s 0 $LOGS/configure.log
./configure --ostree=/os-compile --toolprefix=mips-harvard-os161- >> $LOGS/configure.log
cd kern/conf
./config $1 >> $LOGS/configure.log
......@@ -24,12 +38,19 @@ build_k_helper() {
cd ../compile/$1
echo "[CS350] $1 Making Kernel..."
touch $LOGS/depend.log
bmake depend >> $LOGS/depend.log 2>> $LOG/depend.log
truncate -s 0 $LOGS/depend.log
bmake depend >> $LOGS/depend.log 2>> $LOGS/depend.log
# Make and install - step 3
touch $LOGS/make.log
bmake WERROR= >> $LOGS/make.log 2>> $LOG/make.log
bmake install >> $LOGS/make.log 2>> $LOG/make.log
truncate -s 0 $LOGS/make.log
bmake WERROR= >> $LOGS/make.log 2>> $LOGS/make.log
bmake install >> $LOGS/make.log 2>> $LOGS/make.log
cd $OS161_DIR
echo "Building Userspace programs" >> $LOGS/make.log
bmake WERROR= >> $LOGS/make.log 2>> $LOGS/make.log
bmake install >> $LOGS/make.log 2>> $LOGS/make.log
cd /os-compile/
cp /sys161.conf .
......@@ -53,12 +74,13 @@ test_kernel() {
echo "[CS350] Tests $1"
echo "============="
# Run public tests - step 4
for f in /assignments/$1/public*
for f in /assignments/$1/public
do
LOG=$LOGS/test_`basename $f`.log
TOTAL=$(wc -l < $f)
DONE=0
touch $LOG
truncate -s 0 $LOG
for p in `cat $f`
do
echo ">SPLIT<" >> $LOG
......@@ -68,7 +90,7 @@ test_kernel() {
echo "[CS350] Running Tests [$DONE/$TOTAL]"
done
done
cd -
cd - > /dev/null
}
os161-container/run_assignment.sh
View file @ 23a948e3
......@@ -9,4 +9,4 @@ source /helpers.sh
/assignments/$1/run.sh $1
/evalaute $LOGS/test_public.log /assignments/$1
PYTHONDONTWRITEBYTECODE=TRUE /evalaute $LOGS/test_public.log /assignments/$1
run.sh
View file @ 23a948e3
#! /bin/sh
rm -rf logs
#!/usr/bin/env bash
export MSYS_NO_PATHCONV=1
realpath() {
[[ $1 = /* ]] && echo "$1" || echo "$PWD/${1#./}"
}
rm -rf "logs"
set -e
if [ "$#" -ne 1 ];then
......@@ -7,12 +12,17 @@ if [ "$#" -ne 1 ];then
exit 1
fi
mkdir -p logs
KERNFILE=$1
mkdir -p "logs"
chmod a+rwx "logs"
KERNFILE="$1"
docker run -it --rm \
-v `realpath $KERNFILE`:/kernel \
-v `realpath os161-container/assignments`:/assignments \
-v `realpath logs`:/logs \
--entrypoint /bin/bash \
-v "`realpath "$KERNFILE"`":/kernel \
-v "`realpath "os161-container/assignments"`":/assignments \
-v "`realpath "logs"`":/logs \
--entrypoint bash \
--name os161 \
os161-runner
docker kill os161 > /dev/null 2> /dev/null
export MSYS_NO_PATHCONV=0