Organizing Tasks¶

Upload Public Baselines

In the best case, you provide the code, a published docker image, and instructions on how to compile the code into a docker image to simplify participation in your shared tasks.

We have some examples on baselines that you can adopt for your shared task, e.g.:

The clickbait spoiling baselines from SemEval-2023
The ValueEval baseline from SemEval-2023
The baselines for Touché-2023 at CLEF
The baseline for PAN-2023 at CLEF

To simplify testing software submissions locally before they are uploaded to TIRA, we provide a tira-run command that participants can use to test their image locally. The tira-run commands executes a software as it would be executed in TIRA.

You can find some examples of shared tasks that use tira-run in their baselines to simplify participation here:

We recommend that you have for all of your baselines an tira-run example so that you can point participants to this example if their software submission fails in TIRA. For this, you need:

The baseline should be publicly available, e.g., at Dockerhub
You need a small sample of the data (can be artificial, but must have the same structure/format as the test data)

Assuming that you have published the baseline for your task at dockerhub under the name <DOCKER_IMAGE_BASELINE> and that the baseline is executed via the command <BASELINE_SOFTWARE_COMMAND> inside the container and that you have a directory tira-sample-input with the sample data in the git repository of your baseline, you can add the following documentation (replace the placeholders with the correct values) to your README:

You can test docker images that you would submit to TIRA locally via `tira-run`.
The `tira-run` commands executes a software as it would be executed in TIRA (i.e., with sandboxing using the same command pattern).

We recommend that you test your software locally on the sample dataset `tira-sample-input` before uploading it to TIRA to ensure that your software works correctly (this also simplifies debugging as everything is under your control and runs on your machine).

Please install `tira-run` via `pip3 install tira`.

After `tira-run` is installed, you can execute the baseline on the sample dataset `tira-sample-input` via this command:
`tira-run --input-directory ${PWD}/tira-sample-input --image <DOCKER_IMAGE_BASELINE> --command <BASELINE_SOFTWARE_COMMAND>`

This command should create the following output in `tira-output`: <TODO_ADD_EXAMPLE_OUTPUT>

Note

This is the previous version

Modifying virtual machines¶

Login to the machine where the VM exists (ssh tira@betawebXYZ`)
Locate the complete [vmName] from tira/model/users/[users].prototext.

RAM

On dockerized tira hosts: enter the docker container. You can do this using:

docker exec -ti $(docker ps|grep 'tira-io/tira-host'|awk '{print $1}') bash

Then run:

tira vm-shutdown [userId]
VBoxManage modifyvm [vmName] --memory [MBs]
tira vm-start [userId]

CPUs

On dockerized tira hosts: enter the docker container. You can do this using:

docker exec -ti $(docker ps|grep 'tira-io/tira-host'|awk '{print $1}') bash

Then run:

tira vm-shutdown [userId]
VBoxManage modifyvm [vmName] --cpus [number]
tira vm-start [userId]

HDD space (read/write from VM, is sandboxed along with VM)

On dockerized tira hosts: enter the docker container. You can do this using:

docker exec -ti $(docker ps|grep 'tira-io/tira-host'|awk '{print $1}') bash

Then run:

cd /home/tira/VirtualBox\ VMs/[virtualMachineId]
tira vm-stop $(basename "$PWD")
VBoxManage createhd --filename data.vmdk --format VMDK --size [MBs]
# In the following: if "SATA" does not work, try "SATA Controller" or "SATAController"
VBoxManage storageattach $(basename "$PWD") --storagectl "SATA" --port 1 --type hdd --medium data.vmdk
tira vm-start $(basename "$PWD")
tira vm-ssh $(basename "$PWD")

Use fdisk -l to check that the new partition is indeed “/dev/sdb”. Adjust below instructions otherwise.

sudo parted -s -a optimal /dev/sdb mklabel gpt -- mkpart primary ext4 1 -1
sudo mkfs -t ext4 /dev/sdb1
sudo mkdir /mnt/data
sudo nano /etc/fstab</code> and add <code>/dev/sdb1 /mnt/data ext4 defaults 0 2
sudo mount /dev/sdb1

HDD space (read-only from VM, is not sandboxed)

Follow these instructions. Then run

tira vm-shutdown [userId]
VBoxManage sharedfolder add [virtualMachineId] --name data --hostpath /home/[userId]/data --readonly --automount
tira vm-start [userId]

Note

This allows connections by SFTP, but not by SSH (as the home directory is not writable)

Moderate a Task:¶

Log in to tira.io with a reviewer account.
(optional) Add a new organizer using Add organizer on the website.
Create a master-vm (see Set up the master VM)
Reload the VMs on the Admin Panel on the website.
Add a new task using Add Task on the website. Requires a master-vm
Add a new dataset using Add Dataset on the page of the respective task. You can also add the evaluator data during this step.
Install the evaluator on the master-vm in accordance to the data entered during step. 6

Set up the master VM¶

Create the virtual machine using tira vm-create. The name should end with -master.

Connect to the TIRA host container:

docker exec -ti $(docker ps | grep 'tira-io/tira-host' | awk '{print $1}') bash

Give yourself permission to the VM’s group on tira.io (if you followed [the instructions](#get-a-reviewer-account))
Give the master VM access to the test and truth directories.
```
vboxmanage sharedfolder add [virtualMachineId] --name [typeDirectory] --hostpath [typeDirectoryPath] --readonly --automount
```
This should be done for training-datasets-truth, test-datasets, and test-datasets-truth

Organizing Tasks¶

Modifying virtual machines¶

Moderate a Task:¶

Set up the master VM¶

Troubleshooting¶