The intention of hosting automated grand challenges here is to make the scoring and ranking open and transparent. The Midas Platform, the Challenge plugin for hosting grand challenges on Midas, and the Covalic metric executables used for scoring results are all open source and freely available. Bug reports and comments are welcome via email at email@example.com.
Grand challenges are a way to compare algorithms on a common dataset. The challenge will consist of one or more phases, the phases can either occur offsite (all participants are not in the same location) or onsite (all participants are located together, such as at a conference or meeting). A phase of a challenge can either be for competitors to see only their own scores on a dataset (and how their scores change as they make changes to their methods), or it can have a global dashboard where competitors are ranked against each other based on some scoring and ranking metrics.
A common structure is to have a training phase with a dataset for competitors to test their algorithms against, and then a testing phase with a separate dataset and a global dashboard to rank competitors against each other. There are many available metrics to score results, similarly there are many methods to rank competitors based on their metric scores. Each challenge can decide on their own specific metrics and ranking schemes.
At the high level, the way automated grand challenge hosting works is that there is a set of input data cases with a ground truth for each case. The ground truth may be kept private from the competitors or shared with them. Competitors will submit a result for each case in the input data. The Midas Grand Challenge system will run the submitted result case against the ground truth for that case, calculating whatever metrics have been decided. These scoring results are then presented to the competitors. If the challenge is in a phase that involves a global dashboard, the rankings are calculated for each of the competitors and a global dashboard with the rankings is displayed.
Each grand challenge will be associated with one (or possibly many) Communities in Midas. Identify the specific community for your Grand Challenge, and then look to that Community's info tab for more specific information. The following instructions should apply generally to all grand challenges.
You may also upload a zip file containing your submissions.
In order to submit results, click on the Join The Community button for the correct community, this will create some specifically named data folders in your user's space. If these folders have not been created, navigate to the community page, click on the leave the community link, and then join the community again. The folders that will have been created are in the My folders link to the left (which leads to your Midas user's space). You will have to upload your results in the appropriate folder for the phase of the challenge. These folders are created with permissions such that you can read/edit/delete the files there, and moderators of the challenge can view these files, but other competitors will not be able to read/edit/delete these files.
You will need to match your results to the subject via a naming convention. For example, in the BRATS-1 Training Dataset, Truth Folder, you can see there is a file corresponding to subject BRATS_HG0001 called BRATS_HG0001_truth.mha. The input data for this subject BRATS_HG0001 is in the BRATS-1 Images subfolder. To submit a result for this subject, for the training data, you would navigate to your Midas user's submission folder for the BRATS-1 training phase, and upload your results file (which should be named BRATS_HG001_result.mha) in that folder.
Once you are satisfied with your current submission, you can have your submissions scored. Whether you can resubmit results later for further scoring or score improvement depends on the setup of that phase of the challenge. Check your challenge for details.
Results folder validation will then occur, and will report to you which of your results files in the Submission Folder will be matched up with subject truths from the Truth folder. This will give you an opportunity to correct the names of any of your submission files, re-upload them to your submission folder, and re-validate them. As long as there is at least one match between your submitted results and the subjects in the Truth folder, you can score your results by clicking on the Score Results button. After clicking this button, you will be taken to a dashboard display of your current scoring. The scoring will be processed via a batch system and will take some time. As the results come in, your scoring dashboard will be updated. You will receive an email at the email address you registered with when your scoring is complete.
To see a listing of previous runs, click on My folders at the left, then click on the My challenge scores tab. You will see a listing of previous scoring runs, clicking on the timestamp of any of those will take you to the detailed scoring for that results submission.
If your question is not answered in the FAQ, you can look in the info tab of the Community associated with your challenge for a support email specific to that challenge. For questions in general about Midas hosting of grand challenges, please email firstname.lastname@example.org.
What is the difference between Community space and My folders?
Midas creates hierarchies of data specific to a particular task or group of people under the name Community. This data is intended to be shared among those with permissions to see it. For individual users, Midas creates a My folders area, accessible via a link at the left, where individual users can store their own folders and items under whatever permission scheme they think appropriate.
The Score Submissions wizard is not finding my folder by default.
It is possible you have deleted your submission folder or that it was not created correctly. If you leave the community associated with your challenge and re-join it, you will have the proper folders created in your My folders area, then you can upload your results to these newly created folders. If you have deleted these folders, do not try to manually recreate them with the same names, as this will prevent automated creation of folders with the same names in the same locations when you rejoin the community.
How can I open the mha file format?
Can I have my results separated into header and data files?
No, you must combine the header and data of each case into a single file.