MRM Highlights
All Posts
The QSM reconstruction Challenge was an open competition which was designed to systematically compare and quantitatively assess the many available QSM algorithms. As described by the organizers in a recent publication, the challenge was first announced during the Electro-Magnetic Tissue Properties (EMTP) study group meeting at the 2016 ISMRM in Singapore on May 12, 2016. Data and instructions were made available starting from this day until September 15, 2016. Overall, 27 susceptibility maps from 13 groups were submitted, and the results of the evaluation were presented in the context of the 4th International Workshop on MRI Phase Contrast and Quantitative Susceptibility Mapping, held September 26–28, 2016, at the Medical University of Graz, Austria. The authors mention that the initial goal was to test the ability of various QSM algorithms to recover the underlying susceptibility distribution from a healthy volunteer’s phase data faithfully. And as a side-goal they also wanted to provide a common reference dataset to help benchmarking not only existing QSM algorithms but also methods that would be developed in the future. We set up a teleconference with Drs. Langkammer, Bilgic and Schweser, as the main organizers of the Challenge, and are proud to present the largest crowd (n=5) participating in an MRM Highlights interview so far! We had an exciting discussion about the challenge itself and also the future of the field, while connecting Bethesda, Maryland (Pinar), Boston, Massachusetts (Berkin), Buffalo, New York (Ferdinand), Graz, Austria, (Christian), and Cardiff, Wales (Erika).
MRMH: Could you share the story behind the work with us and how the idea got started?
Ferdinand: I remember vivid discussions about visual quality versus quantitative accuracy and standardization already at the first and second QSM workshops held 2011 and 2013 in Jena (Germany) and Ithaca (New York). Over the years, the validation idea gained more traction. A few years ago, a point was reached at which a pretty high number of different QSM algorithms had been proposed but it was completely unclear how these algorithms should be compared to one another.
Berkin: The trigger for starting the challenge came from Markus Barth who responded to a Call for Suggestions of Pressing Topics to be discussed at the 2015 ISMRM Electro-Magnetic Tissue Properties (EMTP) study group meeting in Toronto. Markus pretty much outlined the general idea of the challenge.
Christian: Later, three of us thought to join our forces and planned it further during a fruitful discussion at a bar [laughs]. After a year of work, we announced the details of the Challenge at the 2016 EMTP study group meeting in Singapore with the goal to present the results later in the year at the 4th QSM Workshop in Graz [see photos below from the Workshop].
MRMH: Could you give us a summary of the Challenge design and analyses?
Berkin: We provided a comprehensive 3T dataset including GRE data from 12 head-orientations, T1-weighted structural images, background-field removed phase, COSMOS (calculation of susceptibility using multiple orientation sampling) and the full susceptibility tensor (ST). Furthermore, two QSM algorithms and their evaluation scripts were provided as Matlab scripts for benchmarking. We decided to provide only a single-orientation phase data from this multi-orientation scan, which was representative of any real-world acquisition we would routinely perform, including noise and flow artifacts etc., rather than a contrived and ideal numerical phantom simulation.
Ferdinand: Another critical step of the design was the decision on the ground-truth susceptibility data for evaluation. In comparison to the single-orientation methods, multiple-orientation approaches do not depend on the regularization or algorithm parameters, while overcoming the ill-posed inverse problem by sampling at different angles with respect to the magnetic field. Hence, we thought multi-angle data would be more suitable for a ground-truth, although we were well aware that there is actually no real ground-truth. We thought that STI data would be the best reference data set, as it also overcomes the anisotropy of the white-matter better than COSMOS, since the physical model of COSMOS assumes an isotropic magnetic susceptibility.
Christian: The participants calculated their QSM maps from the single orientation data and compared them against the STI data using the following metrics: root mean squared error (RMSE), structure similarity index (SSIM), high-frequency error norm (HFEN), and the absolute error in selected white and grey matter regions. While RMSE would serve as a global error metric, with SSIM we aimed to promote images that appear visually similar to the reference data. HFEN was included to penalize over-regularized, smooth susceptibility maps by emphasizing the fidelity of high-frequency edge structures. And lastly, absolute error in ROIs was used to measure the quantitative accuracy of the reconstruction methods.
MRMH: Who were the winner-teams? And what do you think made the winner-algorithms better than the others?
Christian: We were surprised and very happy with the high participation from the QSM community. We received a total of 27 submissions from 13 groups. One winner was selected for each of the 4 categories: Christian Kames (UBC, Vancouver, RMSE category), Li Guo (Southern Medical University, Guangzhou, tied for first place in ROI accuracy), Zhe Liu (Cornell, New York, also in ROI accuracy), and Xu Li (Johns Hopkins, Baltimore, winner of both HFEN and SSIM categories).
Berkin: A common thread between some of the winners was keeping the acquired data in the well-conditioned frequency region intact, while compensating for the ill-conditioned frequency content using compressed sensing (CS)-like algorithms. Some have also utilized structural information from the magnitude image to provide prior information.
Ferdinand: We observed two main approaches among the winners, with MEDI-like (Morphology Enabled Dipole Inversion, based on L1-minimization) algorithms, which gave higher quantitative accuracy (ROI metric), and CS-like approaches, which scored better in other error categories such as structural similarity and high-frequency error norm. We believe that the choice of the QSM algorithm would depend highly on the area of the application.
MRMH: Where do you see the field in 5 years? Do you see QSM as a ‘push button’ method in the scanners?
Christian: Absolutely! This research field has evolved in the last 15-20 years and today we can produce susceptibility maps with next to no apparent artefacts. With the availability of highly accelerated GRE sequences allowing up to 0.5mm isotropic resolution at 7 Tesla as well as at 3 Tesla in a few minutes, QSM has a big potential to become a clinical tool available at the scanners, as SWI made it 10 years ago.
Ferdinand: Just as a remark, there will also be a member-initiated symposium led by Jose Marques at this year’s ISMRM in Paris which will bring together industry representatives at a round table to discuss a commercial implementation.
Berkin: As a last remark referring to the future of the field, we foresee that machine learning [a huge yeay from the audience] will pl
ay an important role in the solution of this difficult reconstruction problem, and look forward to seeing the first batch of such solutions at the upcoming ISMRM meeting in Paris. However, which will be the best algorithm to implement on a scanner as a black-box tool? We think a Challenge 2.0 will be very important to guide this process.
*The dataset is provided at http://qsm.neuroimaging.at/
