Q&A with Agah Karakuzu and Nikola Stikov

By Pinar S. Özbay

This MRM Highlights Pick interview is with Agah Karakuzu and Nikola Stikov, researchers at the NeuroPoly lab at Polytechnique Montreal. Their paper is entitled “Vendor-neutral sequences and fully transparent workflows improve inter-vendor reproducibility of quantitative MRI”. Their article was chosen as this month’s Highlights pick because it demonstrated exemplary reproducible research practices; specifically, they shared open-source pulse sequences, a Docker image, BIDS-compliant data, and a beautiful interactive Jupyter Book.

MRMH: Could you briefly tell us about your backgrounds and how you got into the field of MRI?

Agah: I studied biomedical engineering for my undergraduate degree in Turkey, and at that time it was really difficult to get my hands on any kind of MR images online. I had to jump through a lot of hoops to find some, but when I did, I managed to write MATLAB code to segment MS lesions on FLAIR images for my undergraduate dissertation. I presented that work at the Turkish Society of Magnetic Resonance conference in Ankara in 2012. I followed that up with an MSc at Bogazici University where I developed a multimodal MRI method for in vivo musculoskeletal mechanics. This led to another poster presentation, this time at the 2014 MR Balkan conference, which was organized by Nikola. Little did I know that that poster would be my ticket to Canada, where I did my PhD with Nikola and with whom I am currently doing a post-doc as well. My focus and passion since arriving in Montreal has been tackling the problem of reproducibility in qMRI research, from scanner to publication. The work presented in this article reflects this passion.

Nikola: I’ve also been in the field of MRI for quite a while. I started with my undergraduate at Stanford, and stayed there for a master’s. During that time, I got to meet John Pauly, who became my PhD supervisor. It was towards the end of my PhD that I decided to organize a conference for MRI researchers in the Balkans. We organized the first two meetings in Macedonia (2008 and 2011), and then the third one was in Turkey in 2014, which is where I met Agah. Two years later he joined my lab, and since then we have been working together. And as Al Macovski (the PhD advisor of my PhD advisor) once said, you just need to find good students and then get out of their way. When you get out of Agah’s way he accomplishes a lot, so I’m very happy that he’s still with me getting things done in the lab.

MRMH: Could you share the story behind this paper, and tell us how the idea evolved?

Agah: The story begins with qMRLab, which is an open-source toolbox for quantitative MRI analysis and simulation. During my second year of involvement with qMRLab, we brought it into the cloud so that it could be integrated into any data processing workflow. This development got me excited about the idea of trying various projects at hackathons in Canada and Europe, such as the Brainhack in Montreal, and OpenMRI in the Benelux countries.

It was during these hackathons that I realized that there is no common data standard that would make it possible to perform comparative quantitative MRI studies. So, I started working on qMRI BIDS (BEP001), a BIDS (brain imaging data structure) extension proposal. It took many years to get a consensus for it, but now it is in active use and facilitates the sharing of quantitative MRI data, and it also enables multiple software packages, developed in different languages, to use the same input-output conventions for easier data workflow development.

Later, I wanted to find more ways to make publications reproducible, but one problem we encountered was that we didn’t know where to publish these reproducible research objects. This led to the creation of another project that I became an active developer on. It is called NeuroLibre, and it is an initiative of the Canadian Open Neuroscience Platform (CONP). My idea was to create a scanner to publication platform, but in this case, the missing component was the acquisition step, so for that we turned to the RTHawk real-time imaging system. My hypothesis was that if we could standardize the pulse sequences across vendors before the measurement stage, we could get rid of the variability sources that are not easy to identify. On this basis, I developed a vendor-neutral 3D SPGR sequence and magnetization transfer preparation block to acquire T₁, MTR, and MTsat maps. And then I connected these ad hoc applications with qMRLab and other software packages using qMRI-BIDS. And that’s how the VENUS workflows came to existence. 

Nikola: I think this article touches on a central theme of the work that we do in the lab, which stems from our recognition that there are problems with quantitative MRI, and MRI in general. The solutions we’ve focused on developing are related to standards, workflows, sharing code, and making it easier for people to reuse openly available tools. What the VENUS project does is bring quantitative MRI under one umbrella. That is, it makes it possible to open up the scanner, deploy an open-source sequence on that system, and do all of the processing using an open-source tool. And it’s also important to note that if you read the description of the Siemens scanner, for example, you’ll actually find it says “this scanner is not a measurement device”. So, what we are trying to do is bring measurements back into MRI by standardizing measurement techniques and their quantities.

MRMH: What was the biggest challenge in the implementation, both sequence-wise and vendor-wise?

Agah: I guess the biggest challenge was that we were trying to run an experiment on three different scanners in two different cities, Montreal and Toronto, in the middle of a pandemic. But my main problem, in general, when I’m developing any kind of software is the lack of documentation, or a good application programming interface design of the programming environment. But that wasn’t the case this time for the pulse sequence development; actually, it went quite smoothly thanks to Spin Bench and RTHawk. Another challenge was making sure that the quantitative MRI protocols can be properly executed on different vendors’ systems that come with different interfaces and result in different data outputs. Actually experiencing these problems was crucial in helping us to address them through this project. As a result, now, with VENUS, for example, we can export data in the standardized qMRI-BIDS format, and it has a unified user interface.

Nikola: I would like to address the whole administrative/regulatory side of things, because here we are, you know, trying to open up a system that is inherently closed and driven by IP. And I’m very grateful that both GE and Siemens understand what we’re trying to accomplish. Because it’s not always easy to use open-source MRI with a closed-source black box system. 

MRMH: Could you briefly discuss the outcome?

Agah: We made it so that the outcome of this work will be the most visited section of the article, and the outcome is summed up in its title: Vendor-neutral sequences and fully transparent workflows improve inter-vendor reproducibility of quantitative MRI. What we are saying is we cannot rely on black box implementations to achieve precise measurements across different centers. And we showed significant reduction in non-biological variants with this approach, and this brings us closer to capturing true biological variability in quantitative MRI applications. Another important outcome is that everything from pulse sequences to statistical analyses is on GitHub at qMRLab.org/VENUS. 

Nikola: Just to put it in context, variability in quantitative MRI is very big. If you think about T₁ at 3 Tesla in white matter, it should be a very, very sharp peak. When you implement different protocols, you get variability in T₁ that goes from 700 to 1100 ms. We ran a T₁ mapping challenge to show that even when you follow the same protocol, the values will change between sites. We observed this kind of variability (even though it was the same subject), between three scanners from different vendors, and with VENUS, we saw that variability dramatically decreased. And I think with our paper, we managed to kind of hint at what is causing this. 

MRMH: Do you have any ideas on how to improve the current workflow? 

Agah: We achieved vendor neutrality, but we should also aim to achieve scanner upgrade immunity. Because, most longitudinal data collections that run over years are affected by at least one scanner upgrade. And in the context of clinical trials, this is a big cost in terms of developing quantitative biomarkers. The idea is usually to use a single scanner so that the biomarkers better capture the effect size, but then the scanner gets upgraded, and the measurements can change by a considerable amount. And lastly, the benefits of vendor neutrality are not limited to quantitative applications; we are also working on an integration of the Spinal Cord Toolbox (SCT), which is another open-source software, developed by our lab and led by Julien Cohen-Adad, and together with a new member of our lab, Nadia Blostein, we are trying to perform robust spinal cord cross sectional area (CSA) measurements with VENUS. 

Nikola: In our lab, we spend a lot of time on outreach, as we think this is the only way to make sure that the work is properly vetted, and can also then be reused. And I think that we could be doing a better job of convincing people to work collaboratively rather than in silos. Because while all of us are good engineers, good physicists, who can do very important work, improving our own workflows, it’s much more challenging to get everybody to work collaboratively on a project, and make sure that the same project is installed at several sites. And yet, for clinical trials, this is a big money saver, while for research, it is a big timesaver.

MRMH: Anything to add?

Nikola: I have to say it feels good to be on the other side of the Highlights interview. It’s really nice that this initiative has continued for 7 years since Erika Raven and I first started it.