University of Otago researchers join the international high-speed data superhighway

Researchers already influencing global leaders have now started another world-leading project thanks to cutting-edge technological advances at the University of Otago.

None of the country’s commercial services could easily provide the service that the Dunedin Multidisciplinary Health & Development Study Research Unit needed to send MRI scan images securely at high speeds to a partner laboratory at Duke University in the United States, for analysis.

But the unit – led by Professor Richie Poulton – had been dreaming about including neuroimaging in its research for more than a decade and had secured the funding.

So, early last year, the researchers started planning to scan the brains of about 1000 New Zealanders they have been studying since their births more than 40 years ago.

From megabits to eternity

Even though the University was struggling to send data off campus faster than about three hundred megabits a second, Information Technology Services staff were determined to make the project work.

Otago researchers had been resorting to couriers and external hard drives because the University’s security and bandwidth controls were slowing down big data transfers.

Stupendous study

The research unit – also known as the Dunedin Study – is so respected, and the worldwide interest in its findings so acute, it has attracted more than $18 million in off-shore funding over the years.

The researchers have produced a paper for a scientific journal every 13 days on average for more than 40 years, providing a treasure trove of information about human health and well-being.

The researchers’ finding that most youths grow out of crime even sparked a law change in the United States, so juveniles can no longer be sentenced to death, saving the lives of more than 70 young people then on death row.

Duke University’s Professor Ahmad Hariri says the study’s next step – analysing study member’s brain scans while also referring to their life histories – will help understand how differences in brain structure and function could reflect people’s past experiences and predict their mental health.

Ducks in a row

The University of Otago partnered with Pacific Radiology to obtain a powerful 3 Tesla Siemens MRI scanner and have professional help running it, says David Maclaurin – University of Otago Information Technology Services (ITS) Infrastructure and Applications, Systems Services Manager.

To create the supporting IT infrastructure, the University’s Brain Research New Zealand Co-Director, Professor Cliff Abraham, formed a project team that met every few weeks for about eight months. It included people from ITS Infrastructure and Applications, Pacific Radiology and subcontractors.

Attribution: 

Starting out … (from left) Professor Richie Poulton, Professor Cliff Abraham, and Pacific Radiology Otago-Southland Managing Partner Dr James Fulton with the new MRI scanner before it went live.

Securely Bridging Networks

They had to make sure only the appropriate scan images flowed between Pacific Radiology and the University, to protect the privacy of both the University’s study members and the company’s patients.

Designing and planning a network-bridge, servers, storage and firewalls took several months, then getting test scans to flow through them at up to 10 gigabits a second took several more weeks.

Speaking different languages

The team also had to get the MRI scanner and the University’s servers to “speak the same language”.

The scanner used a data exchange protocol called Digital Imaging and Communications in Medicine (DICOM) which the ITS team members had never worked with before.

They phoned the manufacturer and other research groups numerous times to unearth software to communicate with DICOM, and discovered an open source Picture Archiving and Communication System (PACS) server.

Once it was operating in a University Data Centre, the PACS server became visible to the MRI scanner as a DICOM endpoint.

Then, the ITS team taught themselves how to run the server, and discovered the best way to anonymise each scan image and tag it, by working with the researchers and Pacific Radiology.

Massive storage

The ITS team decided to configure the new PACS server so the data could be kept in the University’s High Capacity Storage clusters, which authorised “Dunedin Study” researchers can access easily and securely.

Fortunately, about three years previously, the University had created storage clusters that can hold petabytes of data because researchers needed the capacity.

An on-ramp to the superhighway

But Otago was still struggling to send data off campus faster than about three hundred megabits a second and still had to get the MRI scan images to Duke University’s Laboratory of NeuroGenetics.

To create a new, fast, data pathway, the ITS team members started doing their own research and discovered an architecture known as a ‘Science DMZ’, from the Energy Sciences Network (ESnet).

Demilitarised zone

The Science DMZ is a demilitarised zone between the edges of two networks, specifically designed to let big data sets flow in and out with the fewest obstructions, at the highest network speeds possible.

To create a Science DMZ from a combination of new hardware called a network managed edge and several types of software, the ITS team sought help from the Research and Education Advanced Network New Zealand (REANNZ). 

It says the Science DMZ gives Otago security similar to office buildings with concierges: “They’ve seen you before and they know to let you in … we think of them as on-ramps to the superhighway.”

Now, everyday University internet traffic can still can arrive and leave with the usual checks, while authorised research data bypasses any checks it does not need.

Collaborating with NeSI

But ITS team members still needed an ‘engine’ to drive the transfer of large volumes of data at continuously high speeds so sought more help, from the New Zealand eScience Infrastructure (NeSI).

Mr Maclaurin and the equivalent of 1.5 FTE of his team work part-time for both the University of Otago and NeSI, and the University is a NeSI collaborator.

An engine to drive data transfers

The ITS team worked with NeSI staff at the National Institute of Water and Atmospheric Research to create a secure Data Transfer Service with Data Transfer Node software – provided by Globus – which has not been combined with Science DMZ in New Zealand before.

NeSI helped create the blueprint for the service, then the ITS team worked out how many servers it needed, how the servers had to be organised in relation to each other, and internally – which included gauging how much data they could each transfer and at what speeds.

Mr Maclaurin says the software can “hurl huge volumes of data down the network ‘pipeline’ in one go, by transferring many files at once, and by splitting files into a number of parts that can all be streamed simultaneously then reassembled at the other end.”

NeSI relationship vital

Otago would not have gone down that route without NeSI; its help was critical.

“NeSI provides its partners with access to best High Performance Computing practices from around the world; and with the concepts, ideas and practical help to make these sorts of research IT services a reality,” Mr Maclaurin says.

“Research is very near to my heart and it’s what makes a University special, and I’m determined that by collaborating with partners like NeSI, we’re going to deliver the best IT services we can for our researchers.”

The University also used NeSI’s Globus encryption option to keep the ‘Dunedin Study’ data confidential on its way to the United States.

Duke out at Otago

At Duke University, a receiver for the scan images – also based on Globus software – was installed by Research Project Manager for the Dunedin Brain Imaging Study Annchen Knodt, with help from New Zealand.

While the programmer and analyst’s end point at Duke had less capacity and speed than Otago’s, it could do the job.

When Miss Knodt arrived in Dunedin for six months, she was ecstatic the IT infrastructure had already been created by the ITS team, NeSI and REANNZ.

“I was expecting a whole lot of stress, moving across the world and getting things set up, and I didn’t have to do that. The pipelines were in place.”

Attribution: 

Figure 1. Courtesy Dave Maclaurin - The overall MRI imaging data pipeline design.

Perfecting scan images

She could concentrate on her main task; training “Dunedin Study” researchers to perform the scans, which include study members playing games in the scanner that are designed to highlight how specific areas of their brain are functioning – taking about 70 minutes for each person.

One game involves the possibility of winning money if study members respond fast enough, sparking activity in the reward centre of their brain, showing how each person’s response varies. That information is combined with each person’s life history to investigate how the variations in responses are reflected in other aspects of their personalities.

The results

The study’s scan images started reaching the United States about two months ago. So far, about 116 study members have been scanned and the researchers hope to scan all study members every five years, to find out what has changed or stayed the same, and how changes are reflected in people’s health and wellbeing.

Attribution: 

3T MRI diffusion scan of connecting axon pathways

3T MRI diffusion scan of connecting axon pathways

Health checks

Meanwhile, the quality of the scan data is regularly monitored by New Zealand Brain Research Institute medical physicist Dr Tracy Melzer, in Christchurch.

He helped set up the scanner’s parameters, and examines it remotely from Christchurch regularly to ensure its images maintain a consistent quality.

Success at last

Now the data pipeline is in place and working, Mr Maclaurin knows being a research-intensive organisation at the bottom of the world is not a problem anymore: “We can transfer research data up to 15 times faster.”

Scan images direct from the MRI can even reach Duke University in well under three minutes if necessary.

The flow on

Since the data transfer system was developed, more than 250 terabytes of data – about 15 times the University’s total centralised storage in 2010 – have been securely transferred around the country and the world for a range of Otago projects, at speeds of between two and five gigabits a second.

Mr Maclaurin believes that would not have been possible without NeSI and REANNZ.

Attribution: 

David Maclaurin, ITS & NeSI Otago.

“This is the kind of direction it’s going in. Both these organisations are pushing the boundaries and know about the technology all around the world and they’re sowing the seeds and people like us are running with it – it’s fun and very rewarding,” he says.

The future

Now, Mr Maclaurin and his team are trying to get even more speed out of the system, and are still working closely with the Dunedin Study, because “it’s no good just saying ‘here’s a thing, see you around’. It has to be a partnership”.