Scipediacontent: Scipediacontent moved page Draft Content 173483820 to Plassard et al 2017a

2021-02-01T18:34:22Z

Scipediacontent moved page Draft Content 173483820 to Plassard et al 2017a

Scipediacontent: Created page with " == Abstract == Large scale image processing demands a standardized way of not only storage but also a method for job distribution and scheduling. The eXtensible Neuroimaging..."

2021-02-01T18:34:18Z

Created page with " == Abstract == Large scale image processing demands a standardized way of not only storage but also a method for job distribution and scheduling. The eXtensible Neuroimaging..."

New page

== Abstract ==

Large scale image processing demands a standardized way of not only storage but also a method for job distribution and scheduling. The eXtensible Neuroimaging Archive Toolkit (XNAT) is one of several platforms that seeks to solve the storage issues. Distributed Automation for XNAT (DAX) is a job control and distribution manager. Recent massive data projects have revealed several bottlenecks for projects with >100,000 assessors (i.e., data processing pipelines in XNAT). In order to address these concerns, we have developed a new API, which exposes a direct connection to the database rather than REST API calls to accomplish the generation of assessors. This method, consistent with XNAT, keeps a full history for auditing purposes. Additionally, we have optimized DAX to keep track of processing status on disk (called DISKQ) rather than on XNAT, which greatly reduces load on XNAT by vastly dropping the number of API calls. Finally, we have integrated DAX into a Docker container with the idea of using it as a Docker controller to launch Docker containers of image processing pipelines. Using our new API, we reduced the time to create 1,000 assessors (a sub-cohort of our case project) from 65040 seconds to 229 seconds (a decrease of over 270 fold). DISKQ, using pyXnat, allows launching of 400 jobs in under 10 seconds which previously took 2,000 seconds. Together these updates position DAX to support projects with hundreds of thousands of scans and to run them in a time-efficient manner.

== Original document ==

The different versions of the original document can be found in:

* [http://dx.doi.org/10.1117/12.2254371 http://dx.doi.org/10.1117/12.2254371]

* [http://europepmc.org/articles/pmc5596878?pdf=render http://europepmc.org/articles/pmc5596878?pdf=render]

* [https://dx.doi.org/10.1117/12.2254371 https://dx.doi.org/10.1117/12.2254371]

* [https://europepmc.org/articles/PMC5596878 https://europepmc.org/articles/PMC5596878]

* [https://pubmed.ncbi.nlm.nih.gov/28919661 https://pubmed.ncbi.nlm.nih.gov/28919661],
: [https://www.ncbi.nlm.nih.gov/pubmed/28919661 https://www.ncbi.nlm.nih.gov/pubmed/28919661],
: [https://europepmc.org/articles/PMC5596878 https://europepmc.org/articles/PMC5596878],
: [http://ui.adsabs.harvard.edu/abs/2017SPIE10138E..0CD/abstract http://ui.adsabs.harvard.edu/abs/2017SPIE10138E..0CD/abstract],
: [https://www.spiedigitallibrary.org/redirect/proceedings/proceeding?doi=10.1117/12.2254371 https://www.spiedigitallibrary.org/redirect/proceedings/proceeding?doi=10.1117/12.2254371],
: [https://academic.microsoft.com/#/detail/2595498846 https://academic.microsoft.com/#/detail/2595498846]

← Older revision	Revision as of 18:34, 1 February 2021
(No difference)

Plassard et al 2017a - Revision history

Scipediacontent: Scipediacontent moved page Draft Content 173483820 to Plassard et al 2017a

Scipediacontent: Created page with " == Abstract == Large scale image processing demands a standardized way of not only storage but also a method for job distribution and scheduling. The eXtensible Neuroimaging..."