Message: CSV overwriting/interrupting lines in multithreaded operation Not Logged In (login)
 Next-in-Thread Next-in-Thread
 Next-in-Forum Next-in-Forum

Question CSV overwriting/interrupting lines in multithreaded operation 

Forum: Analysis
Date: 20 Jan, 2017
From: Nicole Ackerman <Nicole Ackerman>

I have written a Geant4 program that is using CSV output (via Analysis). A colleague is running the program on a cluster and is occasionally seeing problematic output. We are seeing errors like these once in one million events, roughly.

One problematic CSV line looked like: 1,47,alp118,14,e-,21.2086,1.16726,1.16726,msc,0.00253289,2.6746,-0.000918663,5.66533,e-,14,-918.663,406.852,0,2,Bi212[115.183],e-,RadioactiveDaughter,24.6569

I think the "correct" line would look like: 118,14,e-,21.2086,1.16726,1.16726,msc,0.00253289,2.6746,-0.000918663,5.66533,e-,14,-918.663,406.852,0,2,Bi212[115.183],e-,RadioactiveDaughter,24.6569 where it has interuppted or overwritten a line that began 1,47,alpha,...

The interruption seems to occur in different places in the line when the problem occurs. My colleague has not reported seeing any output warnings or errors, and otherwise the program appears to be working fine.

My hypothesis is that the harddrive isn't keeping up with the output. I do not know the details of the cluster, but I think there are multiple machines and multiple cores. While each thread (typically 15) is writing to its own file, there is a shared filesystem and I do not know how many of the threads are working on the same physical drive.

I've exhausted my knowledge of parallel computing, for either diagnosing or fixing this problem. The problem is complicated by my lack of access to my colleague's computing system and her lack of knowledge of Geant4. Has anyone seen a similar problem? I would appreciate any recommendations for debugging.

Inline Depth:
 1 1
 All All
Outline Depth:
 1 1
 2 2
 All All
Add message: (add)

1 None: Re: CSV overwriting/interrupting lines in multithreaded operation   (Guy Barrand - 20 Jan, 2017)
2 None: Re: CSV overwriting/interrupting lines in multithreaded operation   (Ivana Hrivnacova - 20 Jan, 2017)
1 None: Re: CSV overwriting/interrupting lines in multithreaded operation   (Nicole Ackerman - 20 Jan, 2017)
(_ None: Re: CSV overwriting/interrupting lines in multithreaded operation   (Ivana Hrivnacova - 20 Jan, 2017)
(_ None: Re: CSV overwriting/interrupting lines in multithreaded operation   (Nicole Ackerman - 20 Jan, 2017)
(_ None: Re: CSV overwriting/interrupting lines in multithreaded operation   (Ivana Hrivnacova - 23 Jan, 2017)
(_ None: Re: CSV overwriting/interrupting lines in multithreaded operation   (Nicole Ackerman - 27 Jan, 2017)
3 Ok: Re: CSV overwriting/interrupting lines in multithreaded operation   (Nicole Ackerman - 28 Feb, 2017)
 Add Message Add Message
to: "CSV overwriting/interrupting lines in multithreaded operation"

 Subscribe Subscribe

This site runs SLAC HyperNews version 1.11-slac-98, derived from the original HyperNews


[ Geant 4 Home | Geant 4 HyperNews | Search | Request New Forum | Feedback ]