Message: Re: superslow make on a cluster over the network Not Logged In (login)
 Next-in-Thread Next-in-Thread
 Next-in-Forum Next-in-Forum

None Re: superslow make on a cluster over the network 

Forum: Installation and Configuration
Re: Sad superslow make on a cluster over the network (Valery Taranenko)
Re: None Re: superslow make on a cluster over the network (Ben Morgan)
Re: Idea Re: superslow make on a cluster over the network (Valery Taranenko)
Date: 05 Nov, 2009
From: Ben Morgan <Ben Morgan>

Hi Valery
On Thu, 05 Nov 2009 01:34:49 GMT, Valery Taranenko wrote:
> i'm not familiar with intricate g4 make system, but if i look into the
> user code intermediate make-process files in
> $G4WORKDIR/tmp/Linux/usercode I see those dependency .d files where all
> header files of g4 as well as the OS(!) are listed. some of those files
> hold 500 or even 800 dependencies. now, imagine, how make build those.
> it must be opening each source or header file, parsing for includes,
> open those includes and so on and so on. one poor g4 user code analysis
> file has 500 dependencies--pretty crazy. this sounds like an overkill.
> the bottleneck must be I/O on the disk system when make process
> constantly opens hundreds of those files while building dependencies.

O.k., if your filesystem is completely remote (i.e. even the system parts are network mounted) then I can see why there would be a problem here. However, I don't think the number of dependencies you're seeing is completely crazy. Even Automake/CMake will generate similar sized ".d" files - I've certainly seen this even in non-Geant4 pure C++ projects.

I'm not aware of any improvement that can be made here (doesn't mean there isn't one though)- as you've found it's the remote filesystem that's the cause of the problem. Once you've done the initial big build, the future incremental changes should run faster.

Cheers,

Ben.

 Add Message Add Message
to: "Re: superslow make on a cluster over the network"

 Subscribe Subscribe

This site runs SLAC HyperNews version 1.11-slac-98, derived from the original HyperNews


[ Geant 4 Home | Geant 4 HyperNews | Search | Request New Forum | Feedback ]