Message: superslow make on a cluster over the network Not Logged In (login)
 Next-in-Thread Next-in-Thread
 Next-in-Forum Next-in-Forum

Sad superslow make on a cluster over the network 

Forum: Installation and Configuration
Date: Nov 03, 14:36
From: Valery Taranenko <Valery Taranenko>

it's g4 9.2.p2, normal and clean installation, working fine, the only problem is performance while at the make stage of a user code--see below

normally, on a computer with a local HDD, the make (of a 20 user files) w/ link takes 1 min. now, on a cluster, with a remote disk storage, i have to ssh to a computational node, and then the make takes 30 min or more. the problem is--i suspect--is a network load (I/O). "Making dependency for file ..." takes several min for each file(!). i suspect, make needs to transfer many files over the network (it's Gb Ethernet). not only making dependencies takes long, compilations do too. the make process is awfully slow. i cannot stand it :(

does anybody knows how to bring the speed back to 1 min? i can do the so-called prestaging (copping the files to a local note hdd and run from there) but in this case all paths will be different, it will be basically a new g4 installation, which is too much.

maybe it's possible to hack g4 make such that it will operate on a local (precoppied to a local hdd) files.

i could build the user code on another linux system, but then the libs issues may arise, since these are diff distros.

best valery

Inline Depth:
 1 1
 All All
Outline Depth:
 1 1
 2 2
 All All
Add message: (add)

1 None: Re: superslow make on a cluster over the network   (Ben Morgan - Nov 04, 03:28)
(_ Idea: Re: superslow make on a cluster over the network   (Valery Taranenko - Nov 04, 17:34)
(_ None: Re: superslow make on a cluster over the network   (Ben Morgan - Nov 05, 03:01)
 Add Message Add Message
to: "superslow make on a cluster over the network"

 Subscribe Subscribe

This site runs SLAC HyperNews version 1.11-slac-98, derived from the original HyperNews


[ Geant 4 Home | Geant 4 HyperNews | Search | Request New Forum | Feedback ]