[ltp] help --size of my program too large

[Tod Harter]

Well, its not quite as straightforward as a 'hard limit' on the total size of 
the process. x86 has essentially 4 pointers to memory segments, 1 code, 1 
data, 1 stack, and an 'extra'. Those in turn point to descriptor tables. 
Linux basically simplifies things by only giving you one descriptor table 
entry for each of these 4 areas. So thats one set of limitations, which 
varies from 386 to 686 as to the size of the limit, but the memory management 
code is I suspect basically assuming you have a 586 at best. 

Then as I say there are I/O limitations. DMA channels and ATA on most mother 
boards cannot directly access above 1 gig physical memory, so the act of 
trying to load a code (text) segment of >1 gig in size might well segfault.

I would not be at all surprised to see processes with more than 1 gig of 
mapped address space total, but like with your java process its going to be 
split up between stack, code, and data segments. No one of them is likely 
going to go over 1 gig, and ESPECIALLY nobody ever built the run time loader 
with 1 gig executeable image in mind!!!!!

On Tuesday 01 July 2003 06:58 pm, Alexander Gran wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I'm sure linux can handle more than 1GB for one process, I've got java
> runnig with up to 1.6GB here, more isn't possible, though(I've got no idea
> why, but increasing the maximum heap to more than 1.6GB isn't possible,
> even on system with 2GB ram. SGIs Irix has(had? haven't tried recent
> versions) the same problem, so it may be a java issue). However, I do not
> see a reason for having such large exceutables, Why don't you just load the
> data via normal IO?
>
> Alex
>
> Am Mittwoch, 2. Juli 2003 00:24 schrieb Tod Harter:
> > Ouch!
> >
> > I'd have to go back and stare at my x86 assembler manuals and whatnot for
> > a while to be sure, but I believe you are simply running into a
> > fundamental limitation of memory mapping here. That is to say that x86
> > simply cannot map a single segment that is larger than 1G... At least not
> > the way Linux's memory management is designed. Maybe there are compiler
> > options that let Linux load a larger than 1gig ELF into a code segment,
> > but just as a guess, probably not...
> >
> > In my wildest dreams I have never heard of a 1 GIG executeable before!
> > How could you have such a huge text segment? I can only suggest a
> > redesign of the program so it loads WHATEVER is in that memory AFTER it
> > begins to run, in which case it will be in a different segment. Even then
> > I'm highly skeptical that Linux on x86 can deal with a single process
> > requiring in excess of a gig of address space to be mapped, period.
> >
> > There are also a host of other potential pitfalls, like I/O to addresses
> > over 1 gig requires special workarounds, and it may be that the kernel
> > simply can't load across to the 1 gig physical memory barrier.
> >
> > Even assuming you CAN find the reason for this, I suspect it will be no
> > end of problems. I think you will just have to segment up the problem
> > into smaller chunks and process it in stages.
> >
> > On Tuesday 01 July 2003 02:56 pm, George Jiang wrote:
> > > Hi ,  sorry to put this message here. But I know there are a lot of
> > > experts on Linux here, and I don't know other pertinent linux
> > > newsletters. Thanks.
> > >
> > > --- Steps ---
> > > During my research I have to use a Fortran program for scientific
> > > computation.
> > > It deals with a large model.
> > > first, I use "f77 aa.f -o aa.exe" to compile it;
> > > second, I use "size aa.exe" to view the size of the exe file;
> > > third, I use "./aa.exe" to run the program.
> > >
> > > --- Problem ---
> > > When the size is larger than 1.0 G, I got the message "Segmentation
> > > fault (cor
> > > e dumped)"  when I run "./aa.exe";
> > > Then I adjusted the model size, and redo step1,2,3. This time the
> > > program runs
> > > well.
> > >
> > > --- Question ---
> > > My computer has 1.5G memory; does anyone know how to fix this problem
> > > so that I can run the program with a larger model (my research requires
> > > me to run a large model)?  such a
> > > s modify the configuration or settings of the Linux system... ( red hat
> > > linux 8.0
> > > )?
> > > ( I've thought about MPI; But it's not easy to modify the program for
> > > parralel
> > > computing, so I want to find another solution)
> > >
> > > Thanks a lot!!
> > >    George Jiang
> > >
> > > _________________________________________________________________
> > > STOP MORE SPAM with the new MSN 8 and get 2 months FREE*
> > > http://join.msn.com/?page=features/junkmail
> >
> > --
> > Tod Harter
> > Giant Electronic Brain
> > http://www.giantelectronicbrain.com
>
> - --
> Some operating systems are called `user friendly',
> 	Linux however is `expert friendly'.
> Encrypted Mails welcome. Send spam to toZodiac@gmx.net, please.
> PGP-Key at http://zodiac.dnsalias.org/misc/pgpkey.asc | Key-ID: 0x6D7DD291
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.1 (GNU/Linux)
>
> iD8DBQE/AhIz/aHb+2190pERAjFnAJ9bT7kG5HDyhHdadmV8RgtlOR3K2QCeIZ3S
> v9NnYJd8kP7jz8NA+/xXoUs=
> =X/s6
> -----END PGP SIGNATURE-----

-- 
Tod Harter
Giant Electronic Brain
http://www.giantelectronicbrain.com