'fatal errors ' accessing image files on network renders (RTFM)

Mailing List

Home

Forum Home

Softimage

Carrara

trueSpace

Dir3d-l

Maya - a powerful 3D animation and visual effects software

Macromedia Flash Development

Subjects

•	Cameras
•	scaleDown command
•	black out solved
•	Aircraft Tutorial
•	Mathematical XYZ ?
•	Its done This vs That
•	Its done first week
•	recommendations for screen video captures?
•	3DExplorer "Oddity "
•	New Director
•	ProTeam renewals
•	Fuel 's new websites (X post)
•	Blue peter create a make toy
•	targeting groups question
•	XPost: Shockwave 3D game ( sort of )
•	RES: RES: RES: Fish Modeling
•	Emitting particles from object intersection
•	Fuel 's new websites (X post)
•	Texturing
•	Big Break Contest Videos
•	New Plugins
•	Models and Texture on my updated site
•	Error Installing Patch tS6 6
•	Plasma?
•	Looking for Inspiration
•	Weird EMail Q
•	It 's done first week ?
•	Cherry not cranberry
•	New game
•	Camera Animation Problem
•	Particle plugins?

'fatal errors ' accessing image files on network renders (RTFM)

'fatal errors ' accessing image files on network renders (RTFM)

2004-05-13 - By Ben Kilgore

Back
FFS, RTFM, Ben.

Memory-Mapped Textures
Rendering textures can be done quicker with the use of memory-mapped textures.
Memory-mapped textures are not loaded into memory but are accessed directly
from the disk whenever the shader needs it. XSI recognizes a memory-mappable
texture and automatically maps it. Only the map image-file format (with the
.map extension) can be mapped.

Before a texture can be memory-mapped, a few steps must be taken:

a.. The texture must be converted to .map format. This is done using the imf
_copy standalone. For more information on this tool, see the Standalones guide
on the Documentation CD. The scene file must reference this texture.
b.. In a distributed rendering network, memory-mapped textures must be loaded
onto every machine on the network and the specified path to the texture must be
valid on every machine.
Copy the image file (.map) to every machine's local hard drive.

c.. Memory-mapped textures are considered local by mental ray, as though the
local keyword was used in the scene file.
So, if the scene is rendered on multiple hosts, each one accesses the given
path rather than transfer the scene across the network.

d.. The memory-map should not be on an NFS-mounted file system.
Although it may seem faster to have the texture on all the hosts, the network
transfers make memory-mapping slower than regular texturing.

e.. Memory-mapping is ideal for very large textures containing dozens of
megabytes that are sampled infrequently, since most of the texture file isn't
loaded into memory.
If the texture and the scene are too large to fit into the physical memory,
then loading a texture is equivalent to loading the file into memory,
decompressing it, and copying it out to swap. From that point on, accessing a
texture means accessing the swap.

On the other hand, a frequently accessed texture and scene are smaller and fit
into memory. Memory-mapped textures are slower than regular textures because
the swap isn't used. For more information about memory mapped textures, see
Creating Memory-Mapped Textures in Chapter 5 of the Shaders, Lights & Cameras
guide.

-- -- Original Message -- --
From: "Ben Kilgore" <ben@(protected)>
To: <XSI@(protected)>
Sent: Monday, May 03, 2004 11:57 AM
Subject: Re: 'fatal errors' accessing image files on network renders

> Kim,
>
> Forgot to mention--the farm completed other scenes without a hitch. It just
> seems to be this texture and/or scene that's having this effect.
>
> Ben
>
> -- -- Original Message -- --
> From: "kim aldis" <kim@(protected)>
> To: <XSI@(protected)>
> Sent: Monday, May 03, 2004 3:49 AM
> Subject: RE: 'fatal errors' accessing image files on network renders
>
>
> > Sounds like that could your problem. Localhost connected to the same
> machine
> > will lift a client licence, so you've effectively taken 6 licences. Go to
> > control panel->administrative tools->Computer Management and look under
> > shared folders->sessions, see how many sessions are active. You can close
> > sessions from there.
> >
> > This is one of the many reasons to consider Linux, at least as a file
> > server. The only other way is to buy more MS client license. The fact that
> > Windows does this is bad enough. That it does it so badly and consts som
> > much is outrageous.
> >
> > > -- --Original Message-- --
> > > From: owner-xsi@(protected)
> > > [mailto:owner-xsi@(protected)] On Behalf Of Ben Kilgore
> > > Sent: 02 May 2004 19:44
> > > To: XSI@(protected)
> > > Subject: Re: 'fatal errors' accessing image files on network renders
> > >
> > > It's 2000 Professional.
> > >
> > > 5 machines (10 CPUs), plus the render control (localhost?) is
> > > rendering, as well.
> > >
> > >
> > >
> > >
> > > -- -- Original Message -- --
> > > From: "kim aldis" <kim@(protected)>
> > > To: <XSI@(protected)>
> > > Sent: Sunday, May 02, 2004 2:09 PM
> > > Subject: RE: 'fatal errors' accessing image files on network renders
> > >
> > >
> > > > How many machines do you have trying to access the files on
> > > the remote
> > > > machine? Remember, if you're serving files from NT
> > > Workstation or XP,
> > > > Microsoft limits you to 10 connections only. In practice
> > > this usually
> > > works
> > > > out to be around 6 after you take into consdiration browsr
> > > connections and
> > > > the fact that a connection usually won't be dropped for
> > > quite a while
> > > after
> > > > it's no longer needed. Take a look in computer management,
> > > under sharing.
> > > >
> > > > > -- --Original Message-- --
> > > > > From: owner-xsi@(protected)
> > > > > [mailto:owner-xsi@(protected)] On Behalf Of Ben Kilgore
> > > > > Sent: 02 May 2004 02:21
> > > > > To: XSI@(protected)
> > > > > Subject: Re: 'fatal errors' accessing image files on
> > > network renders
> > > > >
> > > > > Amazing...no one has seen this before?
> > > > >
> > > > > -- -- Original Message -- --
> > > > > From: "Ben Kilgore" <ben@(protected)>
> > > > > To: <XSI@(protected)>
> > > > > Sent: Thursday, April 29, 2004 2:06 PM
> > > > > Subject: 'fatal errors' accessing image files on network renders
> > > > >
> > > > >
> > > > > > List,
> > > > > >
> > > > > > I know in advance I am probably not supplying enough
> > > > > information but I
> > > > > have
> > > > > > the problem of not being able to access my list account
> > > > > from where the
> > > > > > render farm is:
> > > > > >
> > > > > > I have a scene that's consistently crashing xsi (or
> > > > > xsibatch) on a render.
> > > > > > It'll actually get partway through some frame, actually
> > > > > displaying tiles
> > > > > if
> > > > > > I try and render from the UI, but every time, it'll give me
> > > > > a file error
> > > > > > message about net being able to find a texture (and of
> > > > > course the texture
> > > > > is
> > > > > > right where it is looking and was accessed earlier in the
> > > > > render) and
> > > > > it'll
> > > > > > stop. The textures are not on the local blades (couldn't
> > > > > set that up last
> > > > > > night), are all .map, and none of them are over
> > > 2048x2048. The .map
> > > > > > versions of these textures will actually cause the same
> > > > > crash if projected
> > > > > > all alone in a bare scene onto a sphere, but the targas did
> > > > > not--although
> > > > > > the targas do crash the complex scene. We are not
> > > using batchserve.
> > > > > Scene
> > > > > > renders fine on a completely local render--obviously it's a
> > > > > network-related
> > > > > > problem. I know diddly about network rendering, but this
> > > > > had even someone
> > > > > > quite experienced with network rendering scratching his
> > > > > head well into the
> > > > > > wee hours.
> > > > > >
> > > > > > Poked around the archives a bit--found nothing related to
> > > > > this issue.
> > > > > >
> > > > > > As I left the farm at 5:30 in the morning, I forgot to
> > > bring more
> > > > > > information with me, but would greatly appreciate any
> > > of your input.
> > > > > >
> > > > > > Ben Kilgore
> > > > > > FIT
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ---
> > > > > > Outgoing mail is certified Virus Free.
> > > > > > Checked by AVG anti-virus system (http://www.grisoft.com).
> > > > > > Version: 6.0.667 / Virus Database: 429 - Release Date: 4/23/2004
> > > > > >
> > > > > > ---
> > > > > > Unsubscribe? Mail Majordomo@(protected) with the
> > > > > following text in body:
> > > > > > unsubscribe xsi
> > > > >
> > > > >
> > > > > ---
> > > > > Outgoing mail is certified Virus Free.
> > > > > Checked by AVG anti-virus system (http://www.grisoft.com).
> > > > > Version: 6.0.667 / Virus Database: 429 - Release Date: 4/23/2004
> > > > >
> > > > > ---
> > > > > Unsubscribe? Mail Majordomo@(protected) with the following
> > > > > text in body:
> > > > > unsubscribe xsi
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > ---
> > > > Unsubscribe? Mail Majordomo@(protected) with the
> > > following text in body:
> > > > unsubscribe xsi
> > >
> > >
> > > ---
> > > Outgoing mail is certified Virus Free.
> > > Checked by AVG anti-virus system (http://www.grisoft.com).
> > > Version: 6.0.667 / Virus Database: 429 - Release Date: 4/23/2004
> > >
> > > ---
> > > Unsubscribe? Mail Majordomo@(protected) with the following
> > > text in body:
> > > unsubscribe xsi
> > >
> > >
> > >
> >
> >
> >
> > ---
> > Unsubscribe? Mail Majordomo@(protected) with the following text in body:
> > unsubscribe xsi
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.667 / Virus Database: 429 - Release Date: 4/23/2004
>

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.677 / Virus Database: 439 - Release Date: 5/4/2004
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859 (See http://iso-8859.ora-code.com)-1">
<META content="MSHTML 6.00.2800.1400" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY>
<DIV>FFS, RTFM, Ben.</DIV>
<DIV> </DIV>
<DIV> 
<H4 class=pHeading3>Memory-Mapped Textures
</H4><A name=wp478809></A>
Rendering textures can be done quicker
with the use of memorymapped textures. Memorymapped textures are not loaded
into memory but are accessed directly from the disk whenever the shader needs
it. XSI recognizes a memorymappable texture and automatically maps it. Only
the
map image-file format (with the .map extension) can be mapped. <A
name=wp506965></A>
Before a texture can be memorymapped, a
few steps must be taken: 
<DIV class=pSmartList1>
<UL class=pSmartList1><A name=wp478813></A>
<DIV class=pSmartList1>
<LI>The texture must be converted to .map format.
This
is done using the imf_copy standalone. For more information on this tool, see
the Standalones guide on the Documentation CD. The scene file must reference
this texture.</LI></DIV><A name=wp478855></A>
<DIV class=pSmartList1>
<LI>In a distributed rendering network,
memorymapped textures must be loaded onto every machine on the network and
the specified path to the texture must be valid on every machine.
</LI></DIV><A name=wp506968></A>
Copy the image file (.map
)
to every machines local hard drive. <A name=wp478820></A>
<DIV class=pSmartList1>
<LI>Memorymapped textures are considered local by
mental ray, as though the local keyword was used in the scene file.
</LI></DIV><A name=wp506975></A>
So, if the scene is rendered on
multiple hosts, each one accesses the given path rather than transfer the
scene across the network. <A name=wp478821></A>
<DIV class=pSmartList1>
<LI>The memorymap should not be on an NFSmounted
file system. </LI></DIV><A name=wp506979></A>
Although it may seem faster to
have the texture on all the hosts, the network transfers make memorymapping
slower than regular texturing. <A name=wp478822></A>
<DIV class=pSmartList1>
<LI>Memorymapping is ideal for very large textures
containing dozens of megabytes that are sampled infrequently, since most of
the texture file isnt loaded into memory.</LI></DIV></UL></DIV><A
name=wp478823></A>
If the texture and the scene are too
large to fit into the physical memory, then loading a texture is equivalent to
loading the file into memory, decompressing it, and copying it out to swap.
From
that point on, accessing a texture means accessing the swap. <A
name=wp506993></A>
On the other hand, a frequently accessed
texture and scene are smaller and fit into memory. Memorymapped textures are
slower than regular textures because the swap isnt used. For more information
about memory mapped textures, see <A
onmouseover="WWHShowPopup('Shaders__Lights_and_Cameras', 'tex_concept.html
#wp588039', (document.all||document.getElementById||document.layers)?event:null)
;"
onmouseout=WWHHidePopup();
href="javascript:WWHClickedPopup('Shaders__Lights_and_Cameras', 'tex_concept
.html#wp588039');">Creating Memory-Mapped Textures in
Chapter 5</A> of the <A
onmouseover="WWHShowPopup('Shaders__Lights_and_Cameras', 'Coverpage.html
#wp998508', (document.all||document.getElementById||document.layers)?event:null)
;"
onmouseout=WWHHidePopup();
href="javascript:WWHClickedPopup('Shaders__Lights_and_Cameras', 'Coverpage.html
#wp998508');">Shaders, Lights & Cameras</A> guide. </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>-- -- Original Message -- -- 
<DIV>From: "Ben Kilgore" <<A
href="mailto:ben@(protected)">ben@(protected)</A>></DIV>
<DIV>To: <<A
href="mailto:XSI@(protected)">XSI@(protected)</A>></DIV>
<DIV>Sent: Monday, May 03, 2004 11:57 AM</DIV>
<DIV>Subject: Re: 'fatal errors' accessing image files
on network renders</DIV></DIV>
<DIV> </DIV>
<DIV>> Kim, > > Forgot to mention--the
farm completed other scenes without a hitch.  It just > seems to be
this texture and/or scene that's having this effect. > >
Ben > > -- -- Original Message -- -- > From: "kim aldis"
<<A href="mailto:kim@(protected)">kim@(protected)</A>> > To:
<<A href="mailto:XSI@(protected)">XSI@(protected)</A>> > Sent:
Monday, May 03, 2004 3:49 AM > Subject: RE: 'fatal errors' accessing
image
files on network renders > > > > Sounds like that could
your problem. Localhost connected to the same > machine > > will
lift a client licence, so you've effectively taken 6 licences. Go to >
> control panel->administrative tools->Computer Management and look
under > > shared folders->sessions, see how many sessions are
active. You can close > > sessions from there. > > >
> This is one of the many reasons to consider Linux, at least as a
file > > server. The only other way is to buy more MS client license.
The fact that > > Windows does this is bad enough. That it does it so
badly and consts som > > much is outrageous. > > > >
> -- --Original Message-- -- > > > From: <A
href="mailto:owner-xsi@(protected)">owner-xsi@(protected)</A> > >
> [mailto:owner-xsi@(protected)] On Behalf Of Ben Kilgore > > &gt
;
Sent: 02 May 2004 19:44 > > > To: <A
href="mailto:XSI@(protected)">XSI@(protected)</A> > > >
Subject: Re: 'fatal errors' accessing image files on network renders >
> > > > > It's 2000 Professional. > > > >
> > 5 machines (10 CPUs), plus the render control (localhost?) is >
> > rendering, as well. > > > > > > > >
> > > > > > > -- -- Original Message -- -- >
> > From: "kim aldis" <<A href="mailto:kim@(protected)">kim@(protected)</A>> > > > To: <<A
href="mailto:XSI@(protected)">XSI@(protected)</A>> > >
> Sent: Sunday, May 02, 2004 2:09 PM > > > Subject: RE: 'fatal
errors' accessing image files on network renders > > > > >
> > > > > How many machines do you have trying to access the
files on > > > the remote > > > > machine? 
Remember, if you're serving files from NT > > > Workstation 
or
XP, > > > > Microsoft limits you to 10 connections only. In
practice > > > this usually > > > works > >
> > out to be around 6 after you take into consdiration browsr >
> > connections and > > > > the fact that a connection
usually won't be dropped for > > > quite a while > > >
after > > > > it's no longer needed. Take a look in computer
management, > > > under sharing. > > > > > &gt
;
> > > -- --Original Message-- -- > > > > > From:
<A href="mailto:owner-xsi@(protected)">owner-xsi@(protected)</A> > >
> > > [mailto:owner-xsi@(protected)] On Behalf Of Ben Kilgore &gt
;
> > > > Sent: 02 May 2004 02:21 > > > > > To:
<A href="mailto:XSI@(protected)">XSI@(protected)</A> > > >
> > Subject: Re: 'fatal errors' accessing image files on > > &gt
;
network renders > > > > > > > > > >
Amazing...no one has seen this before? > > > > > > >
> > > -- -- Original Message -- -- > > > > > From:
"Ben Kilgore" <<A href="mailto:ben@(protected)">ben@(protected)</A>> > > >
> > To: <<A href="mailto:XSI@(protected)">XSI@(protected)</A>> > >
> > > Sent: Thursday, April 29, 2004 2:06 PM > > > >
> Subject: 'fatal errors' accessing image files on network renders >
> > > > > > > > > > > > > > &gt
;
List, > > > > > > > > > > > > I know
in advance I am probably not supplying enough > > > > >
information but I > > > > > have > > > > >
> the problem of not being able to access my list account > > >
> > from where the > > > > > > render farm
is: > > > > > > > > > > > > I have a
scene that's consistently crashing xsi (or > > > > > xsibatch
)
on a render. > > > > > > It'll actually get partway
through
some frame, actually > > > > > displaying tiles > >
> > > if > > > > > > I try and render from the UI
,
but every time, it'll give me > > > > > a file error >
> > > > > message about net being able to find a texture (and
of > > > > > course the texture > > > > >
is > > > > > > right where it is looking and was accessed
earlier in the > > > > > render) and > > > >
> it'll > > > > > > stop.  The textures are not on
the local blades (couldn't > > > > > set that up last >
> > > > > night), are all .map, and none of them are over &gt
;
> > 2048x2048.  The .map > > > > > > versions
of
these textures will actually cause the same > > > > > crash
if
projected > > > > > > all alone in a bare scene onto a
sphere, but the targas did > > > > > not--although >
> > > > > the targas do crash the complex scene.  We are
not > > > using batchserve. > > > > >
Scene > > > > > > renders fine on a completely local
render--obviously it's a > > > > > network-related >
> > > > > problem.  I know diddly about network rendering,
but this > > > > > had even someone > > > >
> > quite experienced with network rendering scratching his > >
> > > head well into the > > > > > > wee
hours. > > > > > > > > > > > > Poked
around the archives a bit--found nothing related to > > > > >
this issue. > > > > > > > > > > > >
As
I left the farm at 5:30 in the morning, I forgot to > > > bring
more > > > > > > information with me, but would greatly
appreciate any > > > of your input. > > > > >
> > > > > > > Ben Kilgore > > > > >
> FIT > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > --- > > > > > > Outgoing mail is
certified Virus Free. > > > > > > Checked by AVG anti
-virus
system (<A href="http://www.grisoft.com">http://www.grisoft.com</A>). > >
> > > > Version: 6.0.667 / Virus Database: 429 - Release Date:
4/23/2004 > > > > > > > > > > > >
--- > > > > > > Unsubscribe? Mail <A
href="mailto:Majordomo@(protected)">Majordomo@(protected)</A> with
the > > > > > following text in body: > > > >
> > unsubscribe xsi > > > > > > > > >
> > > > > > --- > > > > > Outgoing mail
is certified Virus Free. > > > > > Checked by AVG anti-virus
system (<A href="http://www.grisoft.com">http://www.grisoft.com</A>). > >
> > > Version: 6.0.667 / Virus Database: 429 - Release Date:
4/23/2004 > > > > > > > > > > --- >
> > > > Unsubscribe? Mail <A
href="mailto:Majordomo@(protected)">Majordomo@(protected)</A> with the
following > > > > > text in body: > > > > >
unsubscribe xsi > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > --- > > > > Unsubscribe? Mail
<A href="mailto:Majordomo@(protected)">Majordomo@(protected)</A> with
the > > > following text in body: > > > >
unsubscribe
xsi > > > > > > > > > --- > > >
Outgoing mail is certified Virus Free. > > > Checked by AVG
anti-virus system (<A href="http://www.grisoft.com">http://www.grisoft.com</A>). > >
> Version: 6.0.667 / Virus Database: 429 - Release Date: 4/23/2004 >
> > > > > --- > > > Unsubscribe? Mail <A
href="mailto:Majordomo@(protected)">Majordomo@(protected)</A> with the
following > > > text in body: > > > unsubscribe
xsi > > > > > > > > > > > >
> > > > > --- > > Unsubscribe? Mail <A
href="mailto:Majordomo@(protected)">Majordomo@(protected)</A> with the
following text in body: > > unsubscribe xsi > > >
--- > Outgoing mail is certified Virus Free. > Checked by AVG
anti-virus system (<A href="http://www.grisoft.com">http://www.grisoft.com</A>). >
Version: 6.0.667 / Virus Database: 429 - Release Date: 4/23/2004 >
</DIV>
<DIV> </DIV>
<DIV> --- Outgoing mail is certified Virus
Free. Checked by AVG anti-virus system (<A
href="http://www.grisoft.com">http://www.grisoft.com</A>). Version: 6.0.677
/
Virus Database: 439 - Release Date: 5/4/2004</DIV></BODY></HTML>