Chapter 6

Distributed Rendering

---

About Distributed Rendering

Distributed rendering is a method of dividing up the task of rendering an image or a scene among many different computers across a network using the mental ray renderer. You can also divide a rendering task among several processors on multi-processor computers.

By spreading the workload, your overall rendering time can be decreased considerably. The number of computers you can use is only limited by the number of mental ray licenses you have. In the case of multiprocessor computers, each processor requires a separate license.

Distributed rendering in XSI uses a tile-based rendering method where the image to be rendered is broken up into segments called tiles and distributed across a network of machines.

To distribute frames across a network of machines, use batch rendering. For more information about batch rendering, see Batch Rendering Basics (XSI -R) in Chapter 4 of the Rendering & Compositing guide.

How Distributed Rendering Works

Distributed rendering is started automatically once a render is initiated on a computer. The initiating computer is referred to as the master and the other computers on the network are slaves. The master and slaves communicate via the mental ray service named mi-ray3xsi3_0, which is the service that listens on a TCP port and passes information to mental ray.

The mental ray service

On Windows systems, the actual application that is listening on the TCP port is named ray3xsi3_0server. It runs the ray3xsi3_0server.bat batch file on each computer. This batch file sets the environment variables required for distributed rendering through setenv.bat and then runs the mental ray renderer (ray3.exe). For more information on how to manage the mental ray service on Windows, see Managing the ray3xsi3_0server service.

On IRIX systems, the inetd.conf file is configured to call the ray3.sh script file on each computer. The ray3.sh file sets the environment variables required for distributed rendering and then runs the mental ray renderer (ray3).

On Linux systems, the xinetd.conf file reads the mi-ray3xsi3_0 service configuration file from the /etc/xinetd.d directory. The service is configured to call the ray3.sh script file, which sets the environment variables required for distributed rendering and then runs the mental ray renderer (ray3).

Ray3hosts file

The master reads a local ray3hosts file, which lists the slaves to be used for the render. The image to be rendered is broken up into segments (tiles), which are placed in a queue. Each computer, master or slave, requests tiles from the queue to render.

Once a tile is finished, it is sent back to the master and another tile is requested from the queue. The master assembles all the tiles to create a complete rendered image.

For more information on how to configure the .ray3host file, see Defining a .ray3hosts File.

Linktab.ini file

During distributed rendering, the master also sends any extra information the slave might need to accomplish a render, such as textures. When using large or memory-mapped images, you can use a linktab file to coordinate file sharing.

SPM Server

The SPM server handles the allocation of mental ray renderer licenses when distributed rendering is initiated.

It is recommended that you use a dedicated computer to act as the SPM license server: this computer should not take part in the network rendering or any other memory-intensive tasks. If a computer’s connection to the SPM license server is lost during rendering, the render image will be corrupted.

Distributed Rendering on Other Platforms

Computers with operating systems other than Windows, IRIX, and Linux can be used for distributed rendering on a network by using the version of mental ray found in the MI_ADD_PLATFORM directory on the Software CD.

The rayinstall executable in the MI_ADD_PLATFORM directory installs the scripts and executables necessary for a mental ray rendering client on computer running operating systems such as AIX, Linux, and SunOS.

For a complete list of supported platforms, see the mental images web site at www.mentalimages.com.

Setting Up For Distributed Rendering

To set up distributed rendering for XSI, several steps must be followed before and after the program is installed.

In a network rendering setup, whichever computer initiates the render (the master) oversees the organization of the images to be rendered and distributes the rendering tasks among the other computers in the setup (the slaves). This is accomplished through the use of the ray3hosts and linktab text files.

The ray3hosts file identifies the computers that will participate in distributed renders. The linktab file is used to coordinate sharing resources when rendering on a network that uses a mix of Windows and other operating systems.

The ray3hosts file must be created on the master computer and the linktab file must be created on every computer that will be part of the distributed rendering setup.

For the master to properly communicate with the slaves, each computer should have the same version of the mental ray rendering software installed and use the same TCP/IP port number. It is particularly important to use the same TCP/IP port number if you are distributed rendering across different platforms. Any port number greater than 1024 is recommended.

The master should have more memory than the slave computers and as much virtual memory as possible.

Installing mental ray

To install mental ray on a computer designated for distributed rendering, you simply choose the Render Slave option during the Setup process and set the TCP/IP port number. See Render Slave Installation and Choosing a TCP/IP Port for mental ray.

Defining a .ray3hosts File

The .ray3hosts file identifies the computers that will participate in distributed renders. If a rayhosts file does not already exist, it is automatically created during the Setup process and written to the %Systemdrive%\users\%username%\Softimage\XSI_3.0 directory. This file must reside on the computer designated as the master.

The .ray3hosts file contains a list of the computers that can participate in distributed rendering (not including the master) and the port number it uses for mental ray. Each computer must be listed by name on a separate line, optionally followed by a colon and the port number you want mental ray to use on that computer. If no port number is listed for a given computer, that computer will use the port defined in its MI_RAY3_SERVICE variable.

Any line preceded by a hash (#) symbol is considered a comment and ignored by SOFTIMAGE|XSI.

Here’s a sample .ray3hosts file:

# The first three computers are always part of the
# render network.  The last computer listed is only
# used for overnight renders; remove the # to make
# it available.
larry:7003
moe:7003
curly:7003
# shemp:7024 

The .ray3hosts file must be present on the master computer, as specified by the MI_RAY_HOSTSFILE environment variable. MI_RAY_HOSTSFILE can specify both the path and file name or just the path; if no file name is explicitly set, XSI searches for .ray3hosts, then .rayhosts.

If you are importing scenes from SOFTIMAGE|3D, you can set MI_RAY_HOSTSFILE to point to your existing .rayhosts file.

Depending on your network settings, it can take up to several minutes for a computer to tell mental ray that a host does not answer, so be sure to modify your .ray3hosts file if you know that certain computers are down.

Defining the linktab File

The linktab file is used when the render network uses a mix of Windows and other operating systems. Each line in a linktab file contains a Windows-style path and a UNIX-style path, indicating where XSI or resources such as textures are located on both operating systems. This allows the rendering master to find required files on slaves regardless of operating system.

Most linktab file contain only one line, indicating where XSI is located on both platforms. Here’s a sample linktab file:

C:\Softimage     /usr/Softimage 
 



  

  
    
 


 


    

 
The Windows path must come before the IRIX path and they must be separated by a tab, not spaces.
 
All path names, even for Windows paths, are case-sensitive.
 
Do not use slashes (\ or /) at the end of the linktab paths.

If you are using textures or memory-mapped images, you must have entries that point to the directories containing them.

You can use an exclamation mark to distinguish a mounted volume from the rest of the path. For example, if \\foobar\users\fred (Windows) is equivalent to /home/fred (IRIX) and F:\foobar\users is a mounted volume, the line in the linktab file would look like this:

\\foobar\users!fred     /home/fred 

The linktab file must be present on the master and slave computers, in the directory specified by the SI_LINKTAB_LOCATION environment variable.

SI_LINKTAB_LOCATION can define both the path and file name or just the path; if no file name is explicitly set, XSI assumes the name will be linktab.ini.

When a scene is imported from SOFTIMAGE|3D into XSI, XSI searches in SI_LOCATION for linktab.ini. Define SI_LOCATION in setenv.bat (Windows) or .xsi_3.0 (IRIX and Linux) as the directory where XSI is installed and place a copy of linktab.ini there (if you want to reuse the linktab file used by SOFTIMAGE|3D; otherwise, just make SI_LOCATION point to the same path and linktab name as defined by SI_LINKTAB_LOCATION for XSI).

For details on how XSI uses the linktab file, see Configuring the Linktab.ini File.

Environment Variables for Distributed Rendering

This section lists the environment variables required for distributed rendering. Unless otherwise noted, these variables are automatically set during installation in the environment script (setenv.bat on Windows and .xsi_3.0 on IRIX and Linux).

SPM_HOST must be set to the IP address of an SPM license server (SPM_HOST=192.168.1.50) or host (SPM_HOST=licenservername).

SPM_HOST is set in the environment script during setup.

MI_ROOT defines the location of the ray3rc file.

The default location is the Application/rsrc subfolder of the directory specified by SI_HOME. MI_ROOT is set in the environment script during setup.

On IRIX and Linux it is also set in ray3.sh which is called during distributed rendering.

SI_HOME must be set to the directory in which XSI is installed.

SI_HOME is set in the environment script during setup.

SI_LINKTAB_LOCATION points to the location of the linktab.ini file.

If the linktab file is not named linktab.ini, you must also include the file name.

If you are importing scenes from SOFTIMAGE|3D into XSI, set SI_LOCATION to point to the location of the linktab.ini file.
MI_RAY_HOSTSFILE defines the location and name of the .ray3hosts file on the master computer.

For details on how to define the various environment variables used by XSI, see Appendix A: Environment Variables.

Managing the ray3xsi3_0server service

The ray3xsi3_0server service must be running on each slave computer for it to connect to the master. This is usually an automatic process, but you may need to perform some operations manually when troubleshooting.

For example, from the XSI command prompt type:

ray3xsi3_0server /stop 

This stops the ray3xsi3_0server service on the current machine.

You can run the ray3xsi3_0server executable from an XSI command prompt with the following options:

Option	Function
/install	Installs the ray3xsi3_0server service and starts it.
/remove	Stops ray3xsi3_0server and uninstalls the service.
/start	Starts ray3xsi3_0server on the current computer.
/stop	Stops ray3xsi3_0server on the current computer.

Using Custom Shaders with Distributed Rendering

If you have a renderfarm set up and your scene uses custom shaders, it is crucial that each machine in the renderfarm uses the same version of each custom shader.

Although you can install custom shaders on every machine in your renderfarm, it is quicker and easier to install them into a workgroup path to which all render slaves are pointing.

The procedure for installing custom shader varies depending on what kind of render slaves you’re using.

In an XSI interactive setup, where all of the render slaves have XSI installed, you can install your custom shaders as part of an add-on.
Render slaves using the mental ray standalone do not recognize XSI add-ons, so a few additional steps are required.

Installing Custom Shaders to Workgroups (XSI Interactive Render Slave Installation)

The best way to ensure that all of your renderfarm machines are using the same custom shaders is to copy the shaders to a shared workgroup location. The advantage to using a workgroup location is that the ray3rc file loads workgroup shaders last, so the render slaves will find and use the custom shaders.

For this approach to work, the following requirements must be met:

Every render slave’s ray3xsi3_0server service must be logged in using the same user name and password (see below). Because the workgroup path is set for the user, every render slave will point to the same workgroup path.

The best thing to do is create a user account specifically for rendering. This user must have administrative privileges and have access to the shared workgroup directory.

The custom shaders must be in a shared directory to which the render user has access. A good approach is to use a shared directory on a server. XSI does not have to be installed on the same machine as the shared directory.

Make sure to start and exit XSI on each renderfarm machine. This allows XSI to detect that a new shader has been installed and needs to be registered. The easiest way is to create an empty “dummy” scene which you can load from the command line using: xsibatch -r -scene "dummy.scn". Alternatively, you can start an XSI interactive session to register the shader. XSI will prompt you to restart, which indicates that the shader was registered properly.

To run ray3xsi3_0server as a specific user (Windows systems)

On each render slave where XSI is installed, do the following:

Edit the ray3xsi3_0server service’s properties from the Windows Management Console.

To open the console, right-click the My Computer icon and choose Manage from the menu. The management console opens, from which you can select Services and Applications > Services to display a list of services on your machine.

Right-click the ray3xsi3_0server service and choose Properties from the menu.
From the Log On tab, specify that the service should log on using a particular account.
Enter the user name and password for the rendering user account.
Close the management console. Log off the render slave and log back on using the render account’s username and password.
Open an XSI command prompt and set the workgroup path using the following command.

xsi -w workgrouppath

Now, whenever the ray3xsi3_0server is called, it will render under the specified user and use the workgroup path set for that user.

To Maintain a Custom Shader Workgroup

To keep your distributed rendering setup working smoothly, try using the following process to test, upload, and update your custom shaders.

Create the custom shader locally.
Install the shader locally under “user” and test that it works.
If the shader works correctly, copy it to the workgroup directory using the following command:

xsi -i myshader -dest workgroup

Start and exit XSI on each renderfarm machine.

This allows XSI to detect that a new shader has been installed and needs to be registered. The easiest way is to create an empty “dummy” scene which you can load from the command line using:

xsibatch -r -scene "dummy.scn".

Alternatively, you can start an XSI interactive session to register the shader. XSI will prompt you to restart, which indicates that the shader was registered properly.

To Update Shaders in a Custom Shader Workgroup

If you need to update your custom shaders, you can test the updates locally, backup the shader in the workgroup directory on the server, and promote the updated shader to the server.

Then restart all machines that use the shared workgroup path twice:

The first restart registers the shaders.
The second restart allows the machines to continue to work.

Important: If you are updating shaders in the workgroup directory, make sure that all of the render slaves have stopped rendering. Otherwise, the shaders may be locked by one of the slaves and the copy process will fail.

Window systems allow you to run commands according to a schedule using the “at” command. Using this command with the “net start” and “net stop” commands lets you specify when the Ray3xsi3_0 Server service stops and restarts on each render slave. This makes it easy to create a window of time during which you can upload and register updated shaders.

Installing Custom Shaders to Workgroups (mental ray Standalone Installation)

For render slaves that need to interoperate with the workgroup, the work is fairly simple.

Edit the ray3xsi3_0.bat file and add the following line just before the last line (the one that calls ray3.exe):

set XSI_WORKGROUP_SHADERPLUGINS=\\SERVER\share\addons

On each slave machine, make sure the ray3xsi3_0server .exe service runs as a user that has access to \\SERVER\share\addons directory

To determine if the mental ray standalone can use a custom shader correctly, do the following: Open the shader .spdl file and copy the reference guid (bolded in the example below) at the beginning of the file: Reference = "{GUID}"; Open an XSI command prompt on each machine and do the following: Type: ray3 -verbose on <enter> This makes ray3 work in “stdin” mode, where you can type in the mi2 file directly Type: $lookup "{GUID}" <enter> Press Ctrl + z and then press Enter. If ray3 has loaded the shader’s .spdl and .dll properly, no errors or warnings will be reported. Note: to perform this test, you must have a ray3.exe license.

Installing Custom Shaders to the Local Factory Path (XSI Interactive)

If you want to install a custom shader locally, it is recommended that you use add-ons.

Install your custom shader on a machine using the following command from the XSI command prompt:

xsi -i blah.spdl

Repackage the shader as an add-on. For more information about creating add-ons, see Chapter 24: Plug-ins and Add-ons in the Fundamentals guide.
Install the add-on on each of the machines in the distributed rendering network using the following command from the XSI command prompt:

xsi -i blah.xsiaddon -dest factory

This installs the add-on in the machine’s Factory path instead of its User path.

Using a text editor, edit each machine’s ray3rc file to load factory shaders last. Locate the following entry and move it to the end of the ray3rc file:

	################ load Factory shader plugins
	registry "{SHADER_PLUGIN_HOME}"
		value "{XSI_HOME}"
	end registry
	$lookup "{SHADER_PLUGIN_HOME}"
	registry "{LOAD_FACTORY_PLUGINS}"
		echo "Loading Factory shader plugins"
		mi "{XSI_HOME}/Addons/shader_plugins.mi"
	end registry
	$lookup "{LOAD_FACTORY_PLUGINS}" 

Testing Distributed Rendering

The MI (mental images) file format (*.mi) is the basic render file format to which scenes are exported and rendered by the mental ray rendering software. You can use MI files to test a distributed rendering setup and check the validity of a linktab.ini file.

To create an MI file in XSI

From the Render toolbar, choose Export to mi2 file.
Select Options to choose your render options (number of frames, path to output pictures, and path to the created MI file).

XSI always creates one MI file per frame, so for 10 frames of animation, ten MI files are created. Only one frame should be needed to test distributed rendering.

Testing Image Management Options

When creating MI files for testing, you might want to create a number of different scenes to test your rendering setup. For example, you might want create a specific MI file that makes use of textures.

In XSI, there are different ways to send texture information to the slave. This is defined in the Clip Properties:

Load in Memory — MI file contains the image information. This option can create large MI files.
Use from Disk — MI file contains path to images. Image is sent to slave.
Broadcast — MI file contains path to images. Path is sent to slave.

Launching Distributed Renders with XSI

There are four ways to launch a distributed render:

Interactively from the interface
[xsi|xsibatch] -r from the command line
Command line scripting (xsi -script)
ray3.exe from the command line

All of these methods allow you to have a verbose output in a command prompt, which allows you to “see” what is going on and recognize any point of failure.

The verbose output is not recommended for actual production renders because it slows down the process. For more information on these rendering methods, see Chapter 4: Rendering in the Rendering & Compositing guide.

Troubleshooting

This section contains solutions to common causes of problems with distributed rendering.

Slaves Not Rendering

If a slave is not participating in the distributed rendering, check to see if:

It uses a different version of mental ray from the other computers in the network. All computers in a distributed rendering network must use the same version of mental ray.
Its shaders are located in the same path as the master. To find the specific cause of the problem, open an XSI shell and enter:

ray3.exe -verbose on

This produces a detailed account of ray3’s actions.

Render Not Starting

If the render does not start, make sure that you have enough licenses available. Check the SPM server log for license request information.

Corrupted Images

If your rendered images have corrupted tiles, a slave may be losing its connection to the license server. Ensure that the license server is not running any memory-intensive tasks.

Machine Won’t Load For a Network Render

If a machine won’t load for a network render with mental ray, try the following to see if the machine is responding:

For example, if sparky won’t load, type:

telnet sparky mi-ray3xsi3_0  

 
If you see some messages (“Escape character is...”), this is good. 
 
If you see “Unable to connect...”, there is a problem with the slaves /etc/inetd.conf or /etc/services.

---