bahattab
27-12-2008, 06:04 PM
****DISCLAIMER****
IT HAS BEEN DISCUSSED IN VARIOUS FORUMS THE SOMEWHAT BIASED NATURE OF THESE TESTS. AUTODESK HAS REQUESTED THE OPPORTUNITY TO REFUTE THE CLAIMS MADE BELOW.
I AM AWAITING THEIR APPROVAL TO RELEASE THEIR EXPLANATIONS AS TO WHY THE TEST RESULTS ARE SO SKEWED IN FAVOR OF XSI. I WILL POST THAT INFORMATION AS SOON AS I GET THE GO AHEAD TO DO SO.
http://www.atyafonline.com/vb/imgcache/1262.png
The following pages contain excerpts from the white paper titled "Quantifying performance in 3D modeling and animation software" 3dspeedmachine.com was given permission to summarize the findings of the tests.
Reproduction of this material is strictly prhobited without written permission from Jon Peddie Research or 3dspeedmachine.com. Click here (http://www.jonpeddie.com/special/WhitePapers/qualifying-performance-in-3d-modelling.php) to visit the white paper page and download the scene files
"Softimage recently commisioned Jon Peddie Research to conduct a series of tests on XSI, Maya and Max.
The objective of the tests was to quantify the performance advantages and disadvantages of each software package."Softimage has devoted considerable resources to optimizing their software for multi-core computing on the CPU side.
As a result, they wanted to find a way to quantify and demonstrate the advantages. Jon Peddie Research was hired to certify that the tests were reasonable simulations of a users workload.
We also recreated the tests in house and found that our results were similar to those obtained by Softimages engineers.
We recognize that this is far from perfect, and we hope that other software vendors will participate and recreate the tests in their own products. The .FBX files of the models used to create the tests are available on our website and can be used to create similar tests."
The results of the tests point out several issues that are very helpful to the 3D community.
Seven different tests were conducted in an attempt to recreate typical tasks often encountered by a 3D animator/modeler.
The Tests:
Jogger 6K triangles and High res jogger, 35K triangles - both of these tests are good examples for game development with its emphasis on character and also skin deformation.
Massive Urchin Turning - The massive urchin is just what it claims to be, one million polygons of sea creature with a turn movement to simulate transformations.
Massive Urchin Twisting - The same million-polygon urchin but this time with deformation. This test stresses the machine the most in all cases.
One Million Particles - The one million particle test demonstrates the ability of the software to generate particles along a path.
1000 Cubes Rotating - The turning cubes also demonstrate the ability of the software to handle transformations for 1000 objects.
1000 Cubes Rotating - The turning cubes also demonstrate the ability of the software to handle transformations for 1000 objects.
10000 Particles with Four Goals - The 10000 particles and four goals demonstrate particles being generated and also transformed.
The Rules:
1. Only one application can be open at a time. In fact, the tests were run on a completely clean machine with no other applications running.
2. The tests are run full-screen.
3. The tests are opened through the File Menu and not dragged and dropped because the different products manage models and related information differently.
4. All animations are run at 24 fps.
5. Tests are run in wireframe to be sure the GPU is playing a role in rendering.
6. Nothing is selected in the scene.
Workstation Specs and Software:
- AMD Opteron Quad-Core 2.30 GHz with 2 processors
- 8GB of RAM.
- Graphics board was a FireGL V7600 with the Catalyst 2008.0603.2230.38408 driver.
- The tests were run on both 64-bit Vista and Window XP.
- 3ds Max and Maya (as of Summer 2008) and XSI 7.
http://www.atyafonline.com/vb/imgcache/1263.png
http://www.atyafonline.com/vb/imgcache/1264.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"The jogger tests include a running character. Maya is the clear winner in these tests, especially on Windows XP. Notice. However, that Vista is considerably slower than Windows XP—almost 2X as much."
http://www.atyafonline.com/vb/imgcache/1265.png
http://www.atyafonline.com/vb/imgcache/1264.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"The jogger tests are self-explanatory. They were created to simulate jobs that game developers do while working with characters. Maya handled this test comfortably with XSI coming in second and Max a distant third.
Overall, the difference between the smaller model and the high-res model was predictable. Performance scaled as expected. In general, what we’ll see throughout these tests is that Vista is considerably slower than Windows XP.
And, as a result, we don’t expect to see professionals move to Vista until this disparity, which has been noted in other disciplines (most notably CAD), is addressed. We ran the tests through once with the Task Manager on so we could see how the processors were being called upon as the tests ran.
The jogger makes fairly minimal demands on the system and thus no more than one processor came into play during the test."
Another thing to note is how much more efficient Maya is at handling Vista is this case. Even more efficient than XP 64 bit. Max's performance is very poor compared to the other two in this test.
http://www.atyafonline.com/vb/imgcache/1266.png
http://www.atyafonline.com/vb/imgcache/1267.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"In the Massive Urchin [Turning] tests, the very large model is twisted and turned to demonstrate the ability to manipulate a large model. In this case, Maya was slowest in dealing with such a large model. XSI was the winner. Interestingly, Max seemed to handle Vista’s overhead better, but again, Vista imposes an impressive penalty."
XSI was the clear winner in this test while Maya was a distant 3rd. Max was roughly twice as slow as XSI in this test.
http://www.atyafonline.com/vb/imgcache/1268.png
http://www.atyafonline.com/vb/imgcache/1267.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"The Massive Urchin Twisting test represented the biggest challenge to the systems. The Task Manager window revealed the processors kicking in for all the programs. However, we noticed that in the case of XSI, the additional processors kicked in fairly equally, while in Max and Maya we found more intermittent response.
As XSI ran, the Task Manager reported that 50-55% of the CPU was being used. In the case of Maya and Max, Task Manager reported only about 13-14% use. Obviously, we don’t know what’s going on inside these programs, but it seems safe to infer that XSI is making more efficient use of the processors.
We were never successful getting Max to run the test animation all the way through."
Note: Max was unable to complete this test and thus is not shown in the graph.
"When it comes to multiple processors, particles represent low-hanging fruit. We ran three groups of tests that use particles, One Million Particles, and Particles with Four Goals. In all cases, we found that multiple processors were being called upon to do work. However, we also discovered some unpredictability about how the different software programs handle particle functions.
The sizes of the files were the same, the numbers of the particles remained the same and, in all programs, the particles change—either accumulating, or moving in a particular direction.
However, particles being what they are, each program accomplished its task a little differently. This is a perfect example of the challenge inherent in creating tasks using actual processes rather than rendered scenes.
That said up front, we hope this white paper and these test will inspire participation from the 3D community to help create more tests that can be run across different software platforms."
http://www.atyafonline.com/vb/imgcache/1269.png
http://www.atyafonline.com/vb/imgcache/1270.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"The 1000 cubes turning is a surprise. Maya wins on Windows XP but loses on Vista. Obviously, there’s something going on underneath the hood here. However, Task Manager offers an interesting clue.
When the thousand cubes are turning, only one processor is being impacted and there’s only modest demand being place on the processor. We’ll address this observation further in our conclusions."
All the programs performed relatively well in this test. Max and XSI performed rather similarly. Maya performed roughly twice as well on XP compared to Vista.
http://www.atyafonline.com/vb/imgcache/1271.png
http://www.atyafonline.com/vb/imgcache/1272.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"One Million Particles: In the one million particles test, the animation builds as one million particles are generated. Once again, we see that Maya is better able to handle the overhead imposed by Vista but XSI is the fastest in XP.
Throughout these tests, it’s clear that Max is challenged by particles. The one million particle tests again highlight the unpredictable nature of the “Vista effect.” XSI and Maya are very closely tied in Vista, but in the case of XP, the win goes to XSI."
Again 3DS Max see's a very large performance gap between XP and vista. XP vs Vista on XSI has a similar gap.
http://www.atyafonline.com/vb/imgcache/1273.png
http://www.atyafonline.com/vb/imgcache/1274.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"1000 Particles with Goals: Once again, 1000 particles are directed towards an assigned path, this time toward four goals. XSI wins on both Vista 64-bit and Windows XP. However the race is a little faster on XP.
The 1000 Particles with Goals raises another interesting question about testing different packages against each other and trying to find comparable scenes and procedures. XSI performed faster but Max and Maya performed the procedure most predictably.
The particles flowed gracefully to the goals. In Maya and XSI, the particles flowed, but they missed the goals.""In all cases, the particle tests challenge multiple processors. In the case of XSI, we could see all processors being hit—somewhat equally. In the case of Max and Maya, the CPUs seemed to pick up the slack more intermittently and unevenly."
In conclusion, it is obvious that there are some serious issues regarding running 3d applications on Windows Vista. In a world where time is money it is nice to know where we can get the most bang for our dollar.
The authors of this white paper acknowledge that there are several factors that go into any given software's usability. An artist's comfort level in any a 3d application is certianly an important factor.
Personally, I'm a Max user and although it is clear that Max is much slower at performing certian tasks on paper, that advantage might be outweighed by my inability to adapt to new software.
Judging from these tests I think it will be interesting to see what Autodesk decides to do with Max now that it has aquired Softimage XSI.
When performing the same tasks, with the same models, XSI consistently won while Maya was a close second and Max a distant 3rd.
If you were completely new to 3D and were choosing a software application, why would you choose Max given this information?
I think Autodesk will soon realize this if they haven't already. There is no need to have two apps (Max and XSI) geared towards the same users. Especially if one is far more efficient than the other.
You can read the entire white paper and download the scene files HERE (http://www.jonpeddie.com/special/WhitePapers/qualifying-performance-in-3d-modelling.php)
http://jonpeddie.com/special/WhitePapers/Whitepaper-3d-performance/
http://www.jonpeddie.com/special/WhitePapers/qualifying-performance-in-3d-modelling.php
http://www.3dspeedmachine.com/News/compare/compare.htm
IT HAS BEEN DISCUSSED IN VARIOUS FORUMS THE SOMEWHAT BIASED NATURE OF THESE TESTS. AUTODESK HAS REQUESTED THE OPPORTUNITY TO REFUTE THE CLAIMS MADE BELOW.
I AM AWAITING THEIR APPROVAL TO RELEASE THEIR EXPLANATIONS AS TO WHY THE TEST RESULTS ARE SO SKEWED IN FAVOR OF XSI. I WILL POST THAT INFORMATION AS SOON AS I GET THE GO AHEAD TO DO SO.
http://www.atyafonline.com/vb/imgcache/1262.png
The following pages contain excerpts from the white paper titled "Quantifying performance in 3D modeling and animation software" 3dspeedmachine.com was given permission to summarize the findings of the tests.
Reproduction of this material is strictly prhobited without written permission from Jon Peddie Research or 3dspeedmachine.com. Click here (http://www.jonpeddie.com/special/WhitePapers/qualifying-performance-in-3d-modelling.php) to visit the white paper page and download the scene files
"Softimage recently commisioned Jon Peddie Research to conduct a series of tests on XSI, Maya and Max.
The objective of the tests was to quantify the performance advantages and disadvantages of each software package."Softimage has devoted considerable resources to optimizing their software for multi-core computing on the CPU side.
As a result, they wanted to find a way to quantify and demonstrate the advantages. Jon Peddie Research was hired to certify that the tests were reasonable simulations of a users workload.
We also recreated the tests in house and found that our results were similar to those obtained by Softimages engineers.
We recognize that this is far from perfect, and we hope that other software vendors will participate and recreate the tests in their own products. The .FBX files of the models used to create the tests are available on our website and can be used to create similar tests."
The results of the tests point out several issues that are very helpful to the 3D community.
Seven different tests were conducted in an attempt to recreate typical tasks often encountered by a 3D animator/modeler.
The Tests:
Jogger 6K triangles and High res jogger, 35K triangles - both of these tests are good examples for game development with its emphasis on character and also skin deformation.
Massive Urchin Turning - The massive urchin is just what it claims to be, one million polygons of sea creature with a turn movement to simulate transformations.
Massive Urchin Twisting - The same million-polygon urchin but this time with deformation. This test stresses the machine the most in all cases.
One Million Particles - The one million particle test demonstrates the ability of the software to generate particles along a path.
1000 Cubes Rotating - The turning cubes also demonstrate the ability of the software to handle transformations for 1000 objects.
1000 Cubes Rotating - The turning cubes also demonstrate the ability of the software to handle transformations for 1000 objects.
10000 Particles with Four Goals - The 10000 particles and four goals demonstrate particles being generated and also transformed.
The Rules:
1. Only one application can be open at a time. In fact, the tests were run on a completely clean machine with no other applications running.
2. The tests are run full-screen.
3. The tests are opened through the File Menu and not dragged and dropped because the different products manage models and related information differently.
4. All animations are run at 24 fps.
5. Tests are run in wireframe to be sure the GPU is playing a role in rendering.
6. Nothing is selected in the scene.
Workstation Specs and Software:
- AMD Opteron Quad-Core 2.30 GHz with 2 processors
- 8GB of RAM.
- Graphics board was a FireGL V7600 with the Catalyst 2008.0603.2230.38408 driver.
- The tests were run on both 64-bit Vista and Window XP.
- 3ds Max and Maya (as of Summer 2008) and XSI 7.
http://www.atyafonline.com/vb/imgcache/1263.png
http://www.atyafonline.com/vb/imgcache/1264.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"The jogger tests include a running character. Maya is the clear winner in these tests, especially on Windows XP. Notice. However, that Vista is considerably slower than Windows XP—almost 2X as much."
http://www.atyafonline.com/vb/imgcache/1265.png
http://www.atyafonline.com/vb/imgcache/1264.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"The jogger tests are self-explanatory. They were created to simulate jobs that game developers do while working with characters. Maya handled this test comfortably with XSI coming in second and Max a distant third.
Overall, the difference between the smaller model and the high-res model was predictable. Performance scaled as expected. In general, what we’ll see throughout these tests is that Vista is considerably slower than Windows XP.
And, as a result, we don’t expect to see professionals move to Vista until this disparity, which has been noted in other disciplines (most notably CAD), is addressed. We ran the tests through once with the Task Manager on so we could see how the processors were being called upon as the tests ran.
The jogger makes fairly minimal demands on the system and thus no more than one processor came into play during the test."
Another thing to note is how much more efficient Maya is at handling Vista is this case. Even more efficient than XP 64 bit. Max's performance is very poor compared to the other two in this test.
http://www.atyafonline.com/vb/imgcache/1266.png
http://www.atyafonline.com/vb/imgcache/1267.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"In the Massive Urchin [Turning] tests, the very large model is twisted and turned to demonstrate the ability to manipulate a large model. In this case, Maya was slowest in dealing with such a large model. XSI was the winner. Interestingly, Max seemed to handle Vista’s overhead better, but again, Vista imposes an impressive penalty."
XSI was the clear winner in this test while Maya was a distant 3rd. Max was roughly twice as slow as XSI in this test.
http://www.atyafonline.com/vb/imgcache/1268.png
http://www.atyafonline.com/vb/imgcache/1267.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"The Massive Urchin Twisting test represented the biggest challenge to the systems. The Task Manager window revealed the processors kicking in for all the programs. However, we noticed that in the case of XSI, the additional processors kicked in fairly equally, while in Max and Maya we found more intermittent response.
As XSI ran, the Task Manager reported that 50-55% of the CPU was being used. In the case of Maya and Max, Task Manager reported only about 13-14% use. Obviously, we don’t know what’s going on inside these programs, but it seems safe to infer that XSI is making more efficient use of the processors.
We were never successful getting Max to run the test animation all the way through."
Note: Max was unable to complete this test and thus is not shown in the graph.
"When it comes to multiple processors, particles represent low-hanging fruit. We ran three groups of tests that use particles, One Million Particles, and Particles with Four Goals. In all cases, we found that multiple processors were being called upon to do work. However, we also discovered some unpredictability about how the different software programs handle particle functions.
The sizes of the files were the same, the numbers of the particles remained the same and, in all programs, the particles change—either accumulating, or moving in a particular direction.
However, particles being what they are, each program accomplished its task a little differently. This is a perfect example of the challenge inherent in creating tasks using actual processes rather than rendered scenes.
That said up front, we hope this white paper and these test will inspire participation from the 3D community to help create more tests that can be run across different software platforms."
http://www.atyafonline.com/vb/imgcache/1269.png
http://www.atyafonline.com/vb/imgcache/1270.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"The 1000 cubes turning is a surprise. Maya wins on Windows XP but loses on Vista. Obviously, there’s something going on underneath the hood here. However, Task Manager offers an interesting clue.
When the thousand cubes are turning, only one processor is being impacted and there’s only modest demand being place on the processor. We’ll address this observation further in our conclusions."
All the programs performed relatively well in this test. Max and XSI performed rather similarly. Maya performed roughly twice as well on XP compared to Vista.
http://www.atyafonline.com/vb/imgcache/1271.png
http://www.atyafonline.com/vb/imgcache/1272.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"One Million Particles: In the one million particles test, the animation builds as one million particles are generated. Once again, we see that Maya is better able to handle the overhead imposed by Vista but XSI is the fastest in XP.
Throughout these tests, it’s clear that Max is challenged by particles. The one million particle tests again highlight the unpredictable nature of the “Vista effect.” XSI and Maya are very closely tied in Vista, but in the case of XP, the win goes to XSI."
Again 3DS Max see's a very large performance gap between XP and vista. XP vs Vista on XSI has a similar gap.
http://www.atyafonline.com/vb/imgcache/1273.png
http://www.atyafonline.com/vb/imgcache/1274.png (http://javascript%3Cb%3E%3C/b%3E:;)
Click to enlarge screenshot
"1000 Particles with Goals: Once again, 1000 particles are directed towards an assigned path, this time toward four goals. XSI wins on both Vista 64-bit and Windows XP. However the race is a little faster on XP.
The 1000 Particles with Goals raises another interesting question about testing different packages against each other and trying to find comparable scenes and procedures. XSI performed faster but Max and Maya performed the procedure most predictably.
The particles flowed gracefully to the goals. In Maya and XSI, the particles flowed, but they missed the goals.""In all cases, the particle tests challenge multiple processors. In the case of XSI, we could see all processors being hit—somewhat equally. In the case of Max and Maya, the CPUs seemed to pick up the slack more intermittently and unevenly."
In conclusion, it is obvious that there are some serious issues regarding running 3d applications on Windows Vista. In a world where time is money it is nice to know where we can get the most bang for our dollar.
The authors of this white paper acknowledge that there are several factors that go into any given software's usability. An artist's comfort level in any a 3d application is certianly an important factor.
Personally, I'm a Max user and although it is clear that Max is much slower at performing certian tasks on paper, that advantage might be outweighed by my inability to adapt to new software.
Judging from these tests I think it will be interesting to see what Autodesk decides to do with Max now that it has aquired Softimage XSI.
When performing the same tasks, with the same models, XSI consistently won while Maya was a close second and Max a distant 3rd.
If you were completely new to 3D and were choosing a software application, why would you choose Max given this information?
I think Autodesk will soon realize this if they haven't already. There is no need to have two apps (Max and XSI) geared towards the same users. Especially if one is far more efficient than the other.
You can read the entire white paper and download the scene files HERE (http://www.jonpeddie.com/special/WhitePapers/qualifying-performance-in-3d-modelling.php)
http://jonpeddie.com/special/WhitePapers/Whitepaper-3d-performance/
http://www.jonpeddie.com/special/WhitePapers/qualifying-performance-in-3d-modelling.php
http://www.3dspeedmachine.com/News/compare/compare.htm