You can do it -- at a very slow frame rate and at the risk of melting down less-capable computers. Without a hardware-accelerated plugin, the calculations required are going to be too much for the web unless your scene/models/shading are
very simple (in which case you can use CSS Level 3 transitions/transformations and JavaScript or the HTML5 <canvas> element, again with JS). For things that are slightly more complex, but still a bit "cartoony", Flash would be okay. But the state of the art for 3D on the web isn't all that great yet, and the higher-level stuff isn't well-supported in the "real world". People with gaming-class machines (overclocked multi-core CPUs, honkin'-fast GPUs with a lot of GDRAM provisioned) would probably be okay, but your average web user will hear their fans screaming while the image sort of stutters on the screen.
If the effect you want is for something that's fairly simple (that is, the picture is mostly static, and not a complex, moving 3D model that needs to be re-lit and re-shaded for every frame), then you can start with the simplest example I can think of for effectively tracking the mouse pointer -- the effect that Google Chrome uses in its tab highlights. You can find the code for that
here at CSS Tricks. That will at least get you to the point that you are sure you're tracking the mouse properly, and that your JavaScript/jQuery code is talking to the DOM to modify the styling smoothly. Then you can swap out the background-moving code for a set of CSS3 transforms (or add transforms if you still want the spotlight effect). You'll probably still want to separate the content into layers (just like the parallax effect) so you can perform different perspective transforms on different parts of the image. There are a lot of good tutorials out there for transformations; the hard part will be doing the initial math/geometry and separating out the scene elements effectively, and that's specific to your scene.
Again, though, if you want to get more complicated, it's going to require polygons or voxels and shading, and that won't be good for a standard browser with nothing but Flash available to it on an ordinary desktop/laptop machine at all, at all. There is just too much computation required, especially if it all has to happen in a single thread on a single core on the CPU. Without sending it to the GPU and parallelizing the computations, there's no way you can do a complex rendering fast enough for it to be usable -- the screen will lag behind the mouse by quite a bit, even with the processor core pegged at a constant 100%. You'll need to use a plugin that has access to the hardware, since no browser will allow you that level of access (imagine the kind of malware problems you'd have if any old web site could run arbitrary code "on the metal").
The easiest thing to do would be to pick a trusted gaming engine plugin and write to that. Yes, it means forcing a download for people who don't already have it, but at least the code will run, not crawl while setting fire to the user's machine. Sometimes the answer is not the one you want to hear. And the situation will likely change over the next couple of years as more robust NaCl (native client) solutions are developed and see broader acceptance. But right now, the "native" web is not the right place for this kind of software unless what you're doing is very simple.
