Motion capture, 3d scanning etc

[jefferytitan]

I'm curious who uses techniques like these, and who would if they had access to them.

For any who use them, did you get commercial solutions for them, or roll your own? I imagine that rolling your own would be very difficult unless there's open source available.

[ggambett]

I have an academic interest in 3d scanning, 3d reconstruction from photographs, and motion capture. We don't have the tech yet and I treat these topics as an academic hobby, but once I get results in some area we'll probably try to use them.

[jefferytitan]

I did a computer vision paper at university and no method was perfect. I don't know of any that can simply take a bunch of random photos and combine them. Normally many contraints were required, i.e. all pictures are equidistant, limited colour variation, etc.

I know some techniques work to a degree, such as projecting a line of light (preferably laser) onto an object and digitally photographing it from a different angle to get a 2d slice of the object.

I would argue that limited techniques like that would be fine for games because the colouring can be added later by a digital artist.

I have an academic interest in 3d scanning, 3d reconstruction from photographs, and motion capture. We don't have the tech yet and I treat these topics as an academic hobby, but once I get results in some area we'll probably try to use them.

[vjvj]

Rolling your own for 3D reconstruction via convex hulls wouldn't be too bad.

Motion capture and 3D scanning require dedicated hardware which tends to make it expensive. 3D reconstruction via digital images is mostly software, so the cost should be low depending on how crazy they get with the licenses.

[vjvj]

I did a computer vision paper at university and no method was perfect. I don't know of any that can simply take a bunch of random photos and combine them. Normally many contraints were required, i.e. all pictures are equidistant, limited colour variation, etc.

I saw a presentation at SIGGRAPH a few years ago where they did 3D reconstruction with images taken from a handheld camera. No calibration or special camera positioning was required, in fact the series of images they used looked like they just took pictures while walking down the street. It was very impressive.

Granted, their image processing was extremely complicated and I'm way too stupid to be able to write something like that myself.

[jefferytitan]

Oh, BTW, I had a look at your site. I guess you have more of the teacher gene than I. ;) It brings back memories. Drawing line segments using integer maths in assembly on a 1MHz processor, using OpenGL in Java, etc. I did a 4 year degree, so I guess you've beat me by a little. ;)

I have an academic interest in 3d scanning, 3d reconstruction from photographs, and motion capture. We don't have the tech yet and I treat these topics as an academic hobby, but once I get results in some area we'll probably try to use them.

[ggambett]

I did a computer vision paper at university and no method was perfect. I don't know of any that can simply take a bunch of random photos and combine them. Normally many contraints were required, i.e. all pictures are equidistant, limited colour variation, etc. Take a look at the works of Marc Pollefeys (dense, automatic reconstruction from uncalibrated video) and the early work of Paul Debevec (sparse, model-based reconstruction from a few calibrated pictures). What they do is amazing. I've been working on and off trying to reproduce their PhD works. I've made very slow progress... the math involved gets quite thick. I've gotten to the point of a correct projective reconstruction, and relatively good disparity mapping, but upgrading the projective structure to metric is proving to be very complex and numerically unstable :confused:

Motion capture and 3D scanning require dedicated hardware which tends to make it expensive. It is. I suggested the idea of the basic tech needed to do motion capture to the electronic engineering dept at my university (I teach at the computer science dept) and fortunately it was picked by some students as a research project, so if they succeed I'll be able to turn that tech into full mocap tech. I also intend to suggest 3D reconstruction itself as a research topic for my students next semester.

Oh, BTW, I had a look at your site. I guess you have more of the teacher gene than I. I love teaching CG :) I started "dumping" my course in that blog, mostly for the fun of it. I should continue writing articles someday...

[jefferytitan]

I was thinking that if an algorithm suitable for working with images/video captured from cheap hardware could be developed it would really help out developers, because the current trends seem to be that with resolution and poly count going up, artistic costs can go through the roof. Whereas basing it on real life objects/motions should have a nearly fixed cost (probably more tweaking required once it goes digital, but still 90% there).

Take a look at the works of Marc Pollefeys (dense, automatic reconstruction from uncalibrated video) and the early work of Paul Debevec (sparse, model-based reconstruction from a few calibrated pictures). What they do is amazing. I've been working on and off trying to reproduce their PhD works. I've made very slow progress... the math involved gets quite thick. I've gotten to the point of a correct projective reconstruction, and relatively good disparity mapping, but upgrading the projective structure to metric is proving to be very complex and numerically unstable :confused:

It is. I suggested the idea of the basic tech needed to do motion capture to the electronic engineering dept at my university (I teach at the computer science dept) and fortunately it was picked by some students as a research project, so if they succeed I'll be able to turn that tech into full mocap tech. I also intend to suggest 3D reconstruction itself as a research topic for my students next semester.

I love teaching CG :) I started "dumping" my course in that blog, mostly for the fun of it. I should continue writing articles someday...

[ggambett]

I was thinking that if an algorithm suitable for working with images/video captured from cheap hardware could be developed it would really help out developers, because the current trends seem to be that with resolution and poly count going up, artistic costs can go through the roof. Whereas basing it on real life objects/motions should have a nearly fixed cost (probably more tweaking required once it goes digital, but still 90% there).
Exactly... and given that the number of polygons is approaching the number of pixels, voxel rendering could be the next big thing (again)...

[vjvj]

I was thinking that if an algorithm suitable for working with images/video captured from cheap hardware could be developed it would really help out developers, because the current trends seem to be that with resolution and poly count going up, artistic costs can go through the roof. Whereas basing it on real life objects/motions should have a nearly fixed cost (probably more tweaking required once it goes digital, but still 90% there).

I agree, and I've been keeping my eye on this kind of stuff for that very reason. When I walk down the street in downtown San Francisco I always think "this would take freakin' forever to model without some form of image-based acquisition". The only problem I see (maybe this is easy to solve and you guys know the answer already...) is that ideally you'd like to be able to rip out all the luminance information from the data, so that you have strict albedo values that you can apply arbitrary lighting environments to.

Exactly... and given that the number of polygons is approaching the number of pixels, voxel rendering could be the next big thing (again)...

Yep, and I think this is already starting to happen... Ray-marching volume textures in the pixel shader is starting to pick up steam these days, which IMHO is the first practical step.

[Anthony Flack]

I'd love to have the technology to 3d scan my models.

[ggambett]

ideally you'd like to be able to rip out all the luminance information from the data, so that you have strict albedo values that you can apply arbitrary lighting environments to.
For that, this time I refer you to the latest works of Paul Debevec :) Google for Relighting the Parthenon or something like that.

[Sysiphus]

In my first years as an artist, I did dream with all these...I was really close to buy an under 6k one, which was pretty cheap ofr the times.

Remember one thing, 3d scanning requires yet a load of cleaning, retouching and optimizing. I, as a modeler, have had to deal with those clouds of points...You will actually need once again a modeler ;)

No, really. I have modeled with clay, I'd love if this technology would have become cheaper.I *really* enjoy way more traditional techniques than digital.

My purpose was actually do the 80% of the work with no touch of screen, and a last 20% retouching and fixing. Then I realized those werent' the exact %.

A fantasy character is something you actually need to create, in the real clay thing, or digitally, though. Real life objects, certainly not. But also you need the scanner to translate the texture color and make an uvmapping "traslation" of it...Years ago when last I checked, this was possible with one above 12k scan, based on all around video capturing techniques, which supposedly did take both geometry and texturing surface, so it gave as result a textured model...I'd have to see that to believe it, as photogrametry is far from being a perfect thing, at least in what i have seen...

All this is very outdated (unless they have not make a single progress step in years ;) ) ...but what I remember is there were magnetic ones, like Polhemus tracker, which grabbed actually simply coords numbers through some phisical pointer, and dumped em in a txt. Yep, was simple as that. tehre was a free tool, must be in my CDs, that could take the points cloud, and generate a mesh of it. Of course, not uv mapped, not textured.

To me, the practical scanners maybe curiously some of the cheaper ones. That one did have the advantage, you only took some of the points u wanted. Microscribe had this way too, but not magnetic, just a mechanic arm you handled to point whatever, and did grabbed also thos epoints. The arm lenght was limited, was for like one meter models. But the technique to me was convincing (well, I only went to see the Polhemus one, working.No Microscribe in my country then. ) As you did painted over your clay figure, the mesh wires you wanted as final result. This is actually amazing, and is what does make Silo's (modelling software, very good) with its topology brush over very hi res mesh when want a low/mid pol one of it, and done quick and ordered. You went with that articulated arm pointing in the wires painted croses. You so could have painted even the flow of wires for ease animation later on.It had a Max plugin that traslated the stuff as nurbs, and there are , I think , more than one way to convert nurbs points and nurbs curves, to same points in a polygon mesh with its straight edges ;) Sure today that is surpased and it also does translates direct to polies.

there were more expensive laser scanners, one was like a "gun", was quite amazing to see. But I no more convinced , for 3d or real time stuff, with so much verts. Even cleaning it requires quite a good machine...

yet so, today the normal maps, and crazy one million polies models detail for displacement and normal maps, has become a needed fashion, not only in films but in videogames (comercial AAA big titles(not necesarily better for the size ;) )) .So, for crazy detail, wouldn't be practical the only-some-points technique.

there was also that video tech , with a video camara rotating around the model and grabing both color and geometry info, very expensive too.

There those others...Picza? scanners...can't remember...been a real load of years....year '96, '97...? Can't remember...I only modelled (and not in my today's skill) in hi res... picza, if that was the name...did allow only uber small models of clay or wood scanning...it was actually a fine pointer acting only over some few centimeters... (like one inch or something...) This were the cheaper, but only good if u were really good in the traditional skill of doing miniatures...


And several other systems.

Heck, I had to actually asume I had to learn mores seriously modelling in low and mid poly...and here I am...learnt even 3d animation and all fields. LOL! When i think I did all this stuff of research to win my bucks as traditional artist, and not deal with screens.... (and even funnier is that the very today moment, I earn my bucks with pixel art...)

But yes, I do share the dream with Anthony Flack, I'd *love* not only sculpting, but only using my traditional skills (drawing, painting, clay) and no need ever more the screen than the usual scan for drawing (you scan, adjust levels, some rotation, some sharpening, image size, voilà! ), and for 3d (with similar minutes of adjusting, no more...)


edit: Polhemus Isotracker. That was the name of the scanner I was near to buy. Was magnetic technology, and allowed to track movement as well as be used to scan full 3d models. Was curiously cheaper , in that version, pretty usable, than the expensive many million verts laser scanners the company also was selling. I was very kindly attended by them, and was able to see it in action. :) Finally decided not to buy, as the idea was use it as a business of rapidly building hi res human and creature models (clay modeled by me, or the like) and sell em in colections, for games and specially hi res art for design companies. I saw the whole process took in the digital stage time too similar to actually modelling from scratch in computer. Today I think that, if you'r an skilled modeler, or can be helped by one, the work on a hi res model is quite low. I didn't know by then certain really fast techniques to clean, optimize, to model(make it "animatable", unless is just a for normal maps and displacement), uvmap and texture. Today I think a 3d scanner can be of great help, but in good hands ;)

edit2 : One last thing. Google for Metasequoia LE, perhaps Mizno Lab metasequoia will bring you direct there. Go to the english url result, not the japanese one ;) It's a good modeler with fixing options, which really can handle heavy models. Also, you may enjoy Vizup 1.8 optimizer freeware. the comercial version is surely way better. I have the 1.8 and use it often. The freeware version only dealt with vrml format, but respected texturing , and optimizing done with texture and uvmapping was really good! Better than many comercial ways I have seen. vrml forces tris, though, you must be aware.

There's a PLY viewer, very hard to find. I have it in some cd , too. Maybe today is easier to find with google. many 3d scans, already done, at universities, etc, are in ply format. That's why I mention. the converted file as OBJ, was only able to be handled well (just some seconds in each rotation ;) ) with Metasequoia LE(freeware for non comercial purposes.The shareware only exports mqo native format, which can be read by Ultimate Unwrap, but means nothing, as is not legally allowed the comercial use of it without purchasing. Sadly, he didn't sell outside japan, dunno if today he does. It was my fav modeler previous to knowing Wings. To which I jumped for the legally allowed comercial use, only. (today I wouldn't go back) ), of all my modelers. Metasequoia is easy to learn, just remmeber to select "english" language, and disable 'basic mode', at file menu. Nothing could handle that good heavy models, in my experience.