01 / Research · September 15, 2026 · 11 min read

Turning Images into 3D Gaussian splats

A client asked me recently whether there was anything we could do with Gaussian splats. It was a hot topic then, it still is, and as the AI tooling keeps improving it is not going away. We had a good conversation and it left me slightly annoyed with myself, because although I knew how splats worked and how people made them, I had never actually pulled out a phone and walked around a subject to capture one. I knew the theory and none of the practice.

So I went looking for something in that space that had not already been done to death, which for me usually means picking the harder version of the problem. The question I landed on was how difficult it would be to turn a single image into a fully realised splat. The cover is an AI-generated concept illustration, not an experimental output.

The whole pipeline on one frog: the photo, the point cloud MoGe-2 lifts from it, the control video rendered from an orbit I author, the photoreal video a fal.ai model makes from that, and the splat Brush trains on the exact same poses. The last three share one camera, frame 17 of 120, set before anything was generated.

One photo each. Trained splats rendered through the exact COLMAP cameras they were trained on, with the photo each one started from in the corner.

Why not just make a 3D model

The obvious objection is that image-to-3D already exists and is pretty good. You can feed a picture to Tripo or Meshy and get a decent mesh back.

The reason I wanted a splat instead is fidelity to the source. A splat trained from a photograph or a rendered image stays one-to-one with it: the art style, the structure, the material response, all of it survives because the splat is reproducing what the pixels actually looked like rather than rebuilding the object from scratch. A generated mesh approximates, and the approximation shows up hardest in materials and textures, which are the exact things that make an image feel like itself. For what I want these assets for, mostly TouchDesigner and real-time interactive installations, that tradeoff is worth it. In those setups I can keep camera movement and angles fairly restricted, so I can afford to be flexible about geometric accuracy in exchange for something that genuinely looks real.

How splats are normally made, and why one image breaks it

A splat is trained from many images of the same subject taken all the way around it, ideally at several elevations so nothing is missed. You hand that set to a trainer such as Brush or Lichtfeld Studio, it finds the same pixels across images, triangulates, does some fairly crazy maths, and produces a point cloud that the gaussians are then fitted to.

Every part of that assumes you have more than one viewpoint. With a single image you have one, and the entire chain has nothing to bite on. So the whole project became a question of how to manufacture the other viewpoints honestly enough that a trainer would accept them.

The obvious move is to generate a 360 video of the subject from the single image and just train on those frames. I never seriously went down that road, and the reason is buried in the sentence above. Brush and Lichtfeld Studio are built to track the same pixels across frames, and AI video generation does not give you the same pixels twice. Temporal inconsistency is inherent to how these models work: detail shifts between frames, forms warp slightly, small things pop in and out of existence.

To the eye a good generation looks completely consistent, which is exactly what makes this trap easy to fall into. Training does not happen at the level of the eye. It happens at the level of pixel correspondence, and at that level there is no anchor to hold on to, so a traditional trainer will most likely fail on that footage no matter how convincing it looks when you watch it.

Two approaches that did not work

My first thought was the obvious shortcut. Convert the image into a 3D model, orbit a camera around it in Blender, export that footage, and push it through a reference-to-video model like Seedance 2.0 or LTX to make it photoreal. It works, but it introduces a whole extra stage whose only job is to produce geometry I was going to throw away, and decent image-to-3D means paying for API credits on Tripo or Meshy. I dropped it quickly.

Then I looked at 4D splats and at the feed-forward reconstruction models, the ones that take all the frames in a single forward pass rather than matching features from frame to frame. In principle that sidesteps the tracking problem entirely. In practice what I got back was a separate point cloud per frame, and they did not agree with each other well enough to be useful.

Testing the tracking and solver route on a sari character. The recovered orbit is on the left, the point cloud rendered from the estimated camera is in the middle, and the source frame is on the right. The middle panel is the problem.

The approach that worked

Skip the mesh. Lift the image straight into a point cloud, and build the orbit around that.

For the lift I started with lightweight depth models like Depth Anything, which are fast and fine for some subjects but give you relative depth, normalised into a 0 to 1 range with no real-world scale. That is enough to fake a parallax shift and not enough to place a camera. MoGe-2 and SHARP predict metric depth instead, with actual distances rather than a normalised ramp, which is what lets the resulting cloud sit in a world where a camera orbit means something.

Loading the source image and lifting it into a point cloud.

With the cloud in place I author a circular orbit around it at a single elevation and render that as a control video. The render has backface culling on, so once the camera passes behind the cloud there is nothing to see, which matters more than it sounds like it should.

Authoring the orbit. Radius, elevation, aim height, canvas and lens, with a status block underneath to check before rendering.

Before that control video goes anywhere I soften it, downsampling and blurring the frames. This is effectively a creativity slider. The blurrier and lower resolution the control, the more room the video model has to invent, which is exactly what I want for the half of the subject that was never in the source image. Push it too far and it stops respecting the source, so where you set it depends on how much invention the subject can tolerate.

The generate step.

Then it goes to a video model on fal.ai, with the source image as reference and a description of the subject that I have Gemini Flash write automatically, so I am not hand-prompting every run. The prompt itself leads with the lighting, which matters more than it looks like it should, because these models weight the front of an instruction far more heavily than the back.

The approval window shows the real media, the full path of every file and the exact prompt string before anything is uploaded.

The point cloud already carries enough information to constrain roughly 180 degrees of the subject. Behind that, where backface culling left nothing, the model fills in detail from scratch. That division of labour is the whole trick: the geometry I care about is authored, and generation is confined to appearance.

One 120-frame orbit, three ways: the control video, the fal.ai video and the trained splat. Once the camera passes behind the point cloud, the control video is nearly empty, and the back of the head is the model's invention, carried faithfully into the splat.

Choosing a model

I have used LTX 2.3 render-to-real, Wan 3 and MiniMax 3. As of late 2026 my favourite is Wan 3, purely for the realism it achieves, though this is the part of the post most likely to be out of date within a few months.

LTX is the accuracy option. It tracks the control video very closely, and the requirement is that your source image and the first frame of the control video are identical. Wan 3 and MiniMax 3 are looser about that and give you more freedom, at the cost of drifting: the orbit can come back running half a second fast or slow. That is fixable, and my fix is unglamorous. I drop the generated footage into After Effects over the control video and retime it until they line up.

LTX 2.3: source image, control frame, output frame.

Wan 3 on the glass frog that became my main test subject.

MiniMax 3 at 2K.

Wan 2.2 VACE, depth-controlled.

Six frames across one orbit.

One art-direction choice mattered more than I expected. I light everything as though there is a flash on the camera, a single hard source from the viewer's position. Without it the subject tends to come back looking like a flat material with almost no highlight information, and a splat trained on that looks dead. With it, the speculars move across the surface as the camera orbits, and that movement is what sells the result as a real object.

Matting, and why the background matters

Render the control video against a solid background. It makes the next step reliable, and the next step is cutting the subject out of every returned frame.

That cutout, the matte, is what tells the trainer which pixels are the subject and which are not. Without it the trainer happily fits gaussians to the background too, and since the video model reinvents that background on every frame, you end up with a cloud of contradictory geometry floating around your object. A clean matte also gives COLMAP a much easier time separating foreground from background. I use SAM 2.1 or BiRefNet-HR depending on the subject, and BiRefNet in particular is very good at hair and fine edges.

Then it trains in Brush, and that is the whole pipeline. Image in, splat out.

The glass frog through every stage on one orbit: the control video rendered from the point cloud, the video the model returned, and the splat Brush trained on those frames.

The author workspace: stages at left, the point cloud and camera rig in the middle, the camera view at right.

Adding elevations

A single orbit at one elevation only ever sees the subject from one height, and the splat shows it. The fix turned out to be pleasantly recursive.

Once I have a splat from the first ring, I render that splat from two more elevations, above and below, and send those renders through the same reference-to-video model. Because the shape already exists, the model is no longer inventing form, only texture and material on geometry that is already there. Those come back as two more passes, and all three train together into a much more complete result.

The extra-passes workspace. The inherited settings at top left cannot be edited, which is what keeps every ring on one camera.

The upper ring.

The lower ring.

The tool

The first version put six stages in one tab and grew extra tabs as problems appeared. The rebuild is a proper package with one rule that has outlasted everything else: if a condition should stop you, it disables the primary button rather than printing red text somewhere. The console logs every click and the full path of every file written or sent, which is the only reason a failed run is ever diagnosable afterwards.

The first version, with status conveyed by the colour of some text.

The early passes tab.

The retime editor, which resamples a mismatched clip. It works, and doing the retime properly in After Effects works better.

The refine tab, wrapping SplatFormer.

None of it cares what the subject is, which is why I have been testing on typography as much as on creatures. If it reads as a coherent form under a single hard light, it will lift, orbit and train the same way.

The shot step on designed typography.

Limitations

Your mileage will vary, because at the centre of this sits a video model that hallucinates. No two elevation passes come back quite the same, and some subjects simply do not reproduce well.

With Wan 3 and MiniMax 3 you will see banding as you orbit the finished splat, textures shifting and going muddy at certain angles. That is the COLMAP poses disagreeing with the pixel data in those frames, and it shows up as ghosting. I have not run into it with LTX 2.3, which is the tradeoff for its lower realism.

The one I have not solved is black patches around the silhouette, in the same places where the poses and the pixels do not agree. When I measured what was actually happening, the gaussians there were not missing, they were being driven to negative values in their view-dependent colour and clamping to black. Turning the spherical harmonic degree down reduces it. Nothing I have tried removes it.

All of this runs on local models plus API calls to fal.ai, on one 12 GB card. I hope it is a useful starting point for your own version of it.