← Back to Home

AdaPixFlow

Learning Continuous-Time 3D Dynamics Across Scenes from Sparse Multi-view Observations
Preprint, 2026
Department of Computer Science, Purdue University
A held-out scene (PlasticineFall). The object pixels of an observation move along the predicted velocity field to a step that no camera observed.

TL;DR

We learn one continuous-time dynamics operator across scenes. We fuse sparse multi-view RGB-D observations into a single voxel grid in world coordinates, predict a 3D velocity field from them, and advect the object pixels along it. This tells us where objects are between observations and after the last one. We train the operator once, and it runs on new scenes and datasets without per-scene optimization.

The Task

Calibrated RGB-D cameras record the scene every few steps. We predict the 3D state of the objects at the steps in between and after the last observation, and score the predictions against two cameras that no method sees.

Method

AdaPixFlow pipeline: volumetric fusion, the network predicting node velocity fields, and advection
We fuse the observed views into a voxel grid, the network predicts velocity node fields from the fused observations around a gap, and we advect the object points along the resulting field. One network is trained across scenes and evaluated on held-out ones.
1 · observe
Calibrated RGB-D views at the observed steps.
2 · fuse
We back-project every pixel and fuse all views into one grid in world coordinates, in closed form.
3 · predict a velocity field
A 3D U-Net maps the fused observations to a continuous-time velocity field.
4 · advect
The object points follow the field to any time between or after the observations.

Pixels that flow

For a gap that starts at the observation \(k_0\), the context holds the fused states before, at and after the gap, a flag \(e\) that says whether the next observation exists, and the spacing \(s\). From it, the network predicts a velocity field, and every object point follows

dx(t)dt=vθ(x(t),t∣c),c=(zk0,zkp,zk1,e,s).

We train the operator through the advection, comparing the advected points with the observed object points as unordered sets. Training needs no velocity labels and no point correspondences, which suits deformable objects whose surface points have no persistent identity.

Explore the Predictions in 3D

Loading the scene…
drag to rotate · scroll to zoom

Predicted 3D object points at every step that no camera observed, over the fused background. "Both" overlays the ground truth in grey. Points are subsampled for size.

Results

In-between prediction on held-out scenes

Each scene is observed every fourth step, and we show only the steps in between, which no method observes. FreeGave fits each scene separately, Shape of Motion reconstructs the observed frames, and AdaPixFlow is the same trained model on every scene.

ElasticFall: an elastic object falls and bounces.
CrankPush: a rotating crank pushes an object.
DropInLiquid: a solid floats in liquid.

Future prediction

Kubric-4D, zero-shot: the objects keep falling after the last observation.

Zero-shot on other benchmarks

We apply the same model, trained only on PhysInOne, to Spring-Gaus, Kubric-4D and PhysGaia without fine-tuning. These datasets differ in materials, camera rigs and scale.

Spring-Gaus, torus.
Spring-Gaus, rope.
Kubric-4D, rigid objects.
PhysGaia, viscoelastic objects.

Against 4D reconstruction, tracking and per-scene dynamics

Predicted 3D object points of every method on the comparison scenes, with F1 per cell
MethodInputIn-betweenFuture
FreeGaveRGB, D0, M69.131.5
TRACERGB, D0, M29.928.4
Shape of MotionRGB-D, M, 1 view67.331.2
MoScaRGB-D, M, 1 view43.328.6
SpatialTrackerV2RGB-D, M87.850.7
OpenD4RTRGB-D, M84.045.9
AdaPixFlowRGB-D, M90.058.4

F1 (%) of the predicted 3D object points at a 2.5 cm threshold, mean over the six PhysInOne comparison scenes. Every method reads the observed views only, and two cameras are held out of every input.

Citation

@article{viswanath2026adapixflow,
  title={AdaPixFlow: Learning Continuous-Time 3D Dynamics Across Scenes from Sparse Multi-view Observations},
  author={Viswanath, Hrishikesh and Bera, Aniket},
  journal={arXiv preprint},
  year={2026}
}