Shape filters and compound shapes

[mihe]

I'm trying to use a ShapeFilter together with CastShape and CollideShape to conditionally exclude certain sub shapes of the shape provided to the query, which might or might not be a compound shape, from partaking in that particular query.

To give a more concrete example, say I have a StaticCompoundShape consisting of a SphereShape and a BoxShape, and that compound shape is being CastShape across the world. I'd then want to use a ShapeFilter to pretend like the SphereShape isn't part of that compound shape, and the output from CastShape would then be as if I was just shape-casting the BoxShape, while still respecting whatever transform it has within its compound shape of course.

In theory it seems like this should be possible, considering the signature of ShouldCollide(SubShapeID, SubShapeID), that seemingly lets you prevent collisions based on either of the two shapes. In practice however, I'm finding it difficult to identify the shape when using only its sub shape ID, because when using something like GetSubShapeUserData on what should be the compound shape's sub shape ID, I end up getting the user data for random sub shapes within the compound shape, instead of the compound shape's own user data.

I set up the above mentioned scenario as a Samples test, which you can find here.

Is there a bug here or am I doing something wrong, or is there perhaps some other way to go about this?

I would expect to see this:

CastShapeTest_Expected.mp4

But instead I end up seeing this:

CastShapeTest_Actual.mp4

(The video with the expected outcome was achieved by swapping the order of the AddShape calls, which results in the compound shape being identified as the box shape instead of the sphere shape, thus preventing it from being filtered out. So while it generates the correct output it's still not behaving correctly.)

[DocAce]

when using something like GetSubShapeUserData on what should be the compound shape's sub shape ID, I end up getting the user data for random sub shapes within the compound shape, instead of the compound shape's own user data.

I set up the above mentioned scenario as a Samples test, which you can find in a dedicated branch here.

I've been successfully using that functionality quite extensively, my code looks like this:

bool XShapeFilter::ShouldCollide(const JPH::SubShapeID& inSubShapeID2) const
{
	if (inSubShapeID2.IsEmpty()) {
		return true;
	}
	JPH::BodyLockWrite lock(BodyLockInterface, mBodyID2);
	if (lock.Succeeded()) {
		const JPH::Body* body = &lock.GetBody();
		if (size_t userdata = body->GetShape()->GetSubShapeUserData(inSubShapeID2)) {
			const XCompoundChildShape* childshape = reinterpret_cast<const XCompoundChildShape*>(userdata);
			// a bunch of code that checks whether the filter properties allow collisions with this child chape
		}
		return true;
	}
	return false;
}

bool XShapeFilter::ShouldCollide(const JPH::SubShapeID& inSubShapeID1, const JPH::SubShapeID& inSubShapeID2) const
{
	return ShouldCollide(inSubShapeID1) && ShouldCollide(inSubShapeID2);
}

[mihe]

For a moment I thought maybe the IsEmpty() check was the key to the problem, since I'd completely missed the concept of an empty sub shape ID. However, while the sub shape ID for the compound shape does seem to report as being empty, so does the sub shape ID for the sphere in my case, leading to it not getting filtered out as expected.

There's a comment on IsEmpty() suggesting that it might produce false positives, so I'm not quite sure what to think.

Also, that code seems to assume that both sub shapes belong to mBodyID2, which I don't think is applicable in the general case, unless I've misunderstood some part of this whole setup.

[DocAce]

Oh, you're right, that function should (in my case) actually ignore inSubShapeID1. It only works because these shapes are never compound in our game.

[jrouwe]

Hello,

There's indeed a problem here. We call ShouldCollide at every level in the hierarchy, so for a compound with a box you'll get 2 callbacks. The problem is that you currently cannot determine if you're at the compound stage or at the box stage. The SubShape ID is basically a string of bits where at each level some bits are pushed in from the right. The compound shape has 2 shapes, meaning it uses 1 bit.

• First you get a call comparing the compound shape, this is the root so you get 0xffffffff (empty)
• Then you get a call comparing the box shape, this is the first sub shape so the last bit of the sub shape ID is 0: 0xfffffffe (box/sphere shapes don't have any bits because they're a single shape)
• Then you get a call comparing the sphere shape, this is the 2nd sub shape so the last bit is 1, sub shape ID is 0xffffffff ((or 0xffffffff << 1) | 1)

So you see you cannot distinguish the compound from the box. I could reserve an extra bit for the compound shape but we're already really short on bits. I think the best alternative is to pass in the shapes to the ShouldCollide filter so that you can check if you're at a leaf shape or not (see #474). The thing I don't like about the solution is that now you have a sub shape ID that doesn't belong to the shape that's being passed in, but tracking the relative sub shape ID is going to be costly. With this change you can change your code to:

	bool ShouldCollide(const Shape *inShape1, const SubShapeID &inSubShapeID1, const Shape *inShape2, const SubShapeID &inSubShapeID2) const override
	{
		//return inShape1->GetType() == EShapeType::Compound || (ShapeIdentifier)mCastShape.GetSubShapeUserData(inSubShapeID1) != ShapeIdentifier::Sphere; <-- this works too
		return inShape1->GetUserData() != (uint64)ShapeIdentifier::Sphere;
	}

Oh and b.t.w. thanks for the demo, this makes debugging so much easier!

[mihe]

I think the best alternative is to pass in the shapes to the ShouldCollide filter

That would certainly solve my immediate problems at least. :)

The thing I don't like about the solution is that now you have a sub shape ID that doesn't belong to the shape that's being passed in, but tracking the relative sub shape ID is going to be costly.

I must admit that this makes it sound like the sub shape ID wouldn't be terribly useful in this context. What's the reasoning behind passing the IDs at all and not just passing the shapes? Is it to cater to stuff like triangles inside meshes, since they aren't shapes themselves? I honestly wouldn't have expected things like that to even be passed into ShouldCollide to begin with, but maybe that's a valid use-case.

One option is to change the callback so that it will tell you if you're colliding with a leaf shape or not but then the question is what info you can get at non-leaf shape levels. Let me know if you have a better idea.

At the risk of making it even messier, would it be better to split it up into two separate ShouldCollide, one for actual concrete shapes, where you only receive the Shape, and then another overload for things that aren't concrete shapes, like mesh triangles, where you receive both the Shape and the SubShapeID? The latter overload could then be more explicit about the Shape provided always being a parent shape of the SubShapeID.

Disclaimer: I (mostly) have no idea what I'm talking about.

Oh and b.t.w. thanks for the demo, this makes debugging so much easier!

Glad to hear it came of some use!

[jrouwe]

Thanks for the suggestions! The two overload solution has the problem that if you hit a box that's repeated 1000 times in a compound shape that you won't be able to tell the difference.

The best solution I came up with so far is to:

• Remove the per triangle callbacks (you can easily filter in the AddHit callback, you get the same info there and the overhead of calling the shape filter callback for each triangle is substantial)
• Keep the rest of the interface as in Passing the shapes that are colliding into the ShapeFilter #474. This means that you get the shape pointer and the sub shape ID that always takes you from the root shape to shape that's being passed to the callback. This lets you uniquely identify which shape you're colliding with.

[mihe]

The two overload solution has the problem that if you hit a box that's repeated 1000 times in a compound shape that you won't be able to tell the difference.

Right! I didn't think of that.

• Remove the per triangle callbacks (you can easily filter in the AddHit callback, you get the same info there and the overhead of calling the shape filter callback for each triangle is substantial)
• Keep the rest of the interface as in Passing the shapes that are colliding into the ShapeFilter #474. This means that you get the shape pointer and the sub shape ID that always takes you from the root shape to shape that's being passed to the callback. This lets you uniquely identify which shape you're colliding with.

That all sounds reasonable to me. Needing to deal with individual triangles in ShapeFilter sounds like too niche/costly of a use-case to burden every user with, especially if it can be achieved with a hit collector already.

[jrouwe]

I merged the changes in #474, thanks for your help!

[mihe]

Wonderful! It works like a charm.

Thank you yet again for such a quick turnaround.