Josh SimmonsI was investigating how the decl stuff works in id tech, it seems when you override an array in a derived instance, you set the size of the array, and then set individual entries. So if you were to add later an additional array element to the base instance, it wouldn't propagate to the derived unless you I guess, clear the override, and then append again. I wonder if they have tools to detect when you make a change which would break other data, or whether it's all yolo all the time.
In USD I guess they have a bunch of different primitives for overlaying data, which is complex. I wonder if an ideal solution isn't to copy "default" values into the derived object always, and then mark them with a special flag that indicates they are defaulted. So if you make changes to a base class and the defaults no longer match it allows you to decide whether to take those changes into your derived instance. (Plus dependency information so you can see the impacts across the whole project)
Geoff Evans@dotstdy IME there are few “right answers” for these questions, only trade offs that are hard to reason about from the present. What matters more is users knowing and relying on specific behaviors.
A big benefit of USD is standardizing behavior to work a single way.
IME with past engines all inherited properties are sparsely persisted in the child assets. So defaults change everywhere they weren’t specified. You can then make features to “freeze” assets to bake defaults if the user wants.
@gorlak yeah that's how it works for us also, but it's kind of broken / confusing / unclear in the case where you want to say, append to a list in a derived class. I'm uncertain whether USD cooks a bit too hard with the strength hierarchy of "overrides"
@dotstdy have you ever heard of indexing lists with floats? So when you insert in a child it can be resolved more cleanly when flattening things out.
@gorlak yeah, that's one cool approach too, but then you still have the weird case of removing an element from the base array. There's lots of ways you can potentially add tools to automatically resolve, but I can't help but think about the "duplicate + link" snapshot style. Iirc this is basically how Dreams handles linking other content into your own Dream, and you can then look at your Dream and it says "there's new versions of x, y, z". However in a large game I'd be more interested in "push"
@gorlak but also, this is more thinking from the perspective of a property editor for game data, than it is thinking about the usd use-case of building a hugely complex scene out of lego pieces with overrides and templates and whatnot.
Mikko Mononen@dotstdy @gorlak The float approach is tricky, since every add-in-between needs a bit. There are ways to deal with that, but the fractional indexing may need arbitrary amount of bits.
Another way is to use ID per array element. Some of the solutions that does the array merging "right" feel surprisingly random in action (e.g. interlacing).
I made a system a while back which used IDs for array elements, and stored a flag on the derived data if it was added/changed/deleted.
The merging was simply looking for runs of changes vs keeps, and favored the explicit changes. It was able to keep the changes at the beginning and end of the array (which was the most common case).
It did not go through a production, so I cannot say if others liked how it worked, I did :)
@mikkomononen @gorlak Detecting runs is a neat idea, the other thing you could do is explicitly add a sort key for those cases where element order matters, and thus make it a user problem. The classic ____000_my_important_goes_first kind of thing. :') That is, you'd have id's for update v.s. append v.s. remove, and then a sort key for ordering.
@mikkomononen @gorlak An explicit sort key would let you do slightly mad things like have a derived instance merely re-arrange elements in a list, and still have it react to changes in the base object. At least so long as they didn't also try to re-order things. That situation where both sides edit is why part of me is drawn to the Dreams explicit "you have data that somebody else has changed, how do you want to proceed" thing, or the reverse "you are changing data that affects objects X and Y"
@dotstdy @gorlak
Yeah, the unordered set behavior is super simple, things get tricky when you want specific ordering. I put the sorted array to the unordered, since we dont care during the merge.
Most use cases I have had to solve, the ordering did not matter, but it sure is annoying if your arrays shuffle around seemingly at random. Or that new things appear at random position.