2024 BrainChip Discussion, page-5362

This is in relation to an auto car with all the sensors following a trailer with a tree in it .The Auto car is swerving all over the place because it can’t work out what’s happening. This is a good example of why large language models are so powerfull for ADAS, without the LLM, you cannot really solve a moving tree, but when you look at the comment here, you understand immediately how the LLM address the "common sense" part of driving.
This swerving may be due to a known challenge in computer vision/ML-based object detection: When a perception model is faced with an object it was not previously trained to classify as a distinct class, it will not return ‘unknown object’ as a prediction. Instead, it will usually sort the localized object into one of many pre-defined object classes it was trained on - the one with the highest confidence, even if that absolute confidence is low.

So one reason for why this Waymo is swerving in and out of its lane could be a loop as follows:

1. From a position at 180° behind the car/trailer (centered in the ego lane), the object is classified as a tree with a solid barrier behind it.
2. A possible path planning outcome could be ‘change lanes to avoid this obstacle’.
3. Upon leaving the lane, the Waymo’s sensors pick up the same object from a side angle, which might favor a re-classification as car + trailer.
4. Path planning may thus decide to ‘re-center in lane and follow’, leading back to the condition in step 1.

This extends to a theoretically infinite test space of (known) objects in previously non-defined/non-considered combinations: Remember the two Waymos which crashed into the same tow truck pulling a truck at an angle in Phoenix? There are approaches to mitigating this challenge, e.g. occupancy grids, but as we can see there’s still work to be done for ‘big picture’ scenario interpretation … It's both an exciting and humbling time to be working in perception for embodied AI!