Stanford AA274A Lecture 13: How Computer Vision Moved from Hand-Engineered Pipelines to Convolutional Networks

Prof. Marco Pavone opens by saying that the course is fairly complex to teach because of the mathematics and the software involved. He describes it as an entire infrastructure for robotic software engineering rather than a simple script, and he adds that hardware is part of the picture as well.

He then explains the role of the sections. According to him, they are not meant to deepen understanding of the lecture material. They complement the lectures by teaching software engineering skills that are hard to convey through slides and are learned mainly through hands-on activities. This year's sections use lightly graded hands-on work under the guidance of the course assistants.

Students had reported connectivity and software installation problems in the sections. Pavone said that a good share of a roboticist's time goes to debugging connectivity issues, and that the teaching staff is working to fix these problems for later sections.

Pavone reports that some students said the homework assignments were too long. Homework 1 and 2 were perceived as the right length, but Homework 3 was seen as too long for the time allotted to it. He says the staff agrees with this assessment.

The staff discussed the issue at its weekly meeting with all the course assistants, and the deadline for Homework 3 has been extended to November 5. Pavone says a one-week extension better suits the length of the assignment and also fits the lecture schedule.

The announcement will be posted on Canvas and Piazza the same day. He also mentions that feedback from SCPD students led the staff to increase the number of office hours available to them.

Pavone introduces object detection and classification as the problem of finding whether an object of interest is present in an image and identifying what the object is. Before 2012, the general solution was to describe one region of an image, classify that region, shift the focus to another part of the image, and repeat the process. This iterative procedure was how systems handled images of different sizes and scales.

For describing a region, the methods he names include histogram of oriented gradients (HOG), SIFT, a keypoint description method, patch similarity, and code words. For classifying those descriptions, methods from applied probability, statistics, and machine learning were used, such as naive Bayes, nearest neighbor, and support vector machines.

He says the listener does not need to memorize these names. They serve as examples of the methodologies used for the classification subproblem. The shift to the next window was handled with sliding-window approaches, in which the system processes one part of the image and then moves on to the next.

Pavone describes scene understanding as understanding what is happening in the environment around a robot. The problem he focuses on is semantic segmentation, which means classifying every observed pixel in the scene. He shows an example in which the pixels of a simulated camera image are colored according to their class, such as road, vehicle, sidewalk, pedestrian, or traffic sign.

He explains that these classes matter to a robot because they encode regions of importance. A robot needs to distinguish a traffic light from a tree, for instance, and to know where to direct its attention.

He says that before 2016 there were two general classes of methods for this problem: clustering-based and graph-based. The lecture goes on to examine examples of both.

In the clustering approach, each pixel is mapped to a five-dimensional feature vector made of X, Y, R, G, and B values, and K-means clustering is then run in that feature space. Pavone explains that the X and Y dimensions encode spatial locality: if one part of an image is close to another, the two are likely part of the same object in 3D, unless there is a sharp gradient such as an edge between them.

The RGB dimensions encode the intuition that similar things have similar colours, which captures visual locality rather than only spatial locality. He gives the example of a red chair: a pixel on it and a pixel ten pixels away with the same red colour are probably on the same chair.

He applies this to a beach scene taken from a plane or drone. With K equal to three, the simple clustering already separates most of the island from shallow water and deep water, which he calls a strong result for such a simple idea.

Pavone explains that the value of K has to be chosen carefully. If K is one, the result is a single uniform class, which is pointless because everything is labeled the same. If K is three, the segmentation is coarse. If K is much larger, the segmentation becomes too fine, and it may no longer be possible to tell meaningful classes apart.

He describes the goal as picking a Goldilocks K that does not destroy too much information, sitting at the elbow point between coarse and fine segmentation. He also notes that quantizing an image into fewer classes can be seen as a form of image compression.

He then shows another scene with K equal to six. The passage ends mid-sentence as he moves on to graph-based methods.

Pavone describes the key idea of graph-based methods: images have a grid-like structure, which clustering does not explicitly use, while graph-based methods encode it by placing a probabilistic graphical model over the pixels. Each pixel is treated as an observed random variable, and the goal is to estimate a probability for the class that generated its appearance and location.

The usual pipeline starts with a first-pass classifier, such as a sliding-window detector. That classifier can be noisy, especially at edges. Pavone explains the edge noise with an example: a classifier that sees only half of an animal would be about 50% sure an animal is present. The method then performs an initial classification, an approximate global optimization to estimate the probabilistic variables, and produces the final output.

He shows a figure from a paper in which the initial prediction is quite noisy, but after passing through a probabilistic graphical model of a popular kind, the prediction becomes very accurate. He says these graphical-model methods dominated the field before about 2016 and that remnants of them are still used.

Pavone shows ORB-SLAM tracking a drone that moves noisily and rotates a lot, and notes that the method localizes well. He says it is good for localization but not for mapping. The first stage detects feature key points in an image, shown as green dots, that have stability properties, meaning they are likely part of static objects and remain detectable under changes in lighting, aperture, shadows, and rotation.

The method then performs a structure-from-motion style step to estimate the rotation and translation between consecutive images. Detecting these key points first makes the correspondence matching problem easier, because the chosen points do not change much under different conditions.

The trade-off is in mapping. Because SLAM runs only on these key points, the output is not a dense point cloud. Pavone gives a rough example: of about 2,000 pixels, there might be only around 200 key points, so only about 10% of the information is used. He adds that DSO, the direct successor method, produces a much more stable and accurate map in outdoor footage.

Pavone says much of the real engineering effort in these systems lies in maintaining global map information, since a substantial part of the contribution of these papers is how they manage it. He describes a test in one paper in which the authors walked through a subway station, visited different parts of it, and came back to where they started, the same bicycle parking spot.

The estimated trajectory should end where it started, but the accumulated error shows drift. He notes that this drift has plagued SLAM for a long time and always will, because of the nature of the problem.

One remedy is loop closure. When the system detects that it has returned to a place it has already seen, it can perform a post-hoc correction of its trajectories and of the map built up to that point. Pavone also says that dense methods, unlike keypoint methods, produce a much denser map, and that the paper he describes goes on to show further examples.

Pavone says that in 2012 a large paradigm shift spun the entire field of computer vision on its head. To explain it, he first describes a parallel line of research in machine learning, which was already used in computer vision, with support vector machines as an example of a core method developed over decades.

He summarizes the core machine learning idea as predicting a quantity Y from an input X with a parameterized function, defining a loss function that measures how bad the predictor is, and minimizing that loss with respect to the parameters. He notes that this is a heavy oversimplification given only for intuition.

He describes the ImageNet Large Scale Visual Recognition Challenge, developed at Stanford's vision lab. Teams submitted algorithms through a server to a held-out test set, so no one saw the test data. Before 2012, the entries had plateaued, and Pavone cites a jump from 26% error to 16% error, which he calls almost a 50% jump in state-of-the-art performance.

Pavone says that up to this point images were treated as objects of interest, but mainly as matrices, or two- or three-dimensional tensors depending on whether colour was present. The problem, he says, was that existing machine learning methods did not take advantage of the structure of an image.

Graph-based and segmentation methods had recognized that images have spatial locality and visual locality. However, many machine learning methods at the time could not encode spatial locality or visual similarity. He presents this gap as the setting in which the 2012 results appeared.

Pavone divides a convolutional neural network into low-level layers near the raw image input, middle layers, and upper layers that are more abstracted from the image. At the lowest level, the learned filters act as edge detectors. Because they take in colour images, these filters appear in a grayscale sense, with white representing a value of plus one and black representing zero or minus one, so they subtract neighboring pixels along some orientation.

Colour edge detectors also appear, such as a blue and orange filter placed side by side, because filters have channels and different channels subtract at different times. In the middle layers, the filters look like combinations of edges. Pavone points to one that seems to detect corners, the intersection of two edges.

In the upper layers, he describes filters that look more abstract. One appears to respond to something with a red bulb on top and a silver stick below, and he says many other interesting filters appear there.

Pavone opens a visualization of a convolutional neural network that shows activation maps in three dimensions. The demo network has two convolutional layers. The first has six filters, and the second has more. A pooling operation follows, which he notes is not a convolution.

When a hand-drawn 7 is entered, the first-layer filters respond to horizontal and diagonal edges, and the top prediction is 7. A 7 drawn in a more European style is classified as a 7 only barely, with 3 as the second guess. When the input is random scribbling, the network gives confident outputs such as 6, 8, and 1. Pavone uses this to show that such networks are hard to interpret, act as a black box, and always output a class even for meaningless input.

Pavone says that one of the first use cases of convolutional neural networks was classifying mail, specifically optical character recognition on handwritten letters such as USPS post. A lot of the original work was done there, and this was one of the original datasets. He notes the practical problem: if a pen broke or the writing was unclear, a system could still send an envelope to the wrong place.

Pavone says that after 2012, many object detection methods focused on a convolutional network that outputs both regions of interest and classifications of those regions. The loss function has a classification term and a regression term, with the regression predicting bounding-box location in the image.

He says many methods now combine problems into a single CNN. He gives the example of Uber using CNNs to detect objects, track them, and predict their motion with one network. He also describes semantic segmentation as increasingly handled by generating an output image from an input image through convolutional layers, with a pixel-wise loss, in place of the earlier graph-based machinery. Probabilistic graphical models remain mainly as post-correction steps.

Pavone turns to robotics and notes that modern platforms carry a wide array of sensors and sensor data modalities. He gives the example of a Waymo vehicle with one mid-range lidar that gives tens of meters of estimation, short-range high-accuracy lidars for tasks such as parking or close approaches, five cameras for a wide visual view around the car, and further sensors for vehicle pose, including GPS, an IMU, and wheel encoders.

He says that before the deep learning advances in computer vision, integrating these modalities into a classification or a method was a considerable task. Now, he says, it can be as easy, in quotation marks, as adding extra vectors or data channels to the image and letting the CNN take that information into account, with the loss adjusted accordingly. He adds that efficiency and interpretability are still concerns being addressed.