Affine-Invariant Patches

Mikolajczyk and Schmid advanced a method of finding salient points in images and adapting them to the local texture so that they can be found repeatably, even under large changes of viewpoint. The adapted points should actually be thought of as patches, because they include a description of the shape and scale of the surrounding texture, as well as the position of the point itself. Affine-invariant patches form the basis for our modeling method.

We find affine-invariant patches in two steps:

Below are some results of our implementation.

Repeatability Tests

Mikolajczyk and Schmid prescribe a test for the quality of the affine-adaptation based on the repeatability of the points it finds. We assume that the correct homography H between two images L and L' is given. A point x in L is repeated if its projection x' in L' is within some threshold distance of the nearest point in L'. For the tests below, the threshold is 3 pixels. The repeatability rate is given as a percent: the ratio between the number of actual repeated points and the number of points that could possibly repeat between the two images.

In addition to testing if the position repeats, it is also important to test if the shape repeats. Let U be the 2x2 shape matrix associated with x in L. U induces an ellipse in L with equation
y^TU^TUy=1.
This shape can be projected into L' and compared with the ellipse induced by the shape matrix associated with the nearest point in L'. To compare ellipses, we find the ratio of their intersection to their union. The choice of shape threshold determines whether the ellipses are sufficiently similar.

Test Images

We tested repeatability on the graffiti6 dataset collected by Krystian Mikolajczyk. The set includes ground truth homographies. Here are the images:

We tested 12 different combinations of point detectors and affine-adaptation settings. Six of these used binaries graciously provided by Krystian Mikolajczyk. The other six used our own implementations. All the tests were done with the program vibes_test.ln provided by Krystian. The methods tested (along with their abbreviations) were:

HK -- Harris detector provided by Krystian. The program and options were: detect_points.ln -mhar -lap
HK/F -- Harris points with only shape adaptation. ("F" stands for "fixed position".)
HK/M -- Harris points with full adaptation: position, scale, and shape. ("M" stands for "mobile position".)
H -- Harris detector using our implementation. The threshold factor was tuned to return approximately the same number of points per image as detect_points.ln.
H/F -- Harris points with only shape adaptation.
H/M -- Harris points with full adaptation.
SK -- Hessian detector provided by Krystian: detect_points.ln -mhes -lap
SK/F -- Hessian points with only shape adaptation.
SK/M -- Hessian points with full adaptation.
S -- Hessian detector using our implementation. Again, the threshold factor was tuned to return about the same number of points as detec_points.ln. In the case of the Hessian detector, however, we tended to get somewhat fewer points for most of the images.
S/F -- Hessian points with only shape adaptation.
S/M -- Hessian points with full adaptation.

The following tables give percent repeatability between img1 and each of the other images.

Shape threshold = 0.1:

HK HK/F HK/M H H/F H/M SK SK/F SK/M S S/F S/M
img2 2.18 0.38 5.6 10 10 24 2.1 0.51 14 11 16 38
img3 0 0.78 2.1 0 2.5 9 0 3.5 6.5 0 6.3 25
img4 0 0.7 1 0 1 4.7 0 0.87 2.8 0 2.2 16
img5 0 0 0.89 0 0.45 1.7 0 0.1 1.5 0 0.67 7.8
img6 0 0 0 0 0 0.38 0 0.1 0.37 0 0.22 3.1
img7 14 3.9 1.5 15 8 7 20 6.9 5.4 11 16 17
img8 0 0 0.3 0 0.15 2.1 0 0.15 0.92 0 0.89 9.8

Shape threshold = 0.55:

HK HK/F HK/M H H/F H/M SK SK/F SK/M S S/F S/M
img2 46 45 38 49 53 62 58 56 47 45 56 56
img3 31 27 27 36 37 44 47 40 34 38 42 41
img4 24 21 21 26 26 33 33 27 21 28 28 34
img5 14 10 13 15 14 20 19 15 14 15 17 22
img6 2.7 3.9 4.9 4.4 7.5 8.3 3.8 5.5 5.2 3.5 5.7 10
img7 31 28 21 31 31 29 43 42 31 18 37 36
img8 11 8.9 11 14 14 16 17 13 10 13 17 22

	HK	HK/F	HK/M	H	H/F	H/M	SK	SK/F	SK/M	S	S/F	S/M
img2	2.18	0.38	5.6	10	10	24	2.1	0.51	14	11	16	38
img3	0	0.78	2.1	0	2.5	9	0	3.5	6.5	0	6.3	25
img4	0	0.7	1	0	1	4.7	0	0.87	2.8	0	2.2	16
img5	0	0	0.89	0	0.45	1.7	0	0.1	1.5	0	0.67	7.8
img6	0	0	0	0	0	0.38	0	0.1	0.37	0	0.22	3.1
img7	14	3.9	1.5	15	8	7	20	6.9	5.4	11	16	17
img8	0	0	0.3	0	0.15	2.1	0	0.15	0.92	0	0.89	9.8