SPy uses IPython to provide GUI windows without blocking the interactive python interpreter. To enable this, you must start IPython in “pylab” mode. If you have already set your matplotlib backend to either “WX” or “WXAgg” (see note below) in your matplotlibrc file, you should be able to start IPython for SPy like this:
You can also set the backend explicitly when starting IPython this way:
If you are not calling GUI functions (calling save_rgb doesn’t count as a GUI function), then it is not necessary to run IPython - you can run the standard python interpreter.
The SPy imshow function is a wrapper around the matplotlib function of the same name. The main differences are that the SPy version makes it easy to display bands from multispectral/hyperspectral images, it renders classification images, and supports several additional types of interactivity.
The imshow function produces a raster display of data associated with an np.ndarray or SpyFile object.
In : from spectral import * In : img = open_image('92AV3C.lan') In : view = imshow(img, (29, 19, 9))
When displaying the image interactively, the matplotlib button controls can be used to pan and zoom the displayed images. If you press the “z” keyboard key, a zoom window will be opened, which displays a magnified view of the image. By holding down the CONTROL key and left-clicking in the original window, the zoom window will pan to the pixel clicked in the original window.
Changed in version 0.16.0: By default, imshow function applies a linear histogram stretch of the RGB display data. The the color stretch can be controlled by the stretch, bounds, and stretch_all keyword to the imshow function (see get_rgb for the meaning of these keywords). To adjust the color stretch of a displayed image, the set_rgb_options method of the ImageView object can be called.
RGB data limits for a displayed image can be printed from the __str__ method of the ImageView object:
In : print view ImageView object: Display bands : (29, 19, 9) Interpolation : <default> RGB data limits : R: [2054.0, 6317.0] G: [2775.0, 7307.0] B: [3560.0, 7928.0]
To display the ground truth image using imshow, set the classes argument in the imshow function:
In : gt = open_image('92AV3GT.GIS').read_band(0) In : view = imshow(classes=gt)
It is also possible to switch between displaying image bands and class colors, as well as displaying class color masks overlayed on the image data display. To do this, specify both the data and class values when calling imshow:
In : view = imshow(img, (30, 20, 10), classes=gt)
The default display mode is to show the image bands. Press “d”, “c”, or “C” (while focus is on the image window) to switch the display to data, classes, or class overlay, respectively. Setting the display parameters can also be done programatically. For example, to display the image with overlayed class masks, using an alpha transparency of 0.5, type the following commands after calling imshow:
In : view = imshow(img, (30, 20, 10), classes=gt) In : view.set_display_mode('overlay') In : view.class_alpha = 0.5
The ImageView window provides the ability to modify pixel class IDs interactively by selecting rectangular regions in the image and assigning a new class ID. Applying a class ID to a rectangular region is done using the following steps:
- Call imshow, providing an initial array for the classes argument. This must be a non-negative, integer-valued array. A class ID value of zero represents an unlabeled pixel so to start with a completely unlabeled image, pass an array of all zeros for the classes argument.
- While holding the SHIFT key, click with the left mouse button at the upper left corner of the rectangle to be selected. Drag the mouse cursor and release the mouse button at the lower right corner of the rectangular region. Note that releasing the SHIFT key before the mouse button is released will result in cancellation of the selection operation.
- With focus still on the ImageView window, enter the numeric class ID to apply to the region. The class ID can contain multiple digits. The digits will not be echoed on the command line. Press ENTER to apply the class ID. The command line will request confirmation of the opertion. Press ENTER again to apply the class ID or press any other key to cancel the operation.
Classes can be assigned form either a main ImageView window or from an associated zoom window. While the selection tool only produces rectangular regions, you can assign classes to non-rectangular regions by first assigning the class ID to a super-rectangle covering all the pixels of interest, then reassigning sub-rectangles back to class 0 (or whatever was the original class ID).
To get help on ImageView keyboard & mouse functions, press “h” while focus is on the image window to display all keybinds and mouse functions:
Mouse Functions: ---------------- ctrl+left-click -> pan zoom window to pixel shift+left-click&drag -> select rectangular image region left-dblclick -> plot pixel spectrum Keybinds: --------- 0-9 -> enter class ID for image pixel labeling ENTER -> apply specified class ID to selected rectangular region a/A -> decrease/increase class overlay alpha value c -> set display mode to "classes" (if classes set) C -> set display mode to "overlay" (if data and classes set) d -> set display mode to "data" (if data set) h -> print help message i -> toggle pixel interpolation between "nearest" and SPy default. z -> open zoom window See matplotlib imshow documentation for addition key binds.
To customize behavior of the image display beyond what is provided by the imshow function, you can create an ImageView object directly and customize it prior to calling its show method. You can also access the matplotlib figure and canvas attributes from the axes attribute of the ImageView object returned by imshow. Finally, you can customize some of the default behaviours of the image display by modifiying members of the module’s SpySettings object.
To save an image display to a file, use the save_rgb function, which uses the same arguments as imshow but with the saved image file name as the first argument.
In : save_rgb('rgb.jpg', img, [29, 19, 9])
Saving an indexed color image is similar to saving an RGB image; however, save_rgb is unable to determine if the image being saved is a single-band (greyscale) image or an indexed color image. Therefore, to save as an indexed color image, the color palette must be explicitly passed as a keyword argument:
In : save_rgb('gt.jpg', gt, colors=spy_colors)
The image display windows provide a few interactive functions. If you create an image display with imshow (or view) and then double-click on a particular location in the window, a new window will be created with a 2D plot of the spectrum for the pixel that was clicked. It should look something like this:
Note that the row/col of the double-clicked pixel is printed on the command prompt. Since there is no spectral band metadata in our sample image file, the spectral plot’s axes are unlabeled and the pixel band values are plotted vs. band number, rather than wavelength. To have the data plotted vs. wavelength, we must first associated spectral band information with the image.
In : import spectral.io.aviris as aviris In : img.bands = aviris.read_aviris_bands('92AV3C.spc')
Now, close the image and spectral plot windows, call view again and click on a few locations in the image display. You will notice that the x-axis now shows the wavelengths associated with each band.
Notice that the spectra are now plotted against their associated wavelengths.
The spectral plots are intended as a convenience to enable one to quickly view spectra from an image. If you would like a prettier plot of the same data, you can use SPy to read the spectra from the image and create a customized plot by using matplotlib directly.
In the context of hyperspectral imagery, a 3D hypercube is a 3-dimensional representation of a hyperspectral image where the x and y dimensions are the spatial dimensions of the image and the third dimension is the spectral dimension. The analytical utility of hypercube displays is debatable but what is not debatable is that they look extremely cool.
After calling view_cube, a new window will open with the 3D hypercube displayed. After shifting command focus to the newly created window, you can use keyboard input to alter the view of the hypercube. Using keyword arguments, you can change the image displayed on the top of the cube, as well as the color scale used on the sides of of the cube.
In : view_cube(img, bands=[29, 19, 9])
Mouse Functions: ---------------- left-click & drag -> Rotate cube CTRL+left-click & drag -> Zoom in/out SHIFT+left-click & drag -> Pan Keybinds: --------- l -> toggle light t/g -> stretch/compress z-dimension h -> print help message q -> close window
If the window opened by view_cube produces a blank canvas (no cube is displayed), it may be due to your display adapter not supporting a 32-bit depth buffer. You can reduce the size of the depth buffer used by view_cube and view_nd to a smaller value (e.g., 16) by issuing the following commands:
In : import spectral In : spectral.settings.WX_GL_DEPTH_SIZE = 16
Since hyperspectral images contain hundreds of narrow, contiguous bands, there is often strong correlation between bands (particularly adjacent bands). To increase the amount of information displayed, it is common to reduce the dimensionality of the image to a smaller set of features with higher information density (e.g., by principal components transformation). Nevertheless, there are typically still many more than three features remaining in the transformed image so the analyst must decide which are the “best” three features to display to highlight some aspect of the data set (e.g., separability of the spectral classes). It is desirable to be able to quickly switch displayed features to examine many combinations of features in a short time span.
In most cases, the display will be more useful by first performing dimensionality reduction prior to viewint the data (e.g., by selecting some number of principal components).
In : data = open_image('92AV3C.lan').load() In : gt = open_image('92AV3GT.GIS').read_band(0) In : pc = principal_components(data) Covariance.....done In : xdata = pc.transform(data) In : w = view_nd(xdata[:,:,:15], classes=gt)
A new window should open and appear as follows:
The python prompt will display the set of keyboard and mouse commands accepted by the ND window:
Mouse functions: --------------- Left-click & drag --> Rotate viewing geometry (or pan) CTRL+Left-click & drag --> Zoom viewing geometry CTRL+SHIFT+Left-click --> Print image row/col and class of selected pixel SHIFT+Left-click & drag --> Define selection box in the window Right-click --> Open GLUT menu for pixel reassignment Keyboard functions: ------------------- a --> Toggle axis display c --> View dynamic raster image of class values d --> Cycle display mode between single-quadrant, mirrored octants, and independent octants (display will not change until features are randomzed again) f --> Randomize features displayed h --> Print this help message m --> Toggle mouse function between rotate/zoom and pan modes p/P --> Increase/Decrease the size of displayed points q --> Exit the application r --> Reset viewing geometry u --> Toggle display of unassigned points (points with class == 0)
The following figure shows the same window after the display has been rotated, unlabeled pixels (white) have been supressed, and the displayed pixel size has been increased:
To facilitate interactive visual analysis of high-dimensional images, the view_nd function provides several modes for displaying high-dimensional data in a 3-dimensional display window:
The user has the option of switching between display modes. Which display mode is best may depend on a number of factors relating to the data set used or the system on which the application is run. For images with sufficiently large numbers of pixels, the system’s display refresh rate may begin to suffer in mirrored-octant or independent-octant mode because each image pixel must be rendered 8 times. When performance is not an issue, the latter two modes provide more simultaneous projections of the data that may enable a user to more quickly explore the data set.
The initial 3D view of the rendered image data will almost certainly not be the ideal 3D viewing geometry; therefore, the window allows the user to manipulate the view by using the mouse input to zoom, pan, and rotate the viewing geometry. The user can also increase/decrease the size of displayed image pixels and toggle display of unlabeled (class 0) pixels.
While pixel color allows the user to distinguish between classes of displayed pixels, it may not be obvious to which class a given color is associated. To identify the class of a specific pixel, the user can click on a pixel and the python command line will print the pixel’s location within the image (row and column values) and the number (ID) of the class of the pixel. Since clicking on a single screen pixel is difficult, the user can first increase the size of the displayed image pixels such that the image pixel can easily be selected with the mouse.
It is common for an analyst to want to review the results of unsupervised classification and subsequently combine pixel classes or reassign pixels to new or different classes. Similarly, an analyst may want to assess ground truth data and possibly reassign pixels that were erroneously included in a class when the ground truth image was developed. To support this capability, the N-D window allows the user to click and drag with the mouse to select a rectangular region and then reassign all pixels lying within the defined box. The application will not only reassign the pixels visible within the selection box but also all pixels within the projection of the box through the 3D space.
When view_nd is called, it returns a proxy object that has a classes member that represents the current array of class values associated with the ND display. The proxy object can be used to access the updated class array after image pixels have been reassigned through the ND window. The proxy object also has a set_features method to specify the set of features displayed in the 3D window.
Before reassigning pixel class values, press the “c” key to open a second window displaying a raster view of the current image pixel class values. Then, when you reassign pixel classes in the ND window, the class raster view will update automatically, providing and indication of which pixels in the source image were modified.