Images In Java

Paul Cockshott

Faraday partnership

Introduction

The AWT and how to put a simple image onto the screen

Layout managers

The Jimage Class,

Summary of this section

At the end of this lecture you should have an idea of how to display a JPEG image on the screen, and how to load it into the Jimage class to carry out further image processing.

Agenda

AWT Images

Image Producers and Consumers

Jimage class

Pixel Representations

JPEG files

Overview

AWT abstract windows toolkit, supported by JavaSoft

Operating system independent layer for windowing in Java

Fiendishly obscure

Designed around requirements of images being streamed off the web

Connections

Simple image display program to show how to display a JPEG file

Pipeline model of image production

Jimages act as image consumers

Jimages allow arithmetic on image

Jimages provide output to AWT images and JPEG

How to display a picture 1

import java.awt.*;

import java.awt.image.*;

import java.util.*;

class JPEGshow extends Frame {

    ...

    static public void main(String[] args) {

        if (args.length == 1) new JPEGshow(args[0]);

        else System.err.println("usage: java JPEGshow <image file>");

    }

}

This is a standard Java Program class with a public static void main method

Constructor for JPEGshow

JPEGshow(String filename) {

        super("JPEG show Example");

        add(

            new ImageCanvas(getToolkit().getImage(filename) ),

            BorderLayout.CENTER);

        setSize(700, 540);

        show();

    }

The toolkit

Each frame has associated with it a toolkit object the provides an interface to OS specific operations.

CreateImage

CreateMenu

CreateLabel

CreateMenuBar …. etc

Roll your own ImageCanvas

class ImageCanvas extends Component {

    Image image;

    ImageCanvas(Image image)

     {this.image = image;}

    public void paint(Graphics g)

     { g.drawImage(image, 0, 0, this);}

}

Image Class

Pipeline flow model of image processing

Images are just tokens linking producers and consumers

ImageProducer Methods

addConsumer(ImageConsumer ic) This method is used to register an ImageConsumer with the ImageProducer for access to the image data during a later reconstruction of the Image.

removeConsumer(ImageConsumer ic) This method removes the given ImageConsumer object from the list of consumers currently registered to receive image data.

startProduction(ImageConsumer ic) This method starts an immediate reconstruction of the image data

ImageConsumer methods

void setDimensions(int width, int height)

The dimensions of the source image are reported using the setDimensions method call.

Void setPixels(int x, int y, int w, int h, ColorModel model, byte[] pixels, int off, int scansize)

The pixels of the image are delivered using one or more calls to the setPixels method.

Image Class continued

Summary

AWT is operating system independent

Streaming image model

Images as tokens

Producer - consumer pipeline

Jimage implements ImageConsumer

Library of image processing classes developed at Faraday

Available for student practicals

Algebraic rather than stream oriented

Interfaces to MMX hardware under windows

Algebraic orientation

By this we mean the it is structured around algebraic expressions whose values are images

Thus if A and B are images and ‹ is some operator then

A ‹ B is also an image

Jimage operators

Arithmetic

I+J Universal plus(Universal)

I-J Universal minus(Universal)

I‚J Universal times(Universal)

I„J Universal divide(Universal)

½I½Universal abs()

Filtering

Jimage convolve(double[] k) convolve with symmetrical separable kernel.

public abstract Jimage convolve(double[][] kernel)with non separable kernel

Scaling

Jimage getScaledInstance(int nwidth, int nheight)
This scales with bicubic interpolation.

Jimage getScaledInstance(int nwidth, int nheight, int ndepth)
This method allows the depth as well as area of an image to be altered if it is reduced the planes are aggregated if increased they are interpolated.

More operations

Data access

int rgbpixel(int x,int y)

Converts the plane information into a pixel in the direct color model of java.

public abstract int upixel(int x,

                           int y,

                           int plane)

- returns unsigned integer pixel

public abstract float fpixel(int x,

                             int y,

                             int plane)

Returns the pixel in the range -1 to +1.

Data Access

public abstract void setPixel(int x,

                              int y,

                              int plane,

                              double pix)

Pixel information in range -1 to +1

public void setSubImage(int x,

                        int y,

                        int z,

                        Jimage im)

Update an area of an image with another one. The other one must not run off the edge of the one being written to. The source of the copying is the 0th plane of the source jimage.

Jimage input output

public void putJPEGImage(

            java.lang.String fileName,

            int quality)

                  throws java.io.IOException

Outputs the image to a jpeg file

public boolean getImage(java.lang.String fileName)

Initialise the Jimage from the specified file. The file must be jpeg or gif.

Jimage to AWT Image conversion

public java.awt.Image getAWTImage()

public java.awt.image.ImageProducer getProducer()

Jimage implementations

An example program

class Jimageshow extends Frame {

    Jimageshow(String filename) {

        super("Jimage show Example");

        Jimage raw=new ByteImage(100,200,3);

        if (raw.getImage(filename)){

          Jimage cooked = (Jimage)raw.times(0.3);

          add(new ImageCanvas(cooked.getAWTImage()), BorderLayout.CENTER);

        setSize(700, 540);

        show();

        }

    }

Pixel Representations

When dealing with displays it is conventional to assume that pixels are bytes holding numbers in the range 0 to 255.

0 Is assumed to be black

1 Is assumed to be white or maximum brightness of any given colour.

For multicolour displays with 3 colour components, the convention is to have 3 fields of range 0..255 to hold the colour information.

Pixel Representations AWT

For multicolour displays with 3 colour components, the convention is to have 3 fields of range 0..255 to hold the colour information. The AWT does this with the class Color.

public Color(int rgb)

Creates a color with the specified RGB value, where the red component is in bits 16-23 of the argument, the green component is in bits 8-15 of the argument, and the blue component is in bits 0-7. The value zero indicates no contribution from the primary color component.

A Jimage returns this format with int rgbpixel().

Pixel Representations: Bytes

The byte data type in Java does not take on the values 0..255. Instead it takes on the values -128 to 127.

There are no unsigned bytes in Java.

This creates a problem for the representation of pixels in Jimages.

The solution adopted is to adopt the following representation

-128 = black

0 = mid grey

127 = white

Pixel Representations: Floats

If byte pixels are signed then so must other representations be.

The solution adopted is to adopt the following representation for floats

-1 = black

0 = mid grey

1 = white

Conversions between representations

unsigned bytes shorts float

min value 0 -128 -2048 -1

maxval 255 127 2047 1

medianval m 127.5 -0.5 -0.5 0

range r 255 255 4095 2

As shown in table a pixel p_rin representation r is converted to a pixel p_s in representation s by the operation:

p_s = m_s+(r_s(p_r-m_r))¸r_r

Signed Pixels : advantages

Signed pixels seem at first to be counter-intuitive but they have numerous advantages.

A value of 0 or mid grey can be viewed as the ‘most likely’ value that a pixel takes on in the absence of other information.

If you do arithmetic on images, in particular subtract one image from another, then negative values of pixels naturally arise.

Signed pixels allow straightforward implementation of contrast adjustments. For instance multiplying an image by 0.5 halves the contrast in the image.

Signed Pixels : contrast adjustment

Signed pixels allow straightforward implementation of contrast adjustments. For instance multiplying an image by 0.5 halves the contrast in the image.

Image Multiplication

Image Addition

Image subtraction

What Is Convolution

Image Convolution: smoothing

Image Convolution: sharpening

Importance of speed

Image may contain a million pixels,

Arithmetic may be required on each one

Important to optimise operations or they are very time consuming

May need to use assembler kernels

May need to use special purpose instructions

Multimedia Extensions MMX

Intel and other CPU manufacturers have been adding to the instruction sets of their computers new extensions that handle multi-media data.

The aim is to allow operations to proceed on multiple pixels each clock cycle

MMX 2

Standard Intel register set

MMX 3

Standard Intel register set operating in MMX mode

MMX 4 motivation

Existing operating systems must still work unchanged

Applications not using MMX run unchanged

No new state added to the CPU

Hence, shared use of the FP registers, since these are already supported by exising OS’s

MMX data formats

Problem of overflows

A problem with limited precision arithmetic is that overflows frequently occur. This can give rise to meaningless results: consider

200+175 = 375 but in 8 bit binary

11001000

+10101111

=101110111

Using saturation

You can fix this by using conditionals

unsigned char p1,p2,p3;

int I3= (int)p1 + (int)p2;

p3=(I3>255?255:(unsigned char)I3);

Expansion of the code 1

12:               j=(int)(*p1++)+(int)(*p2++);

00401043   mov         ecx,dword ptr [ebp-4]

00401046   xor          edx,edx

00401048   mov         dl,byte ptr [ecx]

0040104A   mov         eax,dword ptr [ebp-8]

0040104D   xor         ecx,ecx

0040104F   mov         cl,byte ptr [eax]

00401051   add         edx,ecx

00401053   mov         dword ptr [ebp-14h],edx

00401056   mov         edx,dword ptr [ebp-8]

00401059   add         edx,1

0040105C   mov         dword ptr [ebp-8],edx

0040105F   mov         eax,dword ptr [ebp-4]

00401062   add         eax,1

00401065   mov         dword ptr [ebp-4],eax

Expansion 2

13:

14:          *p3 = (unsignedchar)(j>255?255:j);

00401068   cmp         dword ptr [ebp-14h],0FFh

0040106F   jle         main+6Ah (0040107a)

00401071   mov         dword ptr [ebp-18h],0FFh

00401078   jmp         main+70h (00401080)

0040107A   mov         ecx,dword ptr [ebp-14h]

0040107D   mov         dword ptr [ebp-18h],ecx

00401080   mov         edx,dword ptr [ebp-0Ch]

00401083   mov         al,byte ptr [ebp-18h]

00401086   mov         byte ptr [edx],al

15:               p3++;

00401088   mov         ecx,dword ptr [ebp-0Ch]

0040108B   add         ecx,1

0040108E   mov         dword ptr [ebp-0Ch],ecx

Alternative using mmx

Iu8vec8 *v1,*v2,*v3;

int i,j,k;

for(i=0;i<31;i++){

*v3=(*v1++)+(*v2++);

v3++;

}

_mm_empty();

Optimised Assembler Loop

   mov ecx ,32     ; load counter with 32

l1: movq mm0,[esi]    ; load 8 bytes

    add esi,8     ; inc src pntr

    paddusb mm0,[edx] ; packed unsigned add bytes

    add edx,8     ; inc src pntr

    movq [edi],mm0    ; store 8 byte result

    add edi,8     ; inc dest pntr

    loop nz,l1     ; dec counter,

                      ; repeat non zero

Speed Gain

On image of 256x256 pixels

Old C code executes 26*256*256 instructions = 1,703,936 instructions

Optimised mmx code executes 6*256*32 instructions = 49,152

Note that no compiler currently will give the optimised code. It has to be hand assembled.

Image Processing Library

Intel Provide an image porcessing library that can be downloaded from their web site.

It provides efficient access to the MMX hardware.

It provides frequently used Image Processing Operations.

It requires a set of DLLs in your path to run

Image Processing Library 2

At the core of IPL is the ability to write to a single API and get the best possible results for any Intel processor. The libraries have as many as six processor-specific branches for each function and six sets of carefully written assembly code, but only one entry point to each function.

Image Processing Library 3

Image Processing Library 4

Use of Intel IPL complex and requires C

I have provided 2 java classes that call the IPL.

IntelBImage and IntelFImage. These are documented in the Jimage web pages. They inherit from ByteImage and FloatImage

To use them the Intel IPL must have been installed on your machine and be on the path.

If you are forced to use Unix machines the libraries will not be available to you.

Where to get more information

http://www.javasoft.com/products/jdk/1.2/docs/api/java/awt/package-summary.html

http://developer.intel.com/vtune/perflibst/ipl/index.htm

http://developer.intel.com/vtune/perflibst/ipl/ipapi.htm

Feedback

What did you not understand here?

What would you like more information on?


	The AWT and how to put a simple image onto the screen
	Layout managers
	The Jimage Class,


	At the end of this lecture you should have an idea of how to display a JPEG image on the screen, and how to load it into the Jimage class to carry out further image processing.


	AWT Images
	Image Producers and Consumers
	Jimage class
	Pixel Representations
	JPEG files


	AWT abstract windows toolkit, supported by JavaSoft
	Operating system independent layer for windowing in Java
	Fiendishly obscure
	Designed around requirements of images being streamed off the web


	Simple image display program to show how to display a JPEG file
	Pipeline model of image production
	Jimages act as image consumers
	Jimages allow arithmetic on image
	Jimages provide output to AWT images and JPEG


	import java.awt.*;
	import java.awt.image.*;
	import java.util.*;
	class JPEGshow extends Frame {
	...
	static public void main(String[] args) {
	if (args.length == 1) new JPEGshow(args[0]);
	else System.err.println("usage: java JPEGshow <image file>");
	}
	}
	This is a standard Java Program class with a public static void main method


	JPEGshow(String filename) {
	super("JPEG show Example");
	add(
	new ImageCanvas(getToolkit().getImage(filename) ),
	BorderLayout.CENTER);
	setSize(700, 540);
	show();
	}


	Each frame has associated with it a toolkit object the provides an interface to OS specific operations.
	CreateImage
	CreateMenu
	CreateLabel
	CreateMenuBar …. etc


	class ImageCanvas extends Component {
	Image image;
	ImageCanvas(Image image)
	{this.image = image;}
	public void paint(Graphics g)
	{ g.drawImage(image, 0, 0, this);}
	}


	Pipeline flow model of image processing
	Images are just tokens linking producers and consumers


	addConsumer(ImageConsumer ic) This method is used to register an ImageConsumer with the ImageProducer for access to the image data during a later reconstruction of the Image.
	removeConsumer(ImageConsumer ic) This method removes the given ImageConsumer object from the list of consumers currently registered to receive image data.
	startProduction(ImageConsumer ic) This method starts an immediate reconstruction of the image data


	void setDimensions(int width, int height)
	The dimensions of the source image are reported using the setDimensions method call.
	Void setPixels(int x, int y, int w, int h, ColorModel model, byte[] pixels, int off, int scansize)
	The pixels of the image are delivered using one or more calls to the setPixels method.


	AWT is operating system independent
	Streaming image model
	Images as tokens
	Producer - consumer pipeline


	Library of image processing classes developed at Faraday
	Available for student practicals
	Algebraic rather than stream oriented
	Interfaces to MMX hardware under windows


	By this we mean the it is structured around algebraic expressions whose values are images
	Thus if A and B are images and ‹ is some operator then
	A ‹ B is also an image


	Arithmetic
	I+J Universal plus(Universal)
	I-J Universal minus(Universal)
	I‚J Universal times(Universal)
	I„J Universal divide(Universal)
	½I½Universal abs()
	Filtering
	Jimage convolve(double[] k) convolve with symmetrical separable kernel.
	public abstract Jimage convolve(double[][] kernel)with non separable kernel


	Jimage getScaledInstance(int nwidth, int nheight) This scales with bicubic interpolation.
	Jimage getScaledInstance(int nwidth, int nheight, int ndepth) This method allows the depth as well as area of an image to be altered if it is reduced the planes are aggregated if increased they are interpolated.


	Data access
	int rgbpixel(int x,int y)
		Converts the plane information into a pixel in the direct color model of java.
	public abstract int upixel(int x,
	int y,
	int plane)
	- returns unsigned integer pixel
	public abstract float fpixel(int x,
	int y,
	int plane)
		Returns the pixel in the range -1 to +1.


	public abstract void setPixel(int x,
	int y,
	int plane,
	double pix)
		Pixel information in range -1 to +1
	public void setSubImage(int x,
	int y,
	int z,
	Jimage im)
		Update an area of an image with another one. The other one must not run off the edge of the one being written to. The source of the copying is the 0th plane of the source jimage.


	public void putJPEGImage(
	java.lang.String fileName,
	int quality)
	throws java.io.IOException
		Outputs the image to a jpeg file
	public boolean getImage(java.lang.String fileName)
		Initialise the Jimage from the specified file. The file must be jpeg or gif.


	public java.awt.Image getAWTImage()

	public java.awt.image.ImageProducer getProducer()


	class Jimageshow extends Frame {
	Jimageshow(String filename) {
	super("Jimage show Example");
	Jimage raw=new ByteImage(100,200,3);
	if (raw.getImage(filename)){
	Jimage cooked = (Jimage)raw.times(0.3);
	add(new ImageCanvas(cooked.getAWTImage()), BorderLayout.CENTER);
	setSize(700, 540);
	show();
	}
	}


	When dealing with displays it is conventional to assume that pixels are bytes holding numbers in the range 0 to 255.
	0 Is assumed to be black
	1 Is assumed to be white or maximum brightness of any given colour.
	For multicolour displays with 3 colour components, the convention is to have 3 fields of range 0..255 to hold the colour information.


	For multicolour displays with 3 colour components, the convention is to have 3 fields of range 0..255 to hold the colour information. The AWT does this with the class Color.
	public Color(int rgb)
		Creates a color with the specified RGB value, where the red component is in bits 16-23 of the argument, the green component is in bits 8-15 of the argument, and the blue component is in bits 0-7. The value zero indicates no contribution from the primary color component.
		A Jimage returns this format with int rgbpixel().


	The byte data type in Java does not take on the values 0..255. Instead it takes on the values -128 to 127.
	There are no unsigned bytes in Java.
	This creates a problem for the representation of pixels in Jimages.
	The solution adopted is to adopt the following representation
	-128 = black
	0 = mid grey
	127 = white


	If byte pixels are signed then so must other representations be.
	The solution adopted is to adopt the following representation for floats
	-1 = black
	0 = mid grey
	1 = white


	unsigned bytes shorts float
	min value 0 -128 -2048 -1
	maxval 255 127 2047 1
	medianval m 127.5 -0.5 -0.5 0
	range r 255 255 4095 2
	As shown in table a pixel p_rin representation r is converted to a pixel p_s in representation s by the operation:
	p_s = m_s+(r_s(p_r-m_r))¸r_r


	Signed pixels seem at first to be counter-intuitive but they have numerous advantages.
	A value of 0 or mid grey can be viewed as the ‘most likely’ value that a pixel takes on in the absence of other information.
	If you do arithmetic on images, in particular subtract one image from another, then negative values of pixels naturally arise.
	Signed pixels allow straightforward implementation of contrast adjustments. For instance multiplying an image by 0.5 halves the contrast in the image.


	Image may contain a million pixels,
	Arithmetic may be required on each one
	Important to optimise operations or they are very time consuming
	May need to use assembler kernels
	May need to use special purpose instructions


	Intel and other CPU manufacturers have been adding to the instruction sets of their computers new extensions that handle multi-media data.
	The aim is to allow operations to proceed on multiple pixels each clock cycle