I was searching for a simple and effective solution for loading and manipulating very large images with BufferedImage. The problem was that BufferedImage stores the whole image in the memory.

Solution

BigBufferedImage can load and store much larger images than the memory limit. The trick is that its buffer is replaced with a file based implementation of DataBuffer. It stores the decoded raw image data in files on the hard drive.

Logical structure

For each image band (Red, Green, Blue, Alpha) there is a buffer file on the storage. The buffer files are accessed with a MappedByteBuffer, which provides a fast random access to the files, you can handle the files as simple byte arrays in the memory.

Loading the image

Loading big image files into the BigBufferedImage is a bit difficult because the decoded raw image can be very large so you can't read the whole image into the memory. It must be loaded in smaller parts. Fortunately ImageIO provides solution for this problem.

Limits

With BigBufferedImage you can store maximum 2,147,483,647 pixels (or 46,340 x 46,340 pixels). With 4 color channels (RGB) it means c.a. 8 GByte raw data. It has the same theoretical capacity as the original BufferedImage has, but BigBufferedImage has the real benefit that it stores the image on the hard drive and you don't have to worry about the heap size or the physical memory limit.

Download

BigBufferedImage.java
 

Licenc: Creative Commons CC0

Developed by Puli Space Technologies

Improved by: Dimitry Polivaev, Freeplane

Join us: Small Step Club

How to use it

After creating a BigBufferedImage you can use it as a simple BufferedImage. You can ose only two image types: TYPE_INT_RGB and TYPE_INT_ARGB.

You can specify the maximum of the used memory with the contant MAX_PIXELS_IN_MEMORY in file BigBufferedImage.java.

BigBufferedImage image = BigBufferedImage.create(  // Create empty image
    [image width],
    [image height],
    [image type]);

BigBufferedImage image = BigBufferedImage.create(  // Load existing image from file
[source file],
    [image type]);

 

Source

/*
 * This class is part of MCFS (Mission Control - Flight Software) a development
 * of Team Puli Space, official Google Lunar XPRIZE contestant.
 * This class is released under Creative Commons CC0.
 * @author Zsolt Pocze, Dimitry Polivaev
 * Please like us on facebook, and/or join our Small Step Club.
 * http://www.pulispace.com
 * https://www.facebook.com/pulispace
 * http://nyomdmegteis.hu/en/
 * https://www.freeplane.org
 */
import java.awt.Point;
import java.awt.Rectangle;
import java.awt.color.ColorSpace;
import java.awt.image.BandedSampleModel;
import java.awt.image.BufferedImage;
import java.awt.image.ColorModel;
import java.awt.image.ComponentColorModel;
import java.awt.image.DataBuffer;
import java.awt.image.Raster;
import java.awt.image.RenderedImage;
import java.awt.image.SampleModel;
import java.awt.image.WritableRaster;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.Hashtable;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.imageio.ImageIO;
import javax.imageio.ImageReadParam;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;
import sun.nio.ch.DirectBuffer;
public class BigBufferedImage extends BufferedImage {
    private static final String TMP_DIR = System.getProperty("java.io.tmpdir");
    public static final int MAX_PIXELS_IN_MEMORY =  1024 * 1024;
    public static BufferedImage create(int width, int height, int imageType) {
        if (width * height > MAX_PIXELS_IN_MEMORY) {
            try {
                final File tempDir = new File(TMP_DIR);
                return createBigBufferedImage(tempDir, width, height, imageType);
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        } else {
            return new BufferedImage(width, height, imageType);
        }
    }
    public static BufferedImage create(File inputFile, int imageType) throws IOException {
        try (ImageInputStream stream = ImageIO.createImageInputStream(inputFile);) {
            Iterator<ImageReader> readers = ImageIO.getImageReaders(stream);
            if (readers.hasNext()) {
                try {
                    ImageReader reader = readers.next();
                    reader.setInput(stream, true, true);
                    int width = reader.getWidth(reader.getMinIndex());
                    int height = reader.getHeight(reader.getMinIndex());
                    BufferedImage image = create(width, height, imageType);
                    int cores = Math.max(1, Runtime.getRuntime().availableProcessors() / 2);
                    int block = Math.min(MAX_PIXELS_IN_MEMORY / cores / width, (int) (Math.ceil(height / (double) cores)));
                    ExecutorService generalExecutor = Executors.newFixedThreadPool(cores);
                    List<Callable<ImagePartLoader>> partLoaders = new ArrayList<>();
                    for (int y = 0; y < height; y += block) {
                        partLoaders.add(new ImagePartLoader(
                            y, width, Math.min(block, height - y), inputFile, image));
                    }
                    generalExecutor.invokeAll(partLoaders);
                    generalExecutor.shutdown();
                    return image;
                } catch (InterruptedException ex) {
                    Logger.getLogger(BigBufferedImage.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        }
        return null;
    }
    private static BufferedImage createBigBufferedImage(File tempDir, int width, int height, int imageType)
        throws FileNotFoundException, IOException {
        FileDataBuffer buffer = new FileDataBuffer(tempDir, width * height, 4);
        ColorModel colorModel = null;
        BandedSampleModel sampleModel = null;
        switch (imageType) {
            case TYPE_INT_RGB:
                colorModel = new ComponentColorModel(ColorSpace.getInstance(ColorSpace.CS_sRGB),
                    new int[]{8, 8, 8, 0},
                    false,
                    false,
                    ComponentColorModel.TRANSLUCENT,
                    DataBuffer.TYPE_BYTE);
                sampleModel = new BandedSampleModel(DataBuffer.TYPE_BYTE, width, height, 3);
                break;
            case TYPE_INT_ARGB:
                colorModel = new ComponentColorModel(ColorSpace.getInstance(ColorSpace.CS_sRGB),
                    new int[]{8, 8, 8, 8},
                    true,
                    false,
                    ComponentColorModel.TRANSLUCENT,
                    DataBuffer.TYPE_BYTE);
                sampleModel = new BandedSampleModel(DataBuffer.TYPE_BYTE, width, height, 4);
                break;
            default:
                throw new IllegalArgumentException("Unsupported image type: " + imageType);
        }
        SimpleRaster raster = new SimpleRaster(sampleModel, buffer, new Point(0, 0));
        BigBufferedImage image = new BigBufferedImage(colorModel, raster, colorModel.isAlphaPremultiplied(), null);
        return image;
    }
    private static class ImagePartLoader implements Callable<ImagePartLoader> {
        private final int y;
        private final BufferedImage image;
        private final Rectangle region;
        private final File file;
        public ImagePartLoader(int y, int width, int height, File file, BufferedImage image) {
            this.y = y;
            this.image = image;
            this.file = file;
            region = new Rectangle(0, y, width, height);
        }
        @Override
        public ImagePartLoader call() throws Exception {
            Thread.currentThread().setPriority((Thread.MIN_PRIORITY + Thread.NORM_PRIORITY) / 2);
            try (ImageInputStream stream = ImageIO.createImageInputStream(file);) {
                Iterator<ImageReader> readers = ImageIO.getImageReaders(stream);
                if (readers.hasNext()) {
                    ImageReader reader = readers.next();
                    reader.setInput(stream, true, true);
                    ImageReadParam param = reader.getDefaultReadParam();
                    param.setSourceRegion(region);
                    BufferedImage part = reader.read(0, param);
                    Raster source = part.getRaster();
                    WritableRaster target = image.getRaster();
                    target.setRect(0, y, source);
                }
            }
            return ImagePartLoader.this;
        }
    }
    private BigBufferedImage(ColorModel cm, SimpleRaster raster, boolean isRasterPremultiplied, Hashtable<?, ?> properties) {
        super(cm, raster, isRasterPremultiplied, properties);
    }
    public void dispose() {
        ((SimpleRaster) getRaster()).dispose();
    }
    public static void dispose(RenderedImage image) {
        if (image instanceof BigBufferedImage) {
            ((BigBufferedImage) image).dispose();
        }
    }
    private static class SimpleRaster extends WritableRaster {
        public SimpleRaster(SampleModel sampleModel, FileDataBuffer dataBuffer, Point origin) {
            super(sampleModel, dataBuffer, origin);
        }
        public void dispose() {
            ((FileDataBuffer) getDataBuffer()).dispose();
        }
    }
    private static final class FileDataBufferDeleterHook extends Thread {
        static {
            Runtime.getRuntime().addShutdownHook(new FileDataBufferDeleterHook());
        }
        private static final HashSet<FileDataBuffer> undisposedBuffers = new HashSet<>();
        @Override
        public void run() {
            final FileDataBuffer[] buffers = undisposedBuffers.toArray(new FileDataBuffer[0]);
            for (FileDataBuffer b : buffers) {
                b.disposeNow();
            }
        }
    }
    private static class FileDataBuffer extends DataBuffer {
        private final String id = "buffer-" + System.currentTimeMillis() + "-" + ((int) (Math.random() * 1000));
        private File dir;
        private String path;
        private File[] files;
        private RandomAccessFile[] accessFiles;
        private MappedByteBuffer[] buffer;
        public FileDataBuffer(File dir, int size) throws FileNotFoundException, IOException {
            super(TYPE_BYTE, size);
            this.dir = dir;
            init();
        }
        public FileDataBuffer(File dir, int size, int numBanks) throws FileNotFoundException, IOException {
            super(TYPE_BYTE, size, numBanks);
            this.dir = dir;
            init();
        }
        private void init() throws FileNotFoundException, IOException {
            FileDataBufferDeleterHook.undisposedBuffers.add(this);
            if (dir == null) {
                dir = new File(".");
            }a
            if (!dir.exists()) {
                throw new RuntimeException("FileDataBuffer constructor parameter dir does not exist: " + dir);
            }
            if (!dir.isDirectory()) {
                throw new RuntimeException("FileDataBuffer constructor parameter dir is not a directory: " + dir);
            }
            path = dir.getPath() + "/" + id;
            File subDir = new File(path);
            subDir.mkdir();
            buffer = new MappedByteBuffer[banks];
            accessFiles = new RandomAccessFile[banks];
            files = new File[banks];
            for (int i = 0; i < banks; i++) {
                File file = files[i] = new File(path + "/bank" + i + ".dat");
                final RandomAccessFile randomAccessFile = accessFiles[i] = new RandomAccessFile(file, "rw");
                buffer[i] = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_WRITE, 0, getSize());
            }
        }
        @Override
        public int getElem(int bank, int i) {
            return buffer[bank].get(i) & 0xff;
        }
        @Override
        public void setElem(int bank, int i, int val) {
            buffer[bank].put(i, (byte) val);
        }
        @Override
        protected void finalize() throws Throwable {
            dispose();
        }
        private void disposeNow() {
            final MappedByteBuffer[] disposedBuffer = this.buffer;
            this.buffer = null;
            disposeNow(disposedBuffer);
        }
        public void dispose() {
            final MappedByteBuffer[] disposedBuffer = this.buffer;
            this.buffer = null;
            new Thread() {
                @Override
                public void run() {
                    disposeNow(disposedBuffer);
                }
            }.start();
        }
        private void disposeNow(final MappedByteBuffer[] disposedBuffer) {
            FileDataBufferDeleterHook.undisposedBuffers.remove(this);
            if (disposedBuffer != null) {
                for (MappedByteBuffer b : disposedBuffer) {
                    ((DirectBuffer) b).cleaner().clean();
                }
                for (RandomAccessFile file : accessFiles) {
                    try {
                        file.close();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
                accessFiles = null;
                for (File file : files) {
                    file.delete();
                }
                files = null;
                new File(path).delete();
                path = null;
            }
        }
    }
}

 

Blog

Sending Emails via Gmail

On May 31, 2022, Google disabled the ability to send emails from external applications using your own password. Therefore, you need to create an app password to continue sending emails from our software. Setting up an app password can be done with very simple steps, requiring no technical expertise—just follow the steps described here!

Easily manage paper invoicing with InvoicePad 2

We're thrilled to unveil InvoicePad 2, the future of hassle-free paper-based invoicing compliant with NAV standards. Blending sophisticated invoicing with integrated CRM and inventory management systems, InvoicePad 2 offers a comprehensive business solution for today's digital age.

Introducing Freight 3: The Ultimate Transport Register Software

We're excited to announce the release of Freight 3, our latest transport register software, meticulously designed for professionals in the transportation industry. Freight 3 serves as the perfect solution for recording and invoicing transport operations, managing vehicles, drivers, and customers. Discover an unparalleled software experience with our newest features and enhanced functionalities.

Introducing CustomerRegister 3: The Simple Customer Registration Software (CRM)

Empower your business with the new and improved CustomerRegister 3, the state-of-the-art customer registration software tailored for companies seeking speed, flexibility, and enhanced functionality. Our platform has been meticulously designed to optimize and transform your customer registration experience.

Introducing CustomerRegister 2: Your Ultimate Free CRM Solution!

Easy-to-use, flexible, and entirely free, the new CustomerRegister 2 is here to redefine the way you manage your customer data.

The Future of Stock Management Software

Experience the evolution in inventory management with InventoryManager 3, a blend of efficiency, flexibility, and innovation. Go beyond traditional stock management to optimize your inventory process with advanced features and AI-driven capabilities.

Redefining Inventory Management for Small Businesses

Streamlining stock management has never been easier. InventoryManager 2 is not just another free inventory software; it's a revolutionary platform that marries user-friendliness with advanced AI capabilities to enhance text processing for businesses.

Introducing JobCard 2: The Simple and Free Service Repair Shop Software

Available for Windows and Linux 64-bit, JobCard 2 is the perfect solution for car repair services, computer repair shops, machine service, repair workshops, and software development companies.

Introducing JobCard 3: The Ultimate Service Repair Shop Software

Customizable software for car repair services, computer repair shops, machine service, repair workshops, software development companies.

Multi-scale information content measurement method based on Shannon information

The new mathematical method presented here provides a more accurate estimate of the (internal) information content of any discrete pattern based on Shannon's original function. The method is tested on different data sets and the results are compared with the results of other methods like compression algorithms.

Effective Handling of Big Images in Java

I was searching for a simple and effective solution for loading and manipulating very large images with BufferedImage. The problem was that BufferedImage stores the whole image in the memory.

Visitors

Today49
Yesterday49
This Week355
This Month523
All Days151007
Statistik created: 2024-11-08T14:16:29+01:00