Java Large file to be saved in database - Object design

Question

I am trying to figure out object design to implement large file(~600 MB) respository in the Database using hibernate. Please suggest a correct approach/design?

class ModelClass{

String name;  //meta data
...

Option 1. 
    byte[] file; // dont want to load the content of the entire file 
                           // in memory by using this but hibernate recognizes 
                           // this datatype
Option 2.
    InputStream inputStream;
    OutputStream outputStream;
    // I can have the methods to provide the input or output stream 
              // but i dont think its a clean approach. I am not sure how 
              // I will be able to work with hibernate with streams
Option 3.
    File fileHandle; 
}

Any other options??

I would like to call save(Object) method of hibernateTemplate to save the object in Database. Dont know if I should have just the meta-data in the class and handle the file save and retreive seperately.

Thanks in advance.

Another workable solution is to use "Work" Interface. The purpose was to avoid loading the file content into memory.

session.doWork(new Work(){
    @Override
    public void execute(Connection conn) {
        //direct sql queries go here 
    }

});

Is saving the file on disk and storing a path in the database not a good solution? — Amir Raminfar
– Amir Raminfar, Commented Aug 24, 2011 at 20:58
Unfortunately, its a business requirement to store large zip files in DB. — nkare
– nkare, Commented Aug 24, 2011 at 20:59
Here stripesframework.org/display/stripes/… suggests to use a BLOB but it is really hard for me to justify why to do this. It slows down the db, its makes backup and restores harder. The list goes on. — Amir Raminfar
– Amir Raminfar, Commented Aug 24, 2011 at 21:02

Maurice Perry · Accepted Answer · 2011-08-25 08:36:02Z

I have written a SerializableFile class that keeps data in a file. When the object is read, it creates a temporary file. Here it is:

public class SerializableFile implements Serializable {
    private static final File TEMP_DIR = getTempDir();

    private transient boolean temporary;
    private transient String name;
    private transient File file;

    public SerializableFile() {
    }

    public SerializableFile(File file) {
        this.file = file;
        this.name = file.getName();
        this.temporary = false;
    }

    @Override
    protected void finalize() throws Throwable {
        dispose();
        super.finalize();
    }

    public void dispose() {
        if (temporary && file != null) {
            file.delete();
            file = null;
        }
    }

    public File keep(String name) throws IOException {
        if (temporary) {
            temporary = false;
        } else {
            File newFile = new File(TEMP_DIR, name);
            keepAs(newFile);
            file = newFile;
        }
        return file;
    }

    public void keepAs(File outFile) throws IOException {
        if ((temporary || file.equals(outFile)) && file.renameTo(outFile)) {
            temporary = false;
            file = outFile;
        } else {
            InputStream in = new FileInputStream(file);
            try {
                OutputStream out = new FileOutputStream(outFile);
                try {
                    byte buf[] = new byte[4096];
                    for (int n = in.read(buf); n > 0; n = in.read(buf)) {
                        out.write(buf, 0, n);
                    }
                } finally {
                    out.close();
                }
            } finally {
                in.close();
            }
            outFile.setLastModified(file.lastModified());
        }
    }

    public String getName() {
        return name;
    }

    public File getFile() {
        return file;
    }

    public long lastModified() {
        return file.lastModified();
    }

    private void writeObject(ObjectOutputStream out) throws IOException {
        int size = (int)file.length();
        long date = file.lastModified();
        out.writeUTF(name);
        out.writeInt(size);
        out.writeLong(date);
        InputStream in = new FileInputStream(file);
        try {
            byte buf[] = new byte[4096];
            while (size > 0) {
                int n = in.read(buf);
                if (n <= 0 || n > size) {
                    throw new IOException("Unexpected file size");
                }
                out.write(buf, 0, n);
                size -= n;
            }
        } finally {
            in.close();
        }
    }

    private void readObject(ObjectInputStream in) throws IOException {
        name = in.readUTF();
        int size = in.readInt();
        long date = in.readLong();
        file = File.createTempFile("tmp", ".tmp", TEMP_DIR);
        OutputStream out = new FileOutputStream(file);
        try {
            byte buf[] = new byte[4096];
            while (size > 0) {
                int n = in.read(buf, 0, size <= buf.length ? size : buf.length);
                if (n <= 0 || n > size) {
                    throw new IOException("Unexpected file size");
                }
                out.write(buf, 0, n);
                size -= n;
            }
        } finally {
            out.close();
        }
        file.setLastModified(date);
        temporary = true;
    }

    private static File getTempDir() {
        File dir;
        String temp = System.getProperty("com.lagalerie.live.temp-dir");
        if (temp != null) {
            dir = new File(temp);
        } else {
            String home = System.getProperty("user.home");
            dir = new File(home, "temp");
        }
        if (!dir.isDirectory() && !dir.mkdirs()) {
            throw new RuntimeException("Could not create temp dir " + dir);
        }
        return dir;
    }
}

Thanks Maurice. But the sample code you have provided reads/writes to a disk. I am looking for read/write into DB using hibernate.
Declare a field of type SerializableFile in your ModelClass, map it to a blob field of your database, and it will be.
down casting file length types to int restricts the maximum file size to 2GiB, or approximately the first 2GiB of a larger file

GaetanZ · Accepted Answer · 2011-08-24 21:44:09Z

0

Open JPA supports a @Persistent annotation with some databases:

MySQL
Oracle
PostgreSQL
SQL Server
DB2

answered Aug 24, 2011 at 21:44

GaetanZ

3,1493 gold badges23 silver badges23 bronze badges

1 Comment

nkare Over a year ago

I dont think i will be allowed to introduce new framework but I am leaning towards using SQL datatype "Blob" and expose two methods which will set/get contentStream and use hibernate's createBlob method. Need to find out if createBlob stores the data in the memory.

HJW · Accepted Answer · 2011-08-25 08:38:13Z

0

Even if you are still using an RDBMS as a data store, you should consider storing this binary data into a file system, and saving the directory / location of the path into the database, instead of storing this as a BLOB or CLOB into the database.

answered Aug 25, 2011 at 8:38

HJW

23.5k51 gold badges147 silver badges218 bronze badges

2 Comments

nkare Over a year ago

Servers are setup in a clustered environment with no common file shares being available. We have a need to store the file in the database.

HJW Over a year ago

An unfortunate requirement, and "solution" if to be delivered using clob/blob stored into rdbms.

Collectives™ on Stack Overflow

Java Large file to be saved in database - Object design

3 Answers 3

4 Comments

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related