How to compress .TIFF files?

[Hack&Lube]

Several groups in our company are generating huge high resolution scans. The result are .TIFF/.TIF files that are up to 10GB in size.

This is starting to become a problem for our storage and backup routines and I'm looking into compressing these files.

I have read that LZW is a standard lossless compression for TIFF files but I can't find a decent application that will do this. Most forums are about Photoshop/Photography, etc. and using the LZW compression inside Photoshop but that's not an option for us.

Anybody know a good way to batch compress a large number of very large .TIFF files with a lossless compression like LZW?

[photon]

Why not just convert them to JPG?

Or even just reduce the size of the original scans, if it's a 10GB file unless there's a real specific reason for keeping that much info, they probably don't even need to be that way.

Scan to PDF in the first place.

[getbak]

Why isn't Photoshop an option? You can set up batch processes in Photoshop.

10GB files might be an issue for Photoshop, but I'd think they'll be a problem for any software.

[Hack&Lube]

Modification of the files or process is not something I have any authority over. I need to preserve the integrity of what the business groups and their applications (often times, 3rd party companies) create and send us (large maps, core samples, etc.) as an ongoing process.

PDF is probably not an ideal format either as it contains extra information in headers (page format, etc.) and may not work with existing industry specific applications.

The only options are maintaining the status quo and finding a more efficient storage/backup solution or lossless compression of the original files as a batch process that we perform in the background without interruption or interference.

Therefore, I'm trying to find a way to batch convert a large number of test files with LZW compression to see what the result is and if it's worth it.

[CubicleGeek]

Look at Irfanview. I've used it to do mass format conversions and mass file renames. Nothing to lzw personally, but I did a quick google and it seems to support tiff to lzw.

[northcrunk]

Wow what kind of DPI are they using? 10gb is huge for a TIFF. Most companies only scan Tiff files in 300dpi

[Hack&Lube]

Originally Posted by getbak

Why isn't Photoshop an option? You can set up batch processes in Photoshop.

10GB files might be an issue for Photoshop, but I'd think they'll be a problem for any software.

Purchasing and licensing mostly. I would consider Photoshop after exhausting other options. I was hoping to find a simpler purpose-built standalone application like 7zip etc. that could also be command-line scripted and batched that would offer something like this but haven't been able to dig up anything yet.

[Huntingwhale]

Open the TIFF image in adobe, then file>save as JPEG. Works everytime.

IF you don't have adobe, open with Powerpoint and save the file as a JPEG.

[photon]

Gotcha, though if you go back to them with a "It will cost $crazy to store them as is" or "Compressing them will only put off $crazy for $smallamountoftime" then that may change

Anyway, where does this all happen? I assume it's part of the backup process which means you probably want automated, is it Windows or Linux?

Either way you should be able to script something using something like ImageMagick, I believe it supports compression in TIFFs so you should be able to run all the TIFFs through a conversion before they go off to where they're supposed to.

[Hack&Lube]

Originally Posted by northcrunk

Wow what kind of DPI are they using? 10gb is huge for a TIFF. Most companies only scan Tiff files in 300dpi

I found some files that are 4000 DPI...but others are as small as 96 DPI.

I opened one of the smallest files, it's GIS satellite imagery of a small tract of land. It's 548MB and only 96 DPI but it's 5911x24333 pixels uncompressed.

[photon]

I can see what you mean then if there's so many different sources, trying to change things at that level would be a nightmare.

Though I'm kind of surprised the apps themselves don't use LZW compression.

[photon]

5911 x 24333 = 143,832,363 bits = 17MB
If it's 32 bits per pixel then that's 548MB.
Just calculating to see if it's already compressed or not, seems like it isn't

[Hack&Lube]

Originally Posted by photon

Gotcha, though if you go back to them with a "It will cost $crazy to store them as is" or "Compressing them will only put off $crazy for $smallamountoftime" then that may change

Anyway, where does this all happen? I assume it's part of the backup process which means you probably want automated, is it Windows or Linux?

Either way you should be able to script something using something like ImageMagick, I believe it supports compression in TIFFs so you should be able to run all the TIFFs through a conversion before they go off to where they're supposed to.

If the problem was just sheer space, it would be much easier to just get approval to buy more drives than it is to convince management to train the business to save money by altering practices that aren't actually a part of daily business operations or getting other companies and 3rd parties on board as well. We just get the data in and it comes in all sorts of formats.

The more nebulous thing is that it's difficult to explain to business why it might be an issue that backups take longer and longer every week and why that causes a serious disaster recovery problem for them in the long run.

Yes, we'll have to figure out the automation part of this afterward... probably as a scripted process that runs on the file servers before the nightly backups start happening (which is another separate process with dedicated architecture for duping/de-duping to tape, etc.).

For now, I'm trying to find a solution to test different compression algorithms in the first place to see what works best.

[Hack&Lube]

Originally Posted by photon

5911 x 24333 = 143,832,363 bits = 17MB
If it's 32 bits per pixel then that's 548MB.
Just calculating to see if it's already compressed or not, seems like it isn't

It's interesting how you can do that so easily isn't it? I was reading up on the history of the TIF format and the first version of it only handled 1 bit pixels, either black or white (not even gray scale) because that's all scanners could do back when they introduced the standard.

[photon]

Hm, do you know the bit depth of the files? One thing I read was that 16 or 32 bit depth doesn't work well with LZW and they suggested to use zip for those kinds of files.

Have you tried zipping one?

[photon]

Originally Posted by Hack&Lube

For now, I'm trying to find a solution to test different compression algorithms in the first place to see what works best.

Makes sense. Infranview is a good suggestion, there are some graphical front ends for that ImageMagick library I mentioned as well. EDIT: Could even download a trail version of Photoshop, or the Gimp is free, not as your end tool necessarily, but just to get an idea as you say.

[Hack&Lube]

Hmm...for now, I'll try to convince our one guy with the Photoshop license to borrow it to try compressing it with LZW to see how small we can get the file. If it's only marginally smaller than a high-compression zip (deflate or LZMA, etc.), I'll probably just go with zipping. That way, the users can open their files easily.

[photon]

I've also seen Flate/Deflate suggested for higher bit depth TIFFs

On linux gzip uses that so you could also try gzipping them on linux (there's probably a windows gzip program out there somewhere).

[Hack&Lube]

So the highest compression seems to result from zipping via PPMd algorithm but that makes it difficult for the business to use files since they will have to unzip them.

Fortunately, I've found saving as a LZW compressed tif (lossless) via GIMP/Imagemagick results in a file that is still about 60-75% smaller on average.

Now I need to make a script to batch this job across several folders. Either imagemagick or gimp would work but I can't make the command line batch process work.

Any ideas? For Imagemagik, the following works:

convert -compress lzw source.tiff destination.tiff

but how can I use wildcards so it will just for instance compress everything in C:\IN and put the same files into C:\OUT? (I'll mount network locations to these folders). I've tried various of *, %, and $ but I think I'm doing it wrong. Any ideas?

[psyang]

Originally Posted by Hack&Lube

So the highest compression seems to result from zipping via PPMd algorithm but that makes it difficult for the business to use files since they will have to unzip them.

Fortunately, I've found saving as a LZW compressed tif (lossless) via GIMP/Imagemagick results in a file that is still about 60-75% smaller on average.

Now I need to make a script to batch this job across several folders. Either imagemagick or gimp would work but I can't make the command line batch process work.

Any ideas? For Imagemagik, the following works:

convert -compress lzw source.tiff destination.tiff

but how can I use wildcards so it will just for instance compress everything in C:\IN and put the same files into C:\OUT? (I'll mount network locations to these folders). I've tried various of *, %, and $ but I think I'm doing it wrong. Any ideas?

It's been a while since I've done DOS batch files, but look at the FOR command - you can use it to specify a group of files (wildcarded) and issue a command against each matched file. Maybe something like:

FOR %file in (c:\in\*.tiff) do convert -compress lzw c:\out\%~nfile