Merge iPhoto libraries. Got a whole bunch of libraries you want to consolidate into one? IPhoto Library Manager lets you merge libraries while weeding out duplicate photos in the process. You will be shown a preview of what your merged library will look like before any modifications are done, to ensure your merged library looks how you want. IPhoto Library Manager can merge multiple libraries while removing duplicates with configurable options.
This explains how I consolidated & de-duped about 45,000 pictures andmovies from four iPhoto libraries on two different machines and variousdirectories full of images on USB drives into one 33,000 file library.
John williams stoner. We had about 25,000 photos in one library from a previous (unsuccessful)attempt at consolidating libraries. We had two further librariescontaining about 13,000 and 10,000 images each, and a fourth iPhotolibrary on a separate machine, plus the aforementioned random grab-bagof directories full of images. (I didn't count how many these were, butthey numbered in the thousands.) After consolidation we ended up with33,000 images and movies in one iPhoto library.
iPhoto has a menu item for finding duplicates, but I was not been ableto make it work; it would just hang trying to find duplicates in one ofthe smaller libraries. And even if I could get that to work I didn'thave the space to copy all the images into one giant library and letiPhoto sort it out. So I wrote a program to help me.
One problem was that the same picture would exist with different filenames, probably due to being imported from devices into multipledifferent iPhoto libraries or to being exported & then imported again inthe aforementioned failed consolidation attempt. We also had differentimages existing with the same name, due to the counter on the cameralooping around. This was OK since iPhoto stores images from differentimports in different directories, but it meant that we couldn't justcopy all the files into one directory and use the names to de-duplicatethem.
I wrote a simple program to iterate over all image & movie files in adirectory and create a copy of each named after theMD5 hash of its content. Thusfiles whose content were identical, but had different names, would nowend up with the same name. I did not want to end up with tens ofthousands of files in one directory, so I made my program put each filein a directory named after the 2 first characters of each file's name.Here it is:
Since I didn't have enough disk space to duplicate all the images, Iused hardlinks. This essentially means you just create a new filenamepointing to the same data on disk, without taking up extra data. I alsomade the program create a log file containing the location of each fileand its MD5 hash so I could find out how many files were processed andhow many duplicates were found.
A problem with the MD5 hash approach would be if duplicate copies ofimages had different image metadata. This could be the case is somecopies had had their dates adjusted or if iPhoto where changing the EXIFinformation when you star photos or tag faces in them. Fortunately, alittle experimentation showed that iPhoto does not change the imagefiles but stores this extra information in a separate database. Myapproach would thus mean that we would lose stars & face tags, but weconsidered this an acceptable trade off. Consolidating and de-duping thelibraries would be worth it.
What if the EXIF information had been edited? (Some image managersdo.) Would all hope be lost? No, it would not. The Imagemagick suite oftools comes with the identify program, which allows you to get a hashof just the image data. This would allow us to find more duplicates,even if the EXIF information had been edited; however it runs much,much slower than just using MD5 so I chose not to use it.
One worry I had was how to make sure I got hold of all the originalsfrom iPhoto. Due to the aforementioned space constraint I couldn't justexport all the photos and thus create new copies. However, this turnedout not to be an issue. iPhoto stores its originals in a Mastersdirectory, along parallel directories for Previews and Thumbnails.Thus I just ran my program on the Masters directory. One risk would belosing edits to images, but I was OK with this.
In the end I had 255 directories with on average 130 files in each. Iimported these directories into a brand new iPhoto library roughly tenat a time. Since iPhoto doesn't use hardlinks but copies the photos intoits library (to avoid nasty surprises if someone decided to edit thefile in a different location, no doubt) this ate up disk fast; and I hadto delete the existing libraries under the different accounts to makeroom. Since I had Time Machine backups (yes, plural—I rotate two TimeMachine disks) I felt OK doing this.
We are really happy with the result :-)
Update 2014/05/08: A few days ago I received a question to this postfrom Jeff Ruth. Because it, and its answer, may be useful to more peopleI asked if I could reproduce it here.
Hello and thanks for the post about your iPhoto consolidation andde-duping. I'd love to do something similar. We have three Applelaptops (two MacBooks from 2008 and 2009) and a new Macbook ProRetina. In all, about 3-5 iPHoto libraries, I think. Your projectseems to be what we'd like to do. However, I'm not sure how to do it.If you have a minute, could you please tell me how I can use yourcode, or do this a different way, without the program? I am not aprogrammer, but have used Terminal now and then and am not afraid toexperiment if necessary. Thanks a lot, in advance.
Jeff Ruth
My response:
Do your libraries contain duplicates? Angel sanctuary scans. If not, I wouldn't bother withthis approach. We only needed to because we had imported the samephotos into multiple photo albums.
The script was intended for developers, but if you want to try youhave to copy the file (dedupe-media.sh) to your local disk, then makeit executable. In Terminal, you do that with ‘chmod +xdedupe-media.sh'. (Without the quotes.) You then run the script likeso:
Almanac of guitar voice leading vol 1. /path/to/dedupe-media.sh /path/to/iPhotoLibrary/Masters /destination/path
Consolidating Iphoto Libraries Mac
If you want to run this on multiple laptops you may want to make thedestination a single USB disk (or something) so probably/Volumes/MyUSBDiskName. This disk will accumulate unique photos.The product is just a big folder of files, which will have to importedinto a new library. All metadata (faces, etc) will have to attachedagain.
Merge Iphoto Libraries App
Please note this script comes with NO WARRANTY OF ANY KIND. You musttake adequate backup of your libraries before running the above.Although I have made my best attempt at making the script work (and itdoes for me!), I take no responsibility for any data loss you mayexperience.
Update 2015/08/16: One annoyance under iTunes is that the 'rolls' or'albums' feature is utterly meaningless, because you've got 255 albumsnamed '00', '01', '.', 'fe', 'ff'. However, you can fix that by runningexiftool over the resulting files to have them grouped by year andmonth (there aremoreexamples in the exiftool documentation):
Consolidate Iphoto Libraries
This moves the images found in DIR_OF_FILES into a directory structurebased on the year and month the data was taken, as taken from the Exifdata in the image itself. The structure will be like so: