hpr-knowledge-base/hpr_transcripts/hpr1791.txt

Episode: 1791
Title: HPR1791: Organizing Photos with Bash
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr1791/hpr1791.mp3
Transcribed: 2025-10-18 09:22:15

---

This is HPR episode 1,791 entitled Organizing Photos with Bash and in part on the series
Bash Cripting.
It is posted by Tony Pelin and in about 31 minutes long, the summary is you smash to simplify
the process of Organizing and backing up photographing.
This episode of HPR is brought to you by an honesthost.com.
At 15% discount on all shared hosting with the offer code HPR15, that's HPR15.
Better web hosting that's honest and fair at An Honesthost.com.
Hi, my name is Tony Pelinis and this is my second episode for Hacker Public Radio.
Today I wanted to talk to you about what I use to import my photographs, get them off
of my camera and onto my computer and how I organize them.
I'm not a professional photographer but I do like to take a lot of pictures.
Looking back through my folders, I have probably taken an average around 3 to 4,000 pictures
a year and it becomes or it can become quite overwhelming to manage.
I often have very little time and so I sit out for a solution that allow me to organize
my pictures quickly, make sure that they're backed up in case something happens to my computer
and that I'll be able to enjoy them for many years to come.
As a Linux user, I have a lot of options available to me in terms of photograph software
that I can use to manage my library.
There's DigiCam which I like a lot, shot well amongst others but what I found most effective
at least for me is to write my own scripts to manage my programs.
I still use software to edit them, to manipulate the images and ultimately to create something
that I can print but to organize my files, make sure that they're backed up regularly, etc.
I find that the best solution is to write my own scripts.
Today I'm going to talk about a script I wrote and hopefully you'll get an idea of what you can
do yourself or you can use my script as a starting point to customize something for your own personal
photographic management needs. The script I wrote will be available in the show notes
and it's also up on GitHub as a gist so if you like what you hear about you don't have to remember
everything but you can go on the show notes and see find the actual script and I also would like
to ask you if you know anything about bash scripting and you think you could improve my script
please feel free to send me a pull request or bank any suggestions because really although I
do know how to write bash scripts I'm certainly no expert and I'm sure there are plenty of people
out there that are much smarter and much better at this than me. Okay so before we take a look at
the script line by line I wanted to give you a high level overview of what the purpose of the script
and how I use it in my everyday workflow to organize my pictures so the most important thing
for me in this particular script is to be able to take images from a multitude of sources
and put them all in a single location and organize them into folders that are structured by the
date that the picture was taken specifically I want them in a folder by the year and then in a subfolder
by the specific date because that makes it very easy for me to go back sometime in the future and
be able to find pictures which were taken on a specific day without the need to have any special
software to look at the meta information on these images in order to identify a specific time
range and because I take a lot of pictures you know I'm using several different
cameras I have a digital SLR I have a little point in shoot I have my cell phone camera so it needs
to be able to interface with all these different devices including other pictures that are
synced to my computer through cloud services such as Dropbox and put them all in a single location
make sure that you know the file names are do not conflict and that I don't delete any
images that maybe duplicates or that I don't create any images that maybe duplicates
and the second thing that I want to do also is make sure that these pictures are backed up
so I use a couple of different services I back them up to Google plus because Google plus offers
a lot of nice features for photographers in terms of being able to share the images and also
it provides for an unlimited number of free uploads if the images are underneath a certain files
or dimension size now I know Google plus is not open source it's not accessible friendly
but these features or more specifically the price and the features are something that
I like even though it's not my primary source of saving these backup images I have also recently
added to the script another destination to back of my photographs and that's using dream hosts
dream objects which is a Amazon S3 compatible object storage and what I like about it basically
is that it's less expensive than Amazon at least to upload in store files for long-term backup
now one thing that I worry about because I live in California is that some sort of natural
disaster or earthquake will completely destroy my house and all the electronics that I have inside
and that the pictures that I've taken for years dating back now to 2005 will be completely destroyed
and lost to me forever now a lot of these pictures really are irreplaceable to me I have a lot of
family you know my kids growing up things that I would never be able to go back in time and take so
I want to make sure that I take all the appropriate precautions to make sure this information is
backed up securely or at least in a way where I feel more confident that it would be able to
withstand the event of some sort of catastrophe affecting my computer now in order to back up
these images I also want to create thumbnails of all the pictures that I take because some of my
cameras have large CCDs take 16 megapixel images which end up being quite large in the file size
so when I upload them to Google plus or something I don't really need them
at those huge dimensions so I want to create thumbnails which allow me to resize the pictures into a size
that Google plus allows me to store them for free now the disadvantage of reducing the size of
an image and backing those up is that in the future who knows what will be the standard screen
resolution we may end up finding out that an image that's a thousand pixels by a thousand pixels
will look extremely tiny in the monitors of the future and to be most the most future proof it's
always best to back up the original files but of course the original files of their larger in
resolution larger in file size it ends up being more expensive to back them up but I've come to a
point where I'm okay with paying the extra price to use something like Amazon S3 or DreamHouse
DreamObjects to store this information because the pictures that I've taken have a value to me
and I want to make sure that they are backed up safely and that I have the original files if I
ever want to go back and edit them or print them or share them in the future now if I look back
I've been taking digital photographs since around 2005 or so actually you know I did have a small camera
that allowed me to take digital photographs a few years before that but I look back at those and
the quality was so poor the image and size was so small that it's almost it's really difficult
to look at some of those pictures because they're very pixelated blurry and small and size now
I first learned to or I first really got into photography when I was in high school so be around
the mid-90s and back then we used film cameras and of course one of the picked disadvantages of film
is that it's relatively expensive back in those days you could get a role of film of which would
only take 24 pictures and you spend a day taking those 24 pictures you have to make sure that you're
metering the light and that you're everything is in focus and then you spend some extra money to
develop the film and then you wait maybe a week or so unless you go to one of those one hour photo
places and you get your pictures back and you realize half of them are out of focus or the
metering wasn't correct so they're dark or too light and you know one one of the really nice things
about digital photography is that we can look at the picture right away we can tell whether
it's out of focus and if it is or if the lighting is not right we can take the picture right
take a new picture again of course the disadvantage of that is now it's become incredibly easy to
take thousands of pictures and it's of course a lot more to manage for someone who's not a
professional like myself so hopefully again this script will help you and address some of those
as it has helped me so without further ado let's go through the script line by line and I
hope to explain to you what it does okay the first thing you'll want to do is download the script
and like I said before I will include a link in the show notes once you have the script downloaded
you can just run it the script itself is just a single file called import-photos.sh so
if you make sure set that to executable and then open up a terminal because this is a
bash script so you'll have to use it from the command line navigate to wherever you save the file
and type in the name of the file name which is import-photos.sh this particular script also
accepts arguments and before to get a summary of the arguments that this script takes just type
in the name of the file then space then dash then h and you will get the little help
some help text it shows the dependencies for this particular script which are gphoto2
dc raw google cl rsync and s3 cmd so if you don't have those particular applications already
on your computer you'll have to install them now you can still use the script
if you're missing some of them such as google cl or s3 cmd or gphoto2
or dc raw depending on whether or not your camera takes raw images but I highly recommend that
you install all these programs to use the script to its full functionality now the different
options that you have are the first one is dash dash destination dash directory and that particular
argument accepts a location where you're ultimately going to copy these images to for me it's an
external hard drive that I connect and I mount through media dash tier and the second argument
that this script can take is a dash dash tmp dash dir and that accepts a location of where you
want to store your or temporarily save the files because it does do processing to the images it
needs a temporary directory the default would be slash temp for me I use a slash slash temp directory
that is in memory so that to make a quick but the disadvantage of having something in memory is
that I can run out of RAM so if I have a lot of pictures multiple gigabytes potentially
a may run out of RAM so I'll have to set a new temp directory to some location
the third option I have available is a dash dash no dash google dash backup if you don't want to
backup your photographs to google plus you can give it this command line option and it will
I'll skip that part the next option I have is dash dash sd dash card and that accepts a location
of where you may have an sd card from which you want to import images now doesn't necessarily have
to be an sd card it can just be any folder you can point it to any folder on your computer and it
will process the that that folder of images so I use that with pictures that are synced from my
phone through Dropbox to my computer and I'll run the script on that folder and it will
organize them into folders by date etc and upload them into google plus and whatnot the next
option is a dash dash no dash delete the script cleans up after itself after runs and it will
delete the temporary directory that it creates if you run this option it will keep the temporary
directory in case you wanted to do any post processing to those files after you run the script
and finally the last option that I have available is a dash dash backup which accepts a folder name
and an s3 bucket and this is for backing up your photographs to an s3 compatible service like I
said Amazon s3 is the the Amazon is a company that created s3 and they have a specific
API but there are other services available that have a similar API which you could use the
program s3 cmd4 to sync the files now I use this infrequently I will only backup maybe once a
year so I'll I have a whole folder of pictures that I've taken that year and then after the years
over I'll back it up into Amazon s3 so this is not something that you necessarily need to do
every day so let me go back to the dependencies and talk a little bit about how each one of these
programs is used so the first dependency is gphoto2 gphoto2 is what the program that is used to
interface with your camera directly in order to download the photographs and it's compatible with
quite a number of models of cameras now I did have a problem with one of my point in shoes in
running Ubuntu and that it was causing some problems at one point but I was able to fix it by
simply updating gphoto2 and also dcraw so if you're having any issues with either one of those
programs I highly recommend that you seek out the latest version of those and download it and install
it onto your computer the second dependency is dcraw that's to convert raw images raw
many of the new cameras take what's called a raw image which is basically a capture of all the
information that the ccd can get now in order to display an image and the image that's displayed
cannot necessarily or may not be able to show the entire dynamic range of the information that's
captured and so the camera by default selects a what it believes to be the the best
way to display an image but that's not always the best way and so a lot of photography these days
I just like to use the raw format because it it allows a greater flexibility in ultimately
editing the final picture that gets printed or displayed online now the raw format the issue
with the raw format is that every camera manufacturer basically has their own
format so the script because I have to Nikon cameras that shoot raw images
specifically looks for those file extensions and if you shoot Canon or Olympus or some other
you have some other camera you may have to edit the script a little to get it to work now the
cameras that I use are Nikons they both shoot raw images because of that my script is really tailored
to use the Nikon raw image file format if you have a different camera let's say a Canon Olympus
Sony whatever it may be you may have to change the script a little to use the raw file formats
that your particular camera creates now using DC raw it should be a pretty simple matter of just
changing the file extension that the script looks for but because I those are the only two cameras
that I have that's all I've been able to test it with so if you have any trouble let me know
and I'll see maybe I can help you out the next dependency is Google CL Google CL is a command line
utility that interfaces with various Google services and the one that I use use it for
used to be called Picasso and now has been integrated into Google Plus and essentially what it does
is take all the images that I have and upload it into my Google Plus account now you'll have to
supply it with your Google account information so the first time you run it's going to ask for
you to authenticate through your web browser and give it your email address and password and
after that initial setup it will remember that information so what I recommend if you're going to
use that particular feature of this script is to run the Google CL independently make sure that
it's registered with your Google account that you authorize it to access your your services and then
run the script because it may prevent it from working correctly if it can't get access to your
account information the next one of course is our sync our sync is the utility which will sync
the images that are organized in their folders to wherever the ultimate destination where
you're going to put them and I find our sync to be very useful because I make sure that I don't
or that I if if I have any newer pictures that it only copies the new pictures and not the old
pictures and the last one is s3 cmd which is a command line utility to interface with s3
like services and that's what I use to back up my files to the dream host cloud objects I believe
it's called I'm sorry dream objects service okay with that out of the way let's open up the
actual script in our favorite text editor I prefer emax so I'm looking at it right now using emax
and the very top of the script you'll find a number of variables some of which you may want to
change so let me go over those the that you'll there is a environment variable that is declared
called temp deer and that is the default temp location like I said it creates a folder in the
slash temp directory called photos and if that's not your preferred location you can always change that
it doesn't have to be in the temp directory it could be anywhere on your computer just set that
to somewhere where you'd want it to store these files that it's processing at the end of the
process unless unless you select the node delete option it deletes this folder so it doesn't really
matter the most important thing to keep in mind is that you have a set a location that is large
enough to store all your information and like we went over before and with some of the command line
command line options that are available to you to the script you can change it at the moment that
you run it the next one that you want to take a look at is called destination currently it's set
to slash media slash tiers slash pictures slash photos that's where I store my pictures that's
the destination that I want them to go to you'll have to change that to yours because more than
likely you won't have the same folder structure on your computer that I have in mind and then after
that you'll see the Google user that's my email address you'll want to change that to your own
email address if you're going to use these Google services a couple of defaults are set as well
Google backup is set to true and by default you can set that to false if you don't want it to run
every time SD is set to false which is the the card or folder that you want to backup images from
and no delete is set to false to s3 backup is also set to false next part of my script looks at
all the dependencies and creates variables that are the same name as the application so you'll
there's several applications that are users find gphoto2 google dc raw are seeing renamed
s3 cmd all those are created as variables in this particular script then I go through those
variables and do the sanity checks make sure all the applications are installed if it's not
installed on your computer you'll receive an error when running this particular script to notify
you to go to go ahead and use your package manager to install them and skipping down a little
farther you'll see the help text that we already went through and you'll see the part of the code
that goes through different command line options and try to identify which ones you had picked now
if there are any bash scripters out there that could offer me some assistance I've had some bugs in
the script that I can't quite figure out and that is sometimes it matters sometimes the order in
which you pass these command line arguments matters and I'd like to find some sort of way to be able to
make that less important or more robust so that it would the script would work no matter what order I
put these command line arguments I find I find that it's usually an issue when I'm mixing command line
arguments that accept an argument with those that do not but again if any any bash scripters out
there it could help me out I'd really appreciate it a little farther down in the script is where
I define the functions that I use the first function I have defined is one called convert to JPEG
which takes a folder of images and goes through all the files in those folders and uses as the DC
RAW program to convert the RAW files to JPEGs later on in the script I'll use these JPEGs to
resize them and upload them to Google Plus because Google Plus will not display RAW images so they
need to be converted into JPEGs first then the next function I have is one called resize to thumb
and that will take those JPEG images and resize them to a size of no more than 2048 pixels by 2048
pixels in my research this is this was or is the largest file size that Google Plus will accept
without charging extra after that there is a function called import photos and that uses the
G photo to program to import pictures off of the camera that I have connected now more recently
I've stopped using that method I don't no longer connect my camera directly to my computer because
I purchased an SD card reader and I prefer to use that instead so I haven't tested that in a while
hopefully it still works okay and hopefully it's not too dependent on my particular models of
camera but using an SD card reader I've found to be a lot faster and it doesn't drain my camera
battery so it's really my preferred method of importing my pictures the function after that is
one called remove spaces and that's really to make the naming of these files a little more
consistent because you know some cameras or some user default defaults may add spaces to the
files which will more make it makes it a little more complicated to deal with in the script so
I just decided let's take them all out then the function after that is to sort images that will
put them in the folder structure that ultimately I want to copy over which like I said before
a structure of a folder named after the year the picture was taken in and then a subfolder named
after the specific date in which the picture was taken and the last function that I have is
to create the s3 backup and it's called archive folder takes a folder of images
converts it into a or compresses it into a tar dot bz2 file and then uses the s3 cmd to
upload it into the s3 bucket that's already been created if you're not familiar with s3
i or the 8th s3 api it would be worth maybe another hpri episode to go over that process
but you do have to create a bucket before you use the script to upload it
moving down a little farther is the actual script part these this is the part that actually
runs and does all the work the first thing that it does is create the temp directory and moves
into the temp directory and then it checks to see whether or not I want to make a s3 backup and I
put that right there because I wanted to exit the script if all I want to do is do the s3 backup
like I said usually I don't do the s3 backup at the same time that I'm importing my pictures
so if if I do decide to do the s3 backup I wanted to exit before it goes through the process of
doing the rest of the script then I moving down a little further is what the normal work process
would be for the script which is to import the photos remove the spaces from the files
convert all the files to lowercase just to make the naming consistent sort them into the structure
that I want them to be in which is again that folder named after the year then a subfolder named
after the specific date and then create the jpegs which which will be uploaded into the google plus
now if you look and this particular section you'll see that it's using the convert to jpeg
function that I talked about earlier and it's converting all the files that are in the dot
nef and dot nrw file format now if you have a camera that's not an icon or if you have a
icon that uses a different raw format you'll have to change this particular file extension
and so it will convert all these files these raw pictures into jpegs and then resize them
into a thumbnail size then it checks if you want to upload them to google plus and if you do
it will upload them into your username into your google account in a folder called backup and the
date of the that the script was run excuse me so it uploads in to your user account to google plus
in a folder which you'll find under photos called backup and the date that this particular script
was run and then almost the last thing is to our sync the temp directory that we've sorted all the
pictures to to the ultimate destination where we're going to keep these pictures for hopefully forever
and then it does a little bit cleanup by removing the temp directory and exiting so let's say
quick overview of my script I hope you enjoyed this episode I look forward to receiving any feedback
and if you have a better way to manage your pictures or I would say a more hacker public radio
way to manage your pictures please create an episode I really find it a pleasure to listen to
so that'll do it for me again my name is Tony Polias feel free to reach out to me if you have any
questions or would like help getting this script started on your computer thanks a lot bye bye
you've been listening to hacker public radio at hackerpublicradio.org we are a community podcast
network that releases shows every weekday Monday through Friday today's show like all our shows
was contributed by an hbr listener like yourself if you ever thought of recording a podcast
then click on our contributing to find out how easy it really is
hacker public radio was founded by the digital dog pound and the infonomicon computer club
and it's part of the binary revolution at binrev.com if you have comments on today's show
please email the host directly leave a comment on the website or record a follow up episode yourself
unless otherwise stated today's show is released on the creative comments
attribution share a light 3.0 license