Letters of recommendation are one of those confusing but important things for job, fellowship and graduate school applications.
When you apply for graduate school or for a research or teaching job you may be asked for letters of recommendation or letters of reference. In the USA three or more letters are typically requested, and for many positions they should accompany the application. In other countries they may be requested just before you are invited to interview, or after you interview (or never). This means:
(1) have a list of letter writers ready! Before you start your job or graduate school search, ask three-five letter writers if they will write letters for you. See if you can find more than three letter writers. This helps spread the load around, and you can then use the letter writers to best advantage.
Before you ask someone to write a letter for you, ask around to find out if they regularly write terrible letters or don't write letters at all.
If a letter writer seems at all unwilling (for example, they say they do not have time, or are not familiar with the process), thank him/her and find someone else.
Sometimes you may be asked to provide the contact information for your letter writers with your application materials, even though the letter writers will not be contacted until later in the process. Ask each letter writer to confirm his/her correct title, position, address, email address and contact phone number.
Do not include all the information about your letter writers on copies of your CV or resume that you upload onto monster, geekfinder or your own personal web page. This will cause your letter writers to be spammed, and they will be unhappy. Just write 'References available on request' if you feel the need to refer to your letter writers in your publicly available job materials.
(2) choose the right letter writers! Sometimes people think that a letter from someone famous will 'sound better' than a letter from someone who knows them. Wrong! I have been on numerous search committees, and I would be surprised to receive a letter of recommendation from someone who was not an applicant's supervisor, or at least close colleague. So get letters of recommendation from people with whom you have worked closely.
For example, I have seen letters of recommendation from: an applicant's current supervisor or chairperson; an applicant's thesis advisor (particularly important for people recently out of a PhD program); someone on an applicant's thesis committee; someone for whom an applicant was TA (for a faculty position, to speak to the applicant's teaching skills); an applicant's internship mentor; and/or an applicant's senior colleague/mentor (for people several years out from the PhD).
If you are applying for a faculty position, (at least part of) one letter should speak to your teaching skills and experience. For an industry position, this is not necessary.
Sometimes I have seen graduate school applicants who send a letter of reference from their totally unrelated internship (clue: the letter writer says, "Sally would make a great box folder at your company"). Remember, if someone cannot address your fit for the position you want (whether that is graduate student or Turing award winner), then they shouldn't be writing a letter for you just now.
Bear in mind cultural differences among letter writers. For example, Americans are used to writing (and reading) long positive letters with lots of adjectives, while British letter writers are used to writing (and reading) shorter, more factual letters.
(3) read the application guidelines! Figure out for each position what they want when. If they want three letters sent directly, ask your letter writers to send them along. If they want three letters with your application materials, ask your letter writers to provide letters in sealed envelopes signed (by them) across the flap, and include these with your materials. If there are forms that need to be filled out, get those forms to your letter writers. If they just want contact information for your letter writers, just provide that.
If a school or company says they want three letters, don't try to send fifteen. More is not always better!
(3) be kind to your letter writers! If I am going to write a letter for someone, I request at a minimum a copy of their CV or resume, a copy of their research statement/teaching statement/personal statement, and a copy of the job description (where possible). I also typically do not write letters if I am given less than a week before the application deadline.
Some very kind and organized people have given me a printout or online spreadsheet (Google Docs is great for this) listing for each position the deadline, the name of the institution, the website, the job title, and why they are interested in the position or school. For paper letters, these same kind and organized people gave me stamped pre-addressed envelopes. Not only does this help me, but I am likely to use words like 'organized' and 'self-starter' in their letters.
One of your letter writers may ask you to draft a letter for him/her. Do not be offended, and do not be terrified. In this case, I suggest the following procedure: write down three of your strengths related to the type of position you are looking for; add supporting evidence for each one that this letter writer would be likely to know; and put it in paragraph form. Ask a friend to go over it with you. Ask someone at your career services center to go over it with you. Spell check! Give this to your letter writer. And do not give the same exact letter to the next letter writer who asks you for a draft!
Your letter writers will be very busy people, and they may forget that a deadline is coming up, so you can remind them. Once.
Once you have chosen a job or graduate school, send each of your letter writers a brief note to thank them, and to let them know where you are going. If someone has put in the work to write letters for you, they will want to know the end of the story!