Building an AI Recruiter using ElevenLabs, Claude 3, and LLamaIndex

Recruiting is a critical process for any organization looking to attract and retain top talent. However, traditional recruiting methods can be time-consuming, inefficient, and often fail to provide a personalized candidate experience. This is where AI-powered recruiting comes in. By leveraging advanced technologies such as ElevenLabs for voice AI, LlamaIndex for document indexing, and Claude 3 for natural language processing, companies can streamline their hiring process and provide a more engaging candidate experience.

In this article, we’ll walk through a sample Python implementation that demonstrates how these AI tools can be integrated into an automated recruiter call workflow. We’ll cover everything from setting up audio processing for capturing candidate answers to creating a vector index for candidate documents and setting up an AI-powered chat engine for conducting the recruiter call.

Setting Up Audio Processing

The first step in our automated recruiter call workflow is to set up a method for capturing candidate answers. We’ll use the PyAudio library to record the candidate’s responses, which we’ll then transcribe and forward to our LLM (Language Model) for processing.


import pyaudio
import wave
import numpy as np
from uuid import uuid4
from openai import OpenAI

CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 44100
SILENCE_THRESHOLD = 1000  # Adjust this value based on your microphone sensitivity
SILENCE_DURATION = 2  # Number of seconds of silence required to stop recording

client = OpenAI(api_key=OPEN_AI_KEY)

def is_silent(data, threshold):
    return np.max(data) < threshold

def is_talking(data, threshold):
    return np.max(data) > threshold

def transcribe(audio_path):
    audio_file = open(audio_path, "rb")
    transcription = client.audio.transcriptions.create(
      model="whisper-1",
      file=audio_file,
      response_format="text"
    )
    return transcription

def record_audio():
    p = pyaudio.PyAudio()
    stream = p.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK)

    print("Recording started...")
    frames = []
    silent_frames = 0

    while True:
        data = stream.read(CHUNK)
        frames.append(data)

        audio_data = np.frombuffer(data, dtype=np.int16)
        if is_talking(audio_data, SILENCE_THRESHOLD):
            break
        else:
            silent_frames += 1

    while True:
        data = stream.read(CHUNK)
        frames.append(data)

        audio_data = np.frombuffer(data, dtype=np.int16)
        if is_silent(audio_data, SILENCE_THRESHOLD):
            silent_frames += 1
            if silent_frames >= RATE * SILENCE_DURATION / CHUNK:
                break
        else:
            silent_frames = 0

    print("Recording stopped.")
    stream.stop_stream()
    stream.close()
    p.terminate()

    return frames

With these components in place, we can create a short method that puts them all together:

def getAnswer():
 audio_path = 'recordings/' + str(uuid4()) + ".wav"
 frames = record_audio()
 save_audio(frames, audio_path)
 text_answer = transcribe(audio_path)
 return text_answer

We’ll use this `getAnswer()` method after sending the audio from the LLM to ask a question. This will handle all candidate-side processing for the solution.

Creating a Vector Index for Candidate Documents

Next, we’ll focus on the AI recruiter portion of the application. To get started, create a folder named `./documents` and put all candidate-related documents in it. You can use a sample resume to get things started.

Then, we can set up our LlamaIndex index and LLM AI recruiter:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, load_index_from_storagepyt
from llama_index.llms.anthropic import Anthropic

llm = Anthropic(api_key=ANTHROPIC_AI_KEY, model='claude-3-opus-20240229') memory = ChatMemoryBuffer.from_defaults(token_limit=10000)

def createVectorIndex(path): 
  PERSIST_DIR = "./storage" 
  # load the documents and create the index 
  documents = SimpleDirectoryReader(path).load_data() 
  if not os.path.exists(PERSIST_DIR): 
    index = VectorStoreIndex.from_documents(documents, embed_model='local') 
    # store it for later 
    index.storage_context.persist(persist_dir=PERSIST_DIR) 
  else: 
  # load the existing index 
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR) 
    index = load_index_from_storage(storage_context, embed_model='local') 
  return index, documents

index, documents = createVectorIndex('./documents')

This code sets up an index that we can use to create a chat engine for asking questions and processing answers. To kick things off, we can give the LLM some context about the phone call and the questions we want answered:

prompt = """"
You are a professional AI recruiter conducting a introduction call with a candidate. 
Use the query_engine_tool to request any important information about the candidate that is required for the 
conversation. In the conversation you must get appropriate answers to all the questions below. If you are not happy 
with the response to a question, drill in deeper, or ask the question another way.

Questions:
1. Can you tell me a bit about your experience?
2. Are you actively interviewing?
3. What is your timeline?

Start the call off with an introduction to get things started. Also, let the candidate know that your are an AI. 
When you have gathered answers to all questions, wrap things up with the candidate and return the text `GOODBYE`.
"""
chat_engine = index.as_chat_engine(llm=llm, system_prompt=prompt, chat_mode=ChatMode.CONTEXT,     
                                   memory=memory, verbose=True)

We use a special word “GOODBYE” to trigger the ending of the call and to signal that we have gathered all the answers we need from the candidate.

Giving the AI Recruiter a Voice

We have all the text-based question-and-answering covered, but we need to give our AI recruiter a voice. To do this, we’ll use ElevenLabs to convert our text to speech. ElevenLabs is a leader in voice AI technologies and provides many voice options to choose from. You can even clone your own voice for replicating a realistic call.

Create a free ElevenLabs account here

from elevenlabs import generate, Voice, play

def getAudio(text):
    audio = generate(
        api_key=ELEVEN_LABS_KEY,
        text=text,
        voice=Voice(
            voice_id='voice_id'
        )
    )
    return audio

Putting It All Together

Now, we can put it all together in a question-and-answer loop:

res = chat_engine.chat('`Call starting`')
print('AI:', res)
audio = getAudio(str(res))
play(audio)
while True:
    answer = candidate_chat_engine.chat(str(res))
    print('Human:', answer)
    res = chat_engine.chat(str(answer))
    print('AI:', res)
    audio = getAudio(str(res))
    play(audio)
    if 'GOODBYE' in str(res):
        print('Found GOODBYE string in the response')
        break
    answers = chat_engine.chat('`Call Ended`. Send me the questions and answers from the candidate.')
    print('Answers', answers)

That’s it! Place your phone next to your laptop and let the AI recruiter handle everything. You could even enhance this solution with an API phone call solution like Twilio.

Optimizing for Low Latency

To further optimize this solution, we can use streaming transcription from AssemblyAI using websockets, streaming LLM generation using server-sent events, and streaming text-to-audio from ElevenLabs using websockets. This This will ensure the lowest latency possible, with no recognizable pause between interactions. By leveraging these streaming technologies, the conversation will flow smoothly, making the candidate feel like they’re talking to a real person.

Conclusion

The integration of ElevenLabs, LlamaIndex, and Claude 3 represents a significant step forward in AI-powered recruiting. By automating the initial screening process, companies can save time and resources while providing candidates with a more engaging and personalized experience.

The sample Python implementation we’ve walked through demonstrates how these technologies can be seamlessly integrated into a recruiter call workflow. From setting up audio processing for capturing candidate responses to creating a vector index for candidate documents and using an AI-powered chat engine to conduct the call, each component plays a crucial role in streamlining the hiring process.

Moreover, the ability to give the AI recruiter a realistic voice using ElevenLabs adds an extra layer of authenticity to the interaction. Candidates may not even realize they’re talking to an AI, making the experience feel more natural and comfortable.

As AI continues to evolve, we can expect to see more innovative solutions like this emerging in the recruiting space. By embracing these technologies, companies can gain a competitive edge in attracting top talent while providing a better candidate experience.

If you’re interested in exploring AI-powered recruiting further, we encourage you to experiment with the code provided and adapt it to your specific needs. With the right tools and approach, you can revolutionize your hiring process and unlock the full potential of your talent acquisition strategy.

References

Contact

Open for contract projects as a Project Leader or Individual Contributor. Let’s chat!

LinkedIn: https://www.linkedin.com/in/davidrichards5/
Email: david.richards.tech (@) gmail.com