Claude 3.5 Sonnet Empowers Audio Data Analysis with Python

Claude 3.5 Sonnet, lately introduced by Anthropic, units new {industry} benchmarks for numerous LLM duties. This mannequin excels in advanced coding, nuanced literary evaluation, and showcases distinctive context consciousness and creativity.

In line with AssemblyAI, customers can now learn to make the most of Claude 3.5 Sonnet, Claude 3 Opus, and Claude 3 Haiku with audio or video information in Python.

Listed here are a couple of instance use circumstances for this pipeline:

• Creating summaries of lengthy podcasts or YouTube movies
• Asking questions concerning the audio content material
• Producing motion objects from conferences

How Does It Work?

Language fashions primarily work with textual content information, necessitating the transcription of audio information first. Multimodal fashions can handle this, although they continue to be in early improvement levels.

To attain this, AssemblyAI’s LeMUR framework is employed. LeMUR simplifies the method by permitting the mix of industry-leading Speech AI fashions and LLMs in only a few strains of code.

Set Up the SDK

To get began, set up the AssemblyAI Python SDK, which incorporates all LeMUR performance.

pip set up assemblyai

Then, import the package deal and set your API key. You may get one without cost right here.

import assemblyai as aai
aai.settings.api_key = "YOUR_API_KEY"

Transcribe an Audio or Video File

Subsequent, transcribe an audio or video file by establishing a Transcriber and calling the transcribe() operate. You may go in any native file or publicly accessible URL. As an example, a podcast episode of Lenny’s podcast that includes Dalton Caldwell from Y Combinator can be utilized.

audio_url = "https://storage.googleapis.com/aai-web-samples/lennyspodcast-daltoncaldwell-ycstartups.m4a"

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_url)

print(transcript.textual content)

Use Claude 3.5 Sonnet with Audio Information

Claude 3.5 Sonnet is Anthropic’s most superior mannequin so far, outperforming Claude 3 Opus on a variety of evaluations whereas remaining cost-effective.

To make use of Sonnet 3.5, name transcript.lemur.activity(), a versatile endpoint that permits you to specify any immediate. It mechanically provides the transcript as extra context for the mannequin.

Specify aai.LemurModel.claude3_5_sonnet for the mannequin when calling the LLM. Right here’s an instance of a easy summarization immediate:

immediate = "Present a quick abstract of the transcript."

end result = transcript.lemur.activity(
    immediate, final_model=aai.LemurModel.claude3_5_sonnet
)

print(end result.response)

Use Claude 3 Opus with Audio Information

Claude 3 Opus is adept at dealing with advanced evaluation, longer duties with many steps, and higher-order math and coding duties.

To make use of Opus, specify aai.LemurModel.claude3_opus for the mannequin when calling the LLM. Right here’s an instance of a immediate to extract particular data from the transcript:

immediate = "Extract all recommendation Dalton provides on this podcast episode. Use bullet factors."

end result = transcript.lemur.activity(
    immediate, final_model=aai.LemurModel.claude3_opus
)

print(end result.response)

Use Claude 3 Haiku with Audio Information

Claude 3 Haiku is the quickest and most cost-effective mannequin, splendid for executing light-weight actions.

To make use of Haiku, specify aai.LemurModel.claude3_haiku for the mannequin when calling the LLM. Right here’s an instance of a easy immediate to ask your questions:

immediate = "What are tar pit concepts?"

end result = transcript.lemur.activity(
    immediate, final_model=aai.LemurModel.claude3_haiku
)

print(end result.response)

Be taught Extra About Immediate Engineering

Making use of Claude 3 fashions to audio information with AssemblyAI and the LeMUR framework is simple. To maximise the advantages of LeMUR and the Claude 3 fashions, check with extra assets offered by AssemblyAI.

Picture supply: Shutterstock