“Something I admired [about Grimes] was the fact that she was the sole producer of her music. […] Grimes is groundbreaking, not only for her current involvement with technology and creativity, but because she followed what her heart and mind asked her to do” — Rooy
How to Make Music with Grimes’ A.I. Elf.Tech
On Tuesday, May 2nd, singer Grimes officially launched Elf.Tech, an artificial intelligence (A.I.) -powered application that takes fans’ vocals and replaces their voice with Grimes’.
HOW TO MAKE MUSIC FEAT GrimesAI— 𝔊𝔯𝔦𝔪𝔢𝔰 (@Grimezsz) April 30, 2023
– we can distribute it for you and you can earn royalties from your work: https://t.co/p598CXaXnD
While Elf.Tech is still in the Beta testing stage, fans have found that many of the samples it regurgitates actually sound pretty good! Indie creators, particularly in the indie music scene, are ecstatic that the now-independent singer and producer has taken such strides to collaborate with her fans.
Grimes is offering to split 50% of royalties with any independent musician who uses her artificial intelligence (A.I.) -generated vocal stems and samples to create their own music that features her voice.
“[…] I prefer organic real voices, but one time, one of my heroes taught me that no matter what medium you use, if you put heart and soul to it, it’ll feel human.” —Rooy
Before Elf.Tech Came Out: Experimenting with Different A.I. to Create Music with Grimes
“I was pretty excited when I read Grimes’ statement on Twitter about allowing her voice to be used for A.I. projects,” says Poingly, an independent muscician who has been making music for over 20 years.
“The challenge, at first, I think was finding something accessible people could use,” the creator explains, nodding to the anticipation of Grimes’ Elf.Tech releasebefore May 2nd. “Most of us aren’t going to be able to write a python script and/or have GPUs capable of doing voice cloning.”
With a background in user experience (UX), Poingly was able to make the leap into music production with Grimes before the release of Elf.Tech. “I knew of some voice cloners, so I went at it,” Poingly says.
Using Eleven Labs to Create Grimes Tracks
Poingly began their journey into collaborating with Grimes by experimenting with Eleven Labs‘ A.I.-powered speech generating software API. The mission eventually resulted in Poingly’s first “ft Grimes” track, The Fastest Path to Communism (ft Grimes). “[Eleven Labs A.I.] was able to make a decent spoken word track, but it didn’t really sing.”
Using UberDuck to Create Grimes Tracks
UberDuck is a popular A.I. tool that musicians and engineers alike use to create vocal model voiceovers, audio applications and D.I.Y. vocal synthesizers. UberDuck was quick to respond to the Grimes’ news — in fact, UberDuck deployed a “Grimes AI” button on their homepage just days after Grimes’ initial announcement on Twitter. Poingly was quick to get in on the action, praising UberDuck’s response and ease-of-use:
“UberDuck came out fast to give us a Grimes clone. I’ve used UberDuck in the past, so I think I was familiar with how their software worked. It’s actually evolved since I last used it…significantly, and for the better! You basically just have a WAV (or AIFF or whatever) of singing (just singing! nothing else!) and you upload it; then it outputs a WAV with the Grimes model singing instead (with a download button of course). It’s that simple these days, and I love it!”
The Grimes collaborator was able to create another track with UberDuck quite easily. “They keep updating their training model to really try to get Grimes just right. The old model isn’t there any more, but this was the track that I made with the old model,” Poingly says, sharing their second Grimes release, a dystopian techno track “-r requirements.txt.”
How To Enter The Grimes A.I. Challenge on UberDuck
“My name is Rooy,” a Grimes A.I. Challenge contestant introduces themself to 3BX. “For the competition, I went as Ukko Tahti, which is a pseudonym I obtained from some Finnish name generator. In Finnish mythology, Ukko is the god of thunder. Originally, Tahti was written as Tähti, Finnish for star, but Tahti, which means a bar or measure in music, fitted perfectly instead.”
Rooy, A.K.A. Ukko Tahti, learned about Grimes’ A.I. project in the same way that many other creators did. “It was by scrolling my Twitter feed, when I saw her comment on a video by Zach Wener, the founder of Uberduck,” Rooy explains. “In the video, Zach was describing the competition and the launch of the Grimes A.I. voice, which came almost at the same time she mentioned her support for A.I. generated voices.”
Rooy praises Grimes, calling her “groundbreaking, not only for her current involvement with technology and creativity, but because she followed what her heart and mind asked her to do, which was to express and create art.” A fan and creator, the Ecuador native created a tribute track with UberDuck’s A.I. software, ‘”Castle in the Sky,” featuring A.I.-generated vocals from the star and samples from Simone Bocchino (D.J. Satomi)’s hit song.
“So, just to clarify, I didn’t produce ‘Castle in the Sky.’ That was a track originally produced by Simone Bocchino, who goes by DJ Satomi, which features vocals from Valentina Dante (Rest In Peace). The reason I used the track for the Grimes competition was mostly because the track brings me a lot of nostalgia due to the fact that I used to listen to it in 2008’s YouTube, and because I wanted to make a tribute to DJ Satomi, who had some other great songs like Waves, Nuclear Sun, and Wonderland, which were predominantly popular in early YouTube. In addition, one time when I relistened to the track, I wondered how it would sound with Grimes’ voice, especially with the concept of the lyrics. In the technical side, it was mostly modification of the tempo of the track, just upping it, isolating the main track from the vocals, and using the Grimes AI software for the new vocals. I also incorporated some nightcore elements to it, such as pitch modification of Grimes AI vocals, and a bunch of reverb too. A quick fact is that in the beginning, there’s a little bit of the intro of ‘Welcome to the Opera’, which is an upcoming Grimes collab with Anyma, but it’s almost listenable.”
Rooy’s biggest challenge in creating Grimes music with UberDuck? “Making it sound like actual Grimes,” Rooy says. “[…] in the first try it really didn’t sound like her at all. There were instances where the voice sounded robotic. In those cases, the best thing to do was to use a lot of reverb, compression, and just upping the tempo.”
Because of the unique nature of Grimes’ voice, like the “incredible is the use of reverb and compression,” that Rooy mentions, recreating an authentic-sounding Grimes vocal can be difficult. Rooy’s advice? “Don’t be scared to exaggerate on reverb. I believe she mentioned that getting rid of low mids helps too. Don’t forget to experiment with acapella samples to check how it sounds.”.
Although Rooy is unsure about whether or not they’ll incorporate A.I. into their own music, the university student and creator plans on continuing their hobby as an independent artist. “I’m at the moment working on stuff for myself,” Rooy says. “I can’t say much for now, but I love to believe that I’ll make something I’d love to listen to.”
You can hear Rooy’s Grimes A.I. music here and his personal discography is available on Bandcamp. We particularly enjoyed “Lamentum” on their album “Por un instante, la mariposa que arde en mi lámpara se hace de oro.”
How to Use Elf.Tech to Collaborate with Grimes
After the May 2nd release, Poingly and an influx of other artists flocked to Elf.Tech to check out Grimes’ own A.I. software. Ben, A.K.A. Hati Hari, is a music producer based out of Melbourne, Australia who released “And I Never!” on the same day that Elf.Tech launched.
“I was already working on “And I Never!” when I read about Grimes A.I. and Elf.Tech on her instagram stories,” the artist says. “Being a complete music production nerd, I had to give it a try, so of course I used the song I was working on at that very moment to test it.”
“And I Never” subsequently took off on Reddit, with commenters praising Hati Hari for creating a track worthy of Grimes’ attention:
“Using Elf.Tech was exciting,” Hati Hari says. “It was a strange way of fulfilling a dream of working with Grimes even though it’s my own voice.”
“I’ve been recording and producing for artists for the last year or so, but this Grimes A.I. song was the first step into the “Hati Hari” project,” he says, continuing on to admit that despite creating this song just for fun, it’s gotten a lot more traction than he intitally expected. “The response from the song has really encouraged me to keep going with my own music. There’s definitely more to look forward to from this project for both myself and the lovely people on the internet.”
How to Produce Music with A.I. Vocals on Elf.Tech
“The idea of working with Grimes A.I. was so exciting to me that I knew I had to give it a try,” says Victoria Dyer, another experienced musician and music producer who couldn’t resist collaborating with Grimes.
“[Grimes] is a true legend and a constant source of inspiration for me,” the arist explains. “As a self-taught producer, she has paved her own way in a male-dominated industry and stayed true to herself and her art, which I find incredibly brave and inspiring. She’s a force to be reckoned with.”
Dyer created “posthuma” shortly after the launch of Elf.Tech, which she found relatively easy to use, especially when leaning on another popular A.I. bot API, ChatGPT.
“Initially, I had a general idea of what I wanted the sound to be like and a vague concept for the lyrics, so I built from there. I started by creating an instrumental demo and recorded some gibberish vocals to get a sense of melody and rhythm. From there, I fed drafts of the lyrics into chatGPT and mixed lines from different responses until I had the first verse.”
Her next step? Vocal recording. Dyer’s process included other A.I. vocal models as well, which she used to get her audio output exactly where she needed it to be:
“As the lyrics started to take shape, I began recording vocals. It took some experimentation to get the sound I was looking for, but I found that recording the lines in my DAW (I used Ableton Live), pre-processing them with EQs and compressors to get clear samples, and then bouncing and uploading the acapella worked best for me.” The artist says. “I would then download and import the result and process it further. In addition to using Grimes A.I. for the vocals, I also ended up layering multiple results from different voice models to get some nice textures.”
“All in all, it was a fun process,” says Dyer.
You can follow Victoria Dyer on Soundcloud, Instagram and Twitter for more inventive music and art! Be sure to pop over to her Instagram (@victoriardyer) to experience Dyer’s very Grimes-esque photography and art.
Elf.Tech Versus Other A.I.
Comparing the system to others like UberDuck and Eleven Labs, Poingly says that Elf.Tech was their favorite to use. “It felt the most Grimes-like, perhaps not coincidentally,” the musician explains. “It works mostly the same as Uberduck’s model of uploading a WAV and downloading the result.”
Poingly’s result? An calming techno ballad that teters between a catchy dance pop beat and a futuristic lofi track, “So Quiet Now (ft. Grimes).“
“[Elf.Tech] felt the most Grimes-like, perhaps not coincidentally” — Poingly
You can follow Poingly on Spotify, Apple Music and Bandcamp to check out their music with and without Grimes. We personally recommend all of Poingly’s work, but we’re especially crazy about Cinema and their singer-less album that they released in the pre-A.I. days, The One With The Singer’s Face On It.
Grimes A.I. for Comedy
The Pittsburg-based musician said he began with “the most unlike-Grimes song” he could possibly think of which, in Puerto Diablo’s case, was “I Want to Live My Life for Jesus” by Ken Snyder.
“I used the original vocal track from Ken Snyder, which caused the resulting Grimes vocals to have his inflections and vibrato, which sounded very different from Grimes. I still found it funny, so I posted it,” Puerto Diablo explains over my laughter.
“For “All Star” by Smash Mouth, I followed the same process, but I actually recorded my own vocals over the original instrumental track by Smash Mouth in order to make it sound more like Grimes. While I didn’t match every single thing about her delivery perfectly (Like her slight lisp), I really enjoyed the result.”
Puerto Diablo has more serious Grimes A.I. music coming after releasing this comedic gold. The artist explains that after doing a bit more testing, he intends to use Grimes’ A.I. software to create an original song.
You can follow Puerto Diablo on Apple Music, Spotify and Soundcloud — be sure to check out his other songs while you wait for their next Grimes track! We recommend starting with our favorite, Fast Wave.
Copyrighting Controversy for Grimes
In a Tweet posted shortly after her initial annoucement, Grimes announced that despite her hatred of copyright, her team will indeed “do copyright takedowns,” but “ONLY for [songs that have] [really, really] toxic lyrics.”
Ok hate this part but we may do copyright takedowns ONLY for rly rly toxic lyrics w grimes voice: imo you'd rly have to push it for me to wanna take smthn down but I guess plz don't be *the worst*. as in, try not to exit the current Overton window of lyrical content w regards to…— 𝔊𝔯𝔦𝔪𝔢𝔰 (@Grimezsz) April 24, 2023
Fans Respond to Grimes’ Copyright Takedown Warning
Twitter user @peacetwinks voiced her concerns and subsequently went viral after Newsweek shared their Tweet in an article covering the backlash Grimes herself has faced in releasing her A.I. stems to fans:
“idk i think you could still end up in an uncomfortable situation here; even if you try to take down an objectionable song, it’s still going to be out there misleading people until the end of time, as is the nature of the internet. be careful!!!!” — @peacetwinks
While some Twitter users have pushed back against Grimes’ A.I. project as more details emerge, others have taken a stance in defending the celebrity’s mission. In total support of Grimes, user @JacobCWalsh replied:
“This is an intelligent move. You have to expect people will use the voice in really sick ways as well, so you should reserve the right to deny them use if you find the content to be too toxic or insulting. You all really should not feel bad about that, like at all” — @JacobCWalsh
Victoria Dyer agrees:
“If I ever met Grimes, I would probably be starstruck and at a loss for words! But if I had to say something, I would express my admiration for her creativity and fearlessness as an artist.”
The A.I. Buzzword Versus Grimes
Geoffrey Hinton became known as the “pioneer of A.I.” after he developed and invented several key technologies behind the A.I. boom. Hinton recently made headlines again when he admitted that he “regrets” his contributions to the technological advancement. “It is hard to see how you can prevent the bad actors from using it for bad things,” Hinton said while meeting with the New York Times for an interview last week.
“I personally think everyone is just jumping all over A.I. as a buzzword like they did with the World Wide Web, crypto, [all of] those things. Big business will probably pervert it and ruin our lives with it, but stop coming for the artists. We’re just having fun.” — Jackson*
Conversely, Grimes is embracing A.I. with vigor. With tracks called “I Wanna Be Software” and “Music For Machines” (dropping “soon,” according to the singer), it is clear where Grimes stands on the issue. “I think it’s cool to be fused [with] a machine and I like the idea of open sourcing all art and killing copyright,” she said in a Tweet in response to the “why.”
HOW TO MAKE MUSIC FEAT GrimesAI— 𝔊𝔯𝔦𝔪𝔢𝔰 (@Grimezsz) April 30, 2023
– we can distribute it for you and you can earn royalties from your work: https://t.co/p598CXaXnD
Rooy has found themself awestruck by the rapid evolution of A.I. “You know, before 2020, I thought A.I. would only reach to levels of those support chats that appear on websites, or just the automated voice systems in calls and that until 70 years from now and so,” the creator says in reflection of his former belief. Today, Rooy feels that A.I. is an asset to our wellbeing as a species. “Today, A.I. can be used to do unimaginable things for the global community. ChatGPT-4 for example, I like to think of it as a type of maximized Google that can provide many answers to prompts that it receives, all well-written and stuff.”
Despite its benefit for the global community, Rooy is unsure about how A.I. might affect the arts. “In terms of the arts,” the creator begins. “That is a bit tricky, since there is a constant debate on the fact that many A.I. software uses art from others without the artists’ permission to generate its own artificially made image. Those are the things that divide me, because although I appreciate the ideas that A.I. can give, I condone the use of art from artists without their permission for these A.I. databases.”
Musicians Poingly, Victoria Dyer and Jackson* all agree with Grimes and are proponents of the crossover between A.I. and art. Poingly elaborates that “A.I. in art is fun! And it should be fun! One of my few fears about AI in art is that it will simply be used to be boring. I see a lot of people who are freaked out about it,” Poingly says. “But if you really want to freak out about AI in something, be afraid of AI being used by big banks and such. That’s where it actually gets scary.”
Victoria Dyer chimes in:
“I believe that A.I. is an extension of human potential and an increasingly integrated part of our lives,” she says. “To me, A.I. has the potential to enhance our creativity and productivity by allowing us to focus less on certain details or processes and more on the end result.”
When asked about the controversy, Dyer goes on to explain that in her view, “most of the digital services we use [today] rely heavily on big data and algorithms that have evolved beyond simple human intervention […] yet our experiences feel so easy. While there are certainly concerns about the impact of A.I. on society, I believe that when used responsibly and ethically, A.I. can make our lives easier and more intuitive.”
Jackson, who is still in the production stage of his Grimes A.I. music says:
“I personally think everyone is just jumping all over A.I. as a buzzword like they did with the World Wide Web, crypto, [all of] those things,” Jackson says. “Big business will probably pervert it and ruin our lives with it, but stop coming for the artists. We’re just having fun.”
Rooy weighs in in agreement:
“In the case of music and A.I. voices, I believe they can be used as instruments. They remind me of Vocaloid vocal banks,” the Ecuador native says. “Vocaloid was a software that used vocal synthesizers and had many vocal banks, the famous ones coming from Crypton Future Media. I used to experiment with Vocaloid as a kid back in the early 2010s. Now-a-days, I prefer organic real voices, but one time, one of my heroes taught me that no matter what medium you use, if you put heart and soul to it, it’ll feel human.”
As for Hinton, the genuis behind A.I. has found solace in what he calls “the normal excuse: If I hadn’t done it, somebody else would have.”
A.I. Is Polarizing — Even for Grimes Fans
Mandy, a Reddit user and Grimes fan who replied to our request for an interview, is a strong opponent of A.I. in music. “My thing is like, I just don’t think it’s a good idea,” they said. “A.I. is actually really scary and a lot of people don’t think about that. They just think it’s cool but like […] it literally stores so much information, how do we know that our voices aren’t being sold or indexed somewhere?”
Mandy likens Grimes’ project to the invention of deep fakes and speaks of the crises it has caused in the pornography industry and in the media. When asked if they would consider trying out Elf.Tech, Mandy replied: “Absolutely not, I’m good on that s**t.”
Mandy’s fears aren’t exactly far-fetched, at least according to the experts. A cybersecurity engineer who goes by Francis* explains that cyberattacks very well may start popping up in response to Grimes’ project. “I work in cybersecurity so I know how common it is for bad actors to literally create malware that looks like one thing, but it’s actually stealing your data,” the software engineer says.
“I’m a huge fan, I’m not saying any of this to bash [Grimes]. She’s incredible, she’s beautiful — but my number one piece of advice to [fans] that want to [create music with] Grimes is this: Stay away from third-party software. Stick to hers. I don’t want to hear about anyone who downloaded a Grimes A.I. app on their phone and now they’re wondering why they’re locked out of their socials and their credit card info is stolen.” — Francis
When asked about UberDuck and Eleven Labs, Francis explains that “those well-known [websites] should be fine, but just do your homework. Read agreements very, very, very closely.”
Final Words About Grimes A.I.
“I don’t know if I would incorporate A.I. in my music personally, but it’s never bad to experiment with different elements available to you, because at the end of the day, music comes from infinite sources. Be honest about it, and it’ll feel honest.” — Rooy
“I’m not super technologically inclined as most of my experience is in classical music production […] but I’m really enjoying working on UberDuck and Elf.Tech. It’s not hard or anything, I just don’t make music very quickly. I’m motivating myself to finish these LP’s by convincing myself it’s gonna help me meet Grimes herself one day.” — Jackson*
“I feel that Grimes A.I. enhanced my music because it allowed me to bring a voice into my work that I wouldn’t have been able to otherwise. Using the AI voiceprint of Grimes gave my music a new dimension that I couldn’t have achieved with traditional text-to-speech software. It also allowed me to experiment with different styles that I might not have been able to access with my own raw vocals. Overall, the use of Grimes A.I. was a unique and exciting addition to my creative process, and I’m thrilled with the results.” — Victoria Dyer
“While I didn’t match every single thing about [Grimes’] delivery perfectly, I really enjoyed the result.” — Puerto Diablo
“Grimes is fantastic. Although you’ll hear me go on and on about A.I. — is it sentient? Whatever! — I think her project is awesome and if I was a little more creative, I’d honestly be all over Elf.Tech.” — Francis*
“I started and finished the song that same day and uploaded it to a burner account on SoundCloud that night. I didn’t think much of it, it was more for personal enjoyment. However, the support on Reddit has been amazing and I’m really glad people are having a good time with the song.” — Hati Hari
“[Elf.Tech] was honestly my favorite [A.I.] to use. It felt the most Grimes-like (perhaps not coincidentally). It works mostly the same as Uberduck’s model of uploading a WAV and downloading the result.” — Poingly
“I don’t think I’m ever gonna get behind A.I. but I still love Grimes no matter what. Anything she does is beautiful even if I don’t agree.” — Mandy
About the Author
Kathryn Litton is the Chief Marketing Officer at 3BX and is based out of Research Triangle Park (RTP), North Carolina. Outside of work, Litton enjoys building and maintaining aquariums, running, camping and playing outdoor sports like snowboarding with her family. Litton is also an amateur animator on Blender in her free time, although she wants to underline the “amateur” part: “I’m still at the ‘donut stage‘ with Blender,” says the executive.