There is no single company more closely associated with rock-and-roll than Southern California-based Fender Musical Instruments Corporation. Founded by Leo Fender in 1946, Fender started by building the world’s best electric guitar amplifiers. However, what really put Fender on the map was the Stratocaster electric guitar, introduced in 1954. Favored by rock and roll legends such as Buddy Holly, George Harrison, and Jimi Hendrix, the Stratocaster remains the most popular and influential electric guitar ever made.

“Artists are angels. My job is to give them wings to fly.” Leo Fender

Fender is famously dedicated to professional artists. In 1987, CEO William Schutz founded the Fender Custom Shop—a “Dream Factory,” which builds custom guitars for such greats as Jimmy Page, Jeff Beck, and Eric Clapton, as well as collectors and professionals of every music genre. Today, Fender has expanded into the digital and mobile landscape with a suite of products that bring music to customers wherever they live or play via their hero product, Fender Play, the complete learning app for guitar, bass, and ukulele.  

That’s where Michael Garski comes in. Garski is the Director of Platform Engineering, leading the charge into Fender’s online presence. He runs Fender’s platform engineering team which creates and maintains APIs and supports Fender’s applications. He also manages DevOps, the data team, and Fender’s data infrastructure team.

Building a Modern, Scalable App

Fender has two apps for iOS and Android devices. Fender Play, their flagship app, provides step-by-step instruction to help learn to play the guitar, bass, or ukulele. Fender Tune helps players easily tune their guitars, find new chords, and provides drum beats to accompany while playing. 

“Our demand is highly variable,” says Garski. “When we introduced Fender Play, the load was light. But it has grown since then and has some pretty impressive peaks. Christmas morning, for example, was through the roof. Everyone got their new guitars and wanted to play.”

Knowing their load would vary and would need to scale exponentially at times—often overnight—Garski built both apps using AWS Lambda. “Last year, we had a big initiative when the stay-at-home order started coming out in March,” says Garski. “We wanted to give back in the best way we knew how. Purely as a goodwill gesture, we gave away three free months of Fender Play, no credit card required, you just get a code. It resonated and helped provide a sense of relief while people were stuck inside. We hit the 100,000-user mark within 24 hours and decided to keep the offer going. As of June, we had 930,000 new subscribers in Fender Play – most of whom were beginners.  The serverless side scaled automatically, we had no problems at all with DynamoDB and Lambda.”

Troubleshooting Apps in a Microservices World

Garski’s team found that resolving issues in a serverless environment was a new challenge that rested almost entirely with the engineering team. Most challenges that arose were due to cloud service APIs that the team did not have direct control over. The remaining issues were defects in the code that only the engineering team could resolve. 

“With serverless, we found that the DevOps role is decreased since the issue is usually managed services, not hard drives going down on a server,” says Garski. “The DevOps role is more of just alerting folks that there’s a problem as opposed to actually being able to take action.”

As Fender Play grew in complexity, the team increasingly relied on cloud services for important processes like payment management. The APIs themselves could be very complex and required the team to provide specific information related to a transaction. “The first thing their support will ask us is to provide the request that called their API so they could look at the issue,” says Garski. “This was a protocol we didn’t have in place at that time so we decided to see if Epsagon would give us more information.”

Epsagon Trial

Fender decided to run the Epsagon free trial to see if it could help. “I don’t write much code anymore,” says Garski. “However, I was able to easily set-up the Epsagon agent, some of the tracing, and did some work to show it to people on the team, as well as our GM to get the budget.”

“Within a couple of days after seeing the issue with Epsagon, we had a fix out in production and were able to confirm with users that the problem was resolved. And that was just during the trial.”

Coincidently, during the trial Fender’s Customer Service team reported a new issue in Fender Play. A feature that allowed users to flag a lesson as a “favorite” and return to later was broken for some customers, causing their list of favorites to disappear. Garski’s team looked into the issue but could not reproduce the error.

“We went into our log manager,” says Garski. “But there were no response errors at all for this user ID. When we log an HTTP 400 or above, we also record the full request body so that we can replicate the error. However, in this case, we didn’t see any problem, so we couldn’t replicate it. We didn’t know what was going on. So for the Epsagon trial that was one of the first services I put tracing on.”

Garski specified the user’s ID in Epsagon to identify the function and examine the request experiencing an issue. Immediately, Garski could tell that a downstream service was causing the problem. The issue occurred when a user flagged a high number of lessons as “favorites.” The service would stumble and return an error but Fender’s app did not cascade that error up to be logged. It was also a specific condition that the team had not thought to replicate. With Epsagon in place, Garski could now understand why users’ favorites were disappearing. 

“Within a couple of days after seeing the issue with Epsagon, we had a fix out in production and were able to confirm with users that the problem was resolved. And that was just during the trial.”

Ongoing Benefits

Garski soon learned that the visibility Epsagon provided added even more value. Engineering teams like Garski’s need a deep understanding of their code in order to develop features quickly and troubleshoot issues in the code itself. This creates a challenge when onboarding new engineers, particularly under the work-from-home restrictions experienced in 2020 and 2021.  

New hires have remarked, ‘wow, this gives me a really good view of what’s going on internally’

“New hires have remarked, ‘wow, this gives me a really good view of what’s going on internally,’” says Garski. “They can visualize a request and think, ‘Okay, this invocation first calls a payment service, then we make a call to DynamoDB,’ and they see how everything plays out. Before Epsagon, you had to experience making a change to that system to really have that insight.”

Epsagon: Core to Fender’s Serverless Future

Garski sees Epsagon as core to Fender’s future. “We plan on continuing to go serverless,” says Garski. “For Fender, serverless is really more than Lambda. It means using a lot of other cloud services, for things like billing and search, as well as monitoring tools like Epsagon.”

Play On 

From the earliest days, Fender’s mission has been to help musicians “fly.” Today, that includes providing world-class online tools to help musicians grow and enjoy the music they play. In keeping with Leo Fender’s innovative roots, those online tools are built using the latest, modern serverless and microservices architectures. And, with Epsagon.