Theator provides customized video-based solutions that address the needs of surgeons with scientific rigor and is focused on improving surgeries by capturing and sharing surgical best practices that can improve patient outcomes.
Modernizing Surgical Training
Theator has brought together surgeons, developers, and computer vision experts to create a SaaS solution that maps procedures, steps, events, and crucial milestones and decision points within surgical videos.
Theator offers hospitals, academic medical centers, and surgeons the ability to review the mapped surgery to enable surgeons and their students to share knowledge and refine skills.
“If you think about it, the method for teaching surgery as a skill has not changed much in centuries—until now,” said Dotan Asselmann, co-founder and CTO of Theator with customers in the US, Canada, and Israel. Surgical training has used an apprenticeship model, limited by where and with whom you study during your training. “You learn from the few surgeons you operate with; it’s limited, subjective, and with very narrow perspectives. With such limited experience, it’s very hard to definitively determine best practices. In a world in which a basketball player can assess every movement post-game, see stats and learn from these, it’s inconceivable that surgeons today, have no way to debrief and assess their performance in the operating room.”
“In a world in which a basketball player can assess every movement post-game, see stats and learn from these, it’s inconceivable that surgeons today, have no way to debrief and assess their performance in the operating room.”
In addition, a seemingly common approach to surgery can differ from surgeon to surgeon. “There is also huge variability in decision-making within a single surgery, Dotan noted. “Two surgeons can work differently and use different techniques, and patient outcome can be different as well.”
Today, surgical videos are increasingly common as cameras are embedded in minimally invasive and robotic tools used by surgeons. Theator indexes these videos and applies sophisticated algorithms to reveal the moments that matter—steps, events, milestones and decision-making junctions.
By mapping the recorded surgery with Theator, you move beyond the traditional apprenticeship model and can assess and learn from a much wider range of techniques, approaches, and decisions, which are linked to patient outcomes. “In surgery, because of differences in both patient population and surgeon characteristics, it is often difficult to understand what best practices look like,” said Dotan.
Theator is creating the world’s largest surgical video dataset that combines a unique understanding of the patient, the surgeon, and the procedure. “Connecting the dots along the entire patient journey will allow us to really understand what best practices look like, in a scientifically rigorous way.”
“Connecting the dots along the entire patient journey will allow us to really understand what best practices look like, in a scientifically rigorous way.”
While surgeons are increasingly accessing surgical footage on sites like YouTube as an educational resource, routine review and debrief of their own procedures does not exist. And while there is a lot of buzz around AI in surgery, there are not many products that leverage computer vision to train surgeons and improve outcomes.
As CTO, Dotan is responsible for two R&D teams: one focused on cutting-edge research and computer vision and the other focused on developing software that analyzes the surgical videos.
From the beginning, Dotan understood that observability was critical to ensure the software and everything in his production environment worked flawlessly. Theator’s first observability solution did not produce the results the team needed. The solution was hard to maintain with many manual tasks, including installing an agent. And it was expensive.
“Observability was critical to ensure the software and everything in Theator’s production environment worked flawlessly.”
Monitoring and Troubleshooting Solution
Today Theator is using Epsagon to monitor its AWS container workloads. Epsagon provides end-to-end observability of the Theator environment that enables developers to troubleshoot less, fix problems faster, and build new, better performing services to speed time to market.
Theator works in the AWS cloud with a stack of mySQL for the user database and Elasticsearch for application data. The backend is Flask running on EKS containers and Kubernetes, while the front end is static and runs using React. The data pipeline uses AWS Batch. Some services are more event-based and work with Lambdas.
Theator uses containers because it is easier to deploy in AWS EKS, but a few services are more event-based and are easier to develop with Lambda functions. For example, the backend of the solution has triggering events like the ability to modify subscriber information.
Developers at Theator also tried to use logs to find the root cause of problems. “Our logs go to CloudWatch. The first solution would not let us search those logs,” Dotan said.
“Our logs go to CloudWatch. The first solution would not let us search those logs.”
At Theator, Epsagon is monitoring 2 environments (staging and production) as well as containers, pipeline and batch services, APIs and all relevant Lambdas. Theator uses Slack as the alerting channel.
Theator first indexes the procedures and can now support five common, high volume procedures. “The Theator solution maps the standard phases, common events, and common decisions in surgery videos. Then the solution uses algorithms that can digest them. The more the solution ingests surgeries, the more it learns. Epsagon monitors the performance of the solution’s application logic for bugs or errors, for example, something as simple but necessary as downloading the video.”
Theator must be able to digest and map any and all videos from multiple customers. “To complicate matters, the video is not running all the time. Rather the video is accessed at a specific time and you need to know if and when it is working,” Dotan explained. Monitoring and alerting from Epsagon keeps track of the health of Theator’s solution as it runs over time.
“Monitoring and alerting from Epsagon keeps track of the health of Theator’s solution as it runs over time.”
Overall, Epsagon has freed up developers, Dotan said, since the logs do not have to be manually searched. Instead, the metrics, logs, traces and payloads are searchable and correlated and can be seen in a single view. “You can search on a specific application in Trace Search, see the history, associated resources, and then drill down through the trace to Epsagon’s unique payload for the application, as well see any correlated logs.”
“Epsagon has freed up developers since the logs do not have to be manually searched. Instead, the metrics, logs, traces and payloads are searchable and correlated and can be seen in a single view.”
Alternatively, you also can start with an architecture view, a service map really, he explained, where you can see everything in production and the traces. By simply hovering over the component in the trace, you see health metrics and then you can drill down to traces and components showing the error, and click to reveal payload.
Fixing Issues Faster
To be on top of issues, Theator also set up alerts integrated with Slack. “We can’t lose a video or have problems mapping it. The videos, which come from different customers, can be watched for several hours. Alerting helps Therator to jump on a problem fast. So we have been mapping and fixing errors faster, and now better understand misconfigurations and logic errors, as a result of using Epsagon.”
“Alerting helps Therator to jump on a problem fast. So we have been mapping and fixing errors faster, and now better understand misconfigurations and logic errors, as a result of using Epsagon.”
“I never achieved a point where I could use the first solution for dev debugging,” Dotan explained. “The billing mechanism was not connected to the way we were working. When we have a video up, the AWS batch services run for 10 to 15 minutes. On a good day, there were 5 to 6 machines. We had to pay for an agent for each of those machines until we removed the agents. As a result, the algorithmic pipeline necessary for the videos was not monitored.”
For Theator, we need Epsagon’s deep service and application monitoring and less information about host performance, Dotan noted. “On the first day of using Epsagon, Theator developers actually discovered lambda functions with run-away invocations that we didn’t know about. We were able to debug them quickly and saved money.”
“On the first day of using Epsagon, Theator developers actually discovered lambda functions with run-away invocations that we didn’t know about. We were able to debug them quickly and saved money.”
Theator gets value from Epsagon every day. “With Epsagon, we have reduced troubleshooting time and have much better visibility into the number of errors and into the error type. We understand Epsagon’s value in giving us the metrics and data necessary to fix issues. My developers are using it every day.”
“We are debugging on a daily basis and fixing issues really fast. We are getting value out of Epsagon every day.”
Early on, Theator realized it needed to make changes to its architecture. In addition, the company was pursuing various necessary security certifications, including ISO and HIPPA. To help scale its team and add expertise, Theator reached out to Cloudride, a professional services company for managing public cloud platforms, with a special focus on security and cost optimization. Cloudride and Theator decided to keep the existing architecture and work towards specific goals, adapting and changing the platform as needed. The two companies collaborated on adding robust governance and reducing cloud costs. Alongside Theator’s development teams, Cloudride also utilized Epsagon to troubleshoot and fix issues.
“Epsagon provided, in very short time, maximum value and helped us understand customer processes, capabilities, and barriers with its visualization, monitoring and troubleshooting of the container and serverless environments at Theator,” according to Cloudride co-founder Danny Lev-Ran.
“We got immediate results from Epsagon,” Danny explained, including rapid implementation and onboarding, information in various formats and views—metrics, logs, traces and payload, as well as the ability to see everything in production in the Architecture View. “The user experience is very easy to use and its pricing very cost-effective.”
“Epsagon’s user experience is very easy to use and its pricing very cost-effective.”
According to Dotan at Theator, the collaboration with Cloudride helped speed Theator’s time to market with the security audits as well as new Theator services.
Cloudride’s services include: cloud migration, on demand services like planning and building complex, distributed systems in the cloud, cloud security, and cost optimization with strategies that can save customers money. Cloudride is focused on high value, high trust relationships with its customers as they are starting the journey to cloud or in the cloud.