Tomer Shiran Podcast Transcript
Tomer Shiran joins host Brian Thomas on The Digital Executive Podcast.
Welcome to Coruzant Technologies, Home of The Digital Executive podcast.
Brian Thomas: Welcome to the Digital Executive. Today’s guest is Tomer Shiran. Tomer Shiran served as Dremio’s CEO for the first four and a half years, overseeing the development of the company’s core technology and growing the team to 100 employees.
Previously, he was the fourth employee and VP product of MapR, a big data analytics pioneer. Tomer held numerous product management and engineering roles at IBM Research and Microsoft. He is the founder of two websites that have served millions of users and a hundred K plus paying customers. He holds a Master of Science in Computer Engineering from Carnegie Mellon University and a Bachelor of Science in Computer Science from Technion Israeli Institute of Technology and is the author of numerous US patents.
Well, good afternoon, Tomer. Welcome to the show!
Tomer Shiran: Thanks for having me here.
Brian Thomas: Absolutely. Tomer, I appreciate you making the time hailing out of the great country of Israel. Sometimes making a podcast doing one of these traversing the globe can be challenging, but again, appreciate the time difference and you jumping on to have a great conversation today.
So Tomer, we’re going to jump right into your questions. Can you share your journey from being Dremio’s founder – serving in various roles, and how has your role evolved as the company has grown?
Tomer Shiran: Yeah, for sure. I actually founded Dremio back in 2015, and I was the CEO for the first five years. Really a, a great journey.
We went from, you know, of course you know, two people in a presentation to you know, dozens of very large enterprise customers you know, over a hundred employees and so forth. And, these days, I’m the chief product officer responsible for all things, you know, roadmap, product strategy and, and, you know, everything to do with what we’re building and have a lot of fun working with many of our customers everyone from kind of the world’s largest you know, tech companies to, you know, a lot of the Fortune 500, you know, financial services Insurance companies, really everything in between
Brian Thomas: that’s awesome. I appreciate the story how you pivoted obviously starting a company you know, close to 10 years ago and serving in those various roles obviously makes a difference. You’ve learned quite a bit during that time, obviously, and Tomer Dremio recently introduced its 1st gen or AI infused tool. Text to SQL aimed at enhancing user productivity by converting natural language queries into SQL code.
Can you elaborate on the inspiration behind this innovation and its impact on users with varying degrees of data literacy?
Tomer Shiran: Yes, of course. We You know, like a lot of companies recognized the opportunity you know, with gen AI, with LLMs. And there are actually a variety of areas that we’ve incorporated the, these capabilities into our product.
And one of them, as you mentioned, is text to SQL, the idea there being you know, how do we, how can we make it easier and more democratized to ask questions about data? And so, if today people have to know how to write you know, queries in the SQL language, and a lot of people do know that, of course, but there are a lot of people who don’t.
Or you have to know how to use like a tool such as, you know, Tableau or Power BI, you know something that can generate SQL. Increasingly, we see an opportunity for English and other languages to be the languages in which questions are asked. And so, by providing kind of the capability to translate.
You know, just a natural language question into a SQL query that is structured and can actually go and execute on the customer’s data, the company’s data. And it just makes it a whole lot easier to ask questions. But that’s not the only thing the only opportunity you know, with Gen AI, for example you know, we’ve added capabilities around automatic understanding of the data.
And so in Dremio, you know, we provide kind of a native catalog of the data for the organizations that that use Dremio. And in the past, they could describe a kind of a data set, provide a kind of a description or kind of like a wiki, basically describing every data set every column.
If they wanted to things like that, they could even label data sets or columns to say, this is a personal information. Sensitive info, this is a you know, specific kinds of information using these labels. Well, now we also offer within the ability for the system to automatically figure all of that out.
And so, it would go document the table for you by understanding not just the schema and the names of the columns, but the actual values in the data. And kind of reasoning from that to understand, okay, this is actually, even though the column is called. I don’t know, maybe P for postal code. These are, you know, American zip codes.
And so being able to figure that out and write that down and make that then searchable. These are the kinds of benefits that you get with these days.
Brian Thomas: I love that and it’s making it a lot easier for folks that traditionally maybe weren’t in it or didn’t have that education kind of get into that fold.
We know there’s always a bottleneck when people are trying to get. You know, data out of a particular source or system, and they’re waiting on someone that knows how to write SQL. So, this is certainly something that I love to see, and I appreciate the technology you guys are developing. So, thank you.
And then the next question for you, Tomer, Gen AI is poised to transform data engineering, analytics and science. As Dremio embraces this technology with enhancements like Texas SQL. Autonomous semantic layer and vector lake house, what challenges and opportunities do you foresee in integrating AI into data analytics platforms?
Tomer Shiran: Yeah, I mean, it’s a, it’s a, it’s a really interesting and evolving space, of course when we look at, you know, the, the companies, and like I said there are now many hundreds and thousands actually of companies that use Dremio and many of them are large enterprises around the world. They have a lot of use cases for doing a lot of really interesting things, everything from, you know, 1, 1 company shipping companies is using to extract structure information from contracts that are written in PDF and word format.
Right? That I was just talking to. So, all sorts of different use cases in different industries. But fundamentally to get the best kind of results you know, they have to leverage their own data, right? There’s only so much you can do by using, you know, an off the shelf kind of you know model, such as you know, GPT 4, right?
Because that doesn’t understand kind of your domain. And also, there are cost reasons to not use the most generalized kind of model like that, right? Whereas if you use a smaller model. That’s more specific to your domain and specific to your data. You need actually better results at a much, much lower cost per prompt per query.
And so these are some of the challenges that are being addressed and what we offer at Dremio, because we offer a kind of a an open and scalable lake house platform where companies can very easily store and manage their data. And it gets automatically optimized and organized for them. And, you know, queries get accelerated for them.
That makes it a whole lot easier for them to. You know, manage all this data, feed it into models using, you know, things like fine tuning, or even just kind of prompt engineering or even, you know, build their own models. So, you know, Dremio and in general kind of Lakehouse becomes the foundation for a lot of these Gen AI use cases that our customers have.
Now that’s just 1 category of, you know, things that we’re, we’re doing with Gen AI is kind of enabling. Our customers to benefit from in the best way, but also even within our own product, you know, in Texas equal and the autonomous semantically are examples of that where we’re basically creating a platform that that is much more autonomous.
That anything that’s been created before and so you mentioned really self-service, right? Is it an important element that every company wants to achieve, because if you have to wait for a technical people. To do something every time you want to ask a question or look into something, ultimately, that’s going to be the bottleneck.
And by eliminating that bottleneck. And empowering more people to do things on their own a platform that allows that can just drive a lot of productivity and efficiency for these organizations. And so that’s what we’re doing. Also, in terms of our internal use of within the product to. You know, automatically produce the SQL queries to automatically document data, to automatically label data and with our last most recent release, even automatically creating different materializations of the data, such as aggregations and, you know, partitions of the data so that queries can be much, much faster without any effort from the, from the organization.
Brian Thomas: I really love that. And I’ve always been. I think it’s important that people have that self-service philosophy. Obviously, you want to have the full service available for use case business case or particular customer consumer. But self-service takes it to that next level. It really does. And it empowers people.
With the data that that’s really there. So, I appreciate the share, Tomer. Tomer last question of the day, looking ahead, what emerging technologies or trends do you believe will have the most significant impact on the data analytics and management sector in the next five years?
Tomer Shiran: Well, we talked a lot about gen AI already. And I think that’s obviously one of the, one of the key ones. And I’m a big believer of this. This is unlike many other trends we’ve; we’ve heard about in the last in the last decade, you know, web three, you know, crypto, all these kinds of things. I don’t think those are going to change the world, but I do think gen AI and LLMs are going to change the world.
And so that’s one of them. But there are other areas, for example you know, if you’ve heard of the project called Apache Iceberg. That’s an open table format that is creating a lot of opportunity for new innovation in the, in the data stack. And so now there’s a way to store data in systems like S3 and Azure storage and so forth in an open.
Format which basically allows tables to be accessed, both written and read by a variety of different tools. So, by having, like, a standard way that the industry is now kind of coming together around. A standard way to store data to store tables, structure data, specifically that’s going to create a lot of opportunity.
For, you know, lower costs for organizations for. You know, more flexibility around the, you know, which tools they can use and how they use them all on kind of that shared the shared data. So that’s another really interesting, I think, a technology and project that’s worth tracking. And then Apache arrow.
It has really emerged now with, I think, something like 100,000,000 downloads a month as a standard way to represent data in memory for fast analysis. And we were actually a Dremio, the creators of Apache arrow. And our entire engine is built off of it. And what we’re seeing now is that that is also becoming the standard transport mechanism.
So, different tools, for example, you know, are adopting arrow as a way to fetch data from databases, replacing kind of the old stuff, like. You know, and so by making it really easy for developers, whether they’re using, you know, Python or rust or go or any other language. Be easy for them to interoperate and access data that just makes it so much easier to build applications and to leverage data. And so that’s another technology. And I think that’s worth keeping an eye on.
Brian Thomas: I absolutely love that. And that’s what the whole podcast here was formed about is, is learning what, how we’re making the world better. And obviously we focus on emerging tech. So, I appreciate you highlighting Apache arrow and iceberg.
That’s phenomenal. And I do see a lot of promise in that. I’ve been reading up on a little bit of, of the technology there. So Tomer, it was such a pleasure having you on today and I look forward to speaking with you real soon.
Tomer Shiran: Likewise. Thank you for hosting me.
Brian Thomas: Bye for now.
Tomer Shiran Podcast Transcript. Listen to the audio on the guest’s podcast page.
