Name: Liberty Energy: Leveraging Spotfire for operational excellence
Uploaded: 2025-07-22T14:49:34.634Z
Duration: 33 min 8 s
Description: Liberty Energy: Leveraging Spotfire for operational excellence

Transcript for "Liberty Energy: Leveraging Spotfire for operational excellence": Good morning. My name is David Van Akren, and I'm a staff engineer at Liberty Energy. Today, we're gonna be discussing how we've used Atlas or Spotfire, within our Atlas platform to leverage data to streamline our operations. Here's an overview, this morning's presentation. We'll discuss our evolving relationship with data over time, challenges posed by modern data demands, the reasons why we chose Spotfire as our analytics platform, an overview of the underlying technology stack. Justin's gonna give us a walk through of the Atlas dashboard. We'll also discuss, the support and resources that we needed in order to get to where we are in our current success, as well as future plans, for Atlas as well as other Spotfire applications. So first off, a little bit about myself. I graduated from Colorado School of Mines with a bachelor's in petroleum engineering. I've been, with Liberty eight years now. I've spent my first five years in the field. Worked in six different states. I saw temperatures as low as negative 45 degrees and upwards of a 110 degrees. Pumped up to 2,700 stages in the field, putting about 760,000,000 pounds of sand in the ground. And I've been working with power or with BI tools for about three years with an emphasis in Spotfire. Good morning. I'm Craig Hayward. I'm from, The United Kingdom. So I'll speak quite slowly so you can understand my accent. I enjoyed, Michael's Joy Division reference earlier being from Manchester, so that was very good. I don't have the extensive oil and gas experience like Justin and David here. But before joining Liberty Energy, I worked at Spotfire and Tibco, for fourteen years. So I have lots of lots of experience in architecting real time and little systems where Spotfire is deployed. I've worked in IT manufacturing, financial services, energy, and sports telemetry. I was very lucky to spend one year at Mercedes Benz Formula One team, where I modernize their data analytics platform. So I'm very happy to be here. Good morning, everyone. My name is, Justin Schneider. I graduated from the University of Alaska Fairbanks with a bachelor's in petroleum engineering. I've been with Liberty Energy since 2017. Started my career as a field engineer. Similar stats to David in the field, and then, have been working with Spotfire over the last three years creating dashboards for our end users. If you're not familiar with Liberty Energy, we are a, upstream, oil and gas completions company started in 2011. We started off in North Dakota, but we now work within 12 states, three provinces, as well as so United States, Canada, and now Australia with about 5,700 employees. Just a couple major milestones in 2013. We saw the first, dual fuel fleets deployed. 2016 saw the first, launch of our, quiet fleets that allowed us to operate within close proximity of population centers. 2020, we acquired OneStim as well as Freedom of Profit Mines. 2023 saw the debut of our Digi Prime fleets. And in 2024, as, Michael discussed earlier, we saw our CEO, Chris Wright, being tapped as the next energy secretary. So how does Liberty Energy use data? Initially, data collection was very simple by comparison to what we have today. Mostly, it was driven by what our customers demanded at the time as well as, you know, anything that we needed to accurately, report and charge our customers. There's no back end infrastructure, and as time has gone on, it was a very incremental approach. We built an infrastructure. We started getting daily, data submissions, to a central dataset. But, you know, then we started to grow really quickly, especially between 2016 and 2020. We saw a 650% increase in our crews and a 4000% increase in data volume. So every minute of every day is accounted for in all of our tech sheets, time trackers, materials trackers. We're seeing all of our consumption, and so that was still on a daily level that that would get reported, in in addition to the state g mails that would go out to our customers. But then, in 2024, we saw the need to really push forward, with with this, journey. And, so we started off by creating an uploader to take that stage data instead of just sending it to, our dataset once per day. It's near instant as soon as the stages stage is completed. Engineer checks all their data and immediately uploads it via our own uploader to the cloud. We also started, streaming our data at that point, which is the the start of our Atlas platform to where we also developed another, live uploader of our own, streaming data twenty four seven from the frac band to the cloud ready for consumption. So, you know, just to kinda give you a little snapshot of how much data we're talking about, 2023 was over 50,000,000,000 points of data. So, you know, along with this came the necessary parts of standardizing, data collection. And this started, you know, long before we began this project. But, you know, standardizing your data collection between crews, customers, basins, you know, that's the only real way that you can consume that information. So, you know, taking a step back, let's, unpack three major parts of, problems associated with data collection and utilization. So the first is combining live and static data. The second is to provide any kind of end users with a practical and intuitive user control of the data. And then the third would be to streamline data collection and transmission. So in order to address these problems, Liberty Energy saw an opportunity to innovate and develop a tool, that would, meet all of these demands. So the Zen product is Atlas, is our Spotfire dashboard built with a complex technology stack as a foundation to provide customers with actionable access to their data and operational metrics. So the first problem I mentioned is, combining live and static data sources. Historically, we've had access we didn't really have access to that live stream from the field. You know, you just aggregate it in, ASCIS and then later upload it and and provide its customers in that, you know, workflow. But now we're getting this time series data, static data, pump emissions, and we're all blending it all from the cloud level. So this gives customers, access to the live data streams, directly from the field. So it also provides them with tools for internal and external users to analyze operations, more effectively. The second issue is data is in control. So previously, you know, some of you are are certainly our customers and thanks for, being here today. You we have the ability to share the stream from from the track van. So you'd be able to see your live data as it's happening, but it would all be configured by the engineer that's, on duty at the time. So, you know, it's it's it's limited because you you lack control of, what you're seeing and how you're seeing it and where you're seeing it. And, oh, I wanna see this extra thing on the screen. Well, you couldn't do that before. So, you know, your decision making capabilities are limited in that sense. So with Atlas, you have full control over the data as if you were sitting in the chair yourself. So you're able to interact with those live streams, view KPIs in real time, configure your own live and historic PRC plots, and interact with the time one second time series data for deeper analysis. So the third step is employee workflow efficiency. So we're adding all these tools. We're adding all this functionality. We don't wanna add additional work. So ATLAS is, configured at the start of the job, and the idea is a set and forget style of data transmission. So the engineer, sets everything up, they they map all the channels, And then, you know, ideally, they don't have to think about this through the end of the job. And so we've been able to reduce routine tasks, like, when it comes to data transmission, because we're already submitting all the information. So, you know, engineers can now focus on verifying their data quality, paying attention to, the job, making sure it's executed as designed, and responding to customer requests. So why did we choose Spotfire as our business intelligence tool of choice for this application? Where we're delivering massive amounts of information to a customer in a controlled manner, where key metrics are embedded within the tool, Spotfire checked all the boxes for us. So, you know, for the first part with the time series and static data, we're able to integrate live and static data into a single dashboard. We're able to utilize Infolinks, SPDS, o data connections, JDBC. You know, I could just keep on talking about all the different data types that we use, in order to most efficiently present that information. Data can also be loaded in chunks or be using, automation services in order to just make it a lot more snappy, and feel less like a a a dashboard and and more like an experience, like a website. So for the second point, Spotfire supports in-depth analysis, you know, going back beyond your just basic data visualization, but in an accessible way. You know, Spotfire features so many, so much support for languages languages, and and it was just much easier, you know, our building window was much, shorter than it would be with any other kind of tool. And in in third point, you know, this is probably unique to us as a service company, but we needed to, but we utilize the, user management and server tools to implement row level security So the internal users can view all the crews at any time, and access all the data, but each of our customers is only able to see their siloed information. So, you know, it's able to handle this, security side of things as well. You know, for, you know, big data, it's it's complicated. It's bulky. It can be cumbersome. But when it's configured correctly, it handles those large datasets, multitude of different strategies there. You know, we're we're we're still working on some of the, ways that we're, using it. But, you know, it's it's really they will provide a experience for users that just makes sense. You know, you're not sitting there waiting, you know, and a minute or more, you know, just in order to access all of your data from, you know, say that month's, operations. And finally, you know, kind of back to the point that I just made with, ease of deployment. You know, Spotfire has extensive documentation. It's easy easily accessible resources, and and that facilitated a much more faster, development time. Justin, I think we'll speak more to that here in a little bit. And with that, I'll hand it over to Craig to give you a glimpse of the underlying technology stack. Thank you, David. So what were the key technical requirements that required this platform? Liberty Energy wanted one second data to deliver to the consumers, the crews, the field engineers, the customers, and it wanted it delivered within one second to allow them to make real time decisions. So it's one second within a second. Liberty Energy wanted to embrace new technologies, adopt modern development practices and leverage cloud infrastructure. Liberty Energy require the platform to be easily extendable with minimal technical debt and the ability to leverage data science and machine learning in the future. So how do we deliver the data to Spotfire to meet those requirements? The telemetry coming from the equipment, from the, pumps, the blenders, pressure transducers, the flow sensors are streamed from the data van, at one second intervals for each crew. The data is streamed over a Kafka event broker. This provides the backbone for all the communication between the different components of the system. All components within the platform are loosely coupled with no point to point communication. Producers of data, like the data bank, publish the telemetry data and only and applications only consume the data they are interested in. It is a fully driven, event driven data streaming platform. Data is checked and validated against an asset database. It is transformed to a common format and published out further for downstream processing. Downstream, we are using streaming analytics to augment, aggregate, and do further analytical and mathematical processing of that data. The data is persisted for historical use. The data is persisted for historical use, and in parallel, it is streamed using time based snapshots to Spotfire of the time series data. We continuously update Spotfire every second with those time based snapshots. Spotfire then combines both those real time streams and the historical data, which displayed on the visualizations that Justin will show you later. Secure access is provided to Cruise customers, engineers, that's the only data that they are interested in. So what Spotify technologies do we use? We leverage the major Spotify products with support from open source technologies, some typical technologies, and some cloud native components. Primarily, this is Spotify, where we deploy all the components, automation services, web player, all, pair and Python services. We use Spotify streaming or streaming analytics and the event processing capabilities. Stream analytics allows us to further aggregate if required and create further calculated channels either through mathematical calculations or in future by using machine learning models. This will allow us to make real time decisions on the incoming frac data using models that have been trained using the historical data. The data is cached using the Spotfire streaming live data streams capabilities, of course, snapshots and provides the continuous queries for Spotfire to consume. We have used typical BW for integration and used the typical distribution of Kafka to provide the event broker capabilities. For our storage requirements, we use cloud native database for our asset and time series data and MongoDB for our unstructured field data. So how do we deploy this? One of the key requirements was that we embrace new development and deployment paradigms. We use Azure DevOps for our source code repositories, our artifacts, the Spotify binaries, and our CIC pipelines. All our infrastructure has been created using infrastructure as components. We do we deploy everything, including Spotfire, to Kubernetes. There's no single point of failure in any of our infrastructure. All our components are deployed with resiliency, and we can easily scale up and scale down each component to meet the demands of the system. We have achieved full observability of our platform, both the infrastructure and the applications, including Spotfire. We've clicked Prometheus metrics and have customized the Grafana dashboards provided by Spotfire engineering to give us the desired level of Spotfire monitoring. Alerts are from tickets throughout the infrastructure at specific threshold. Emails are sent in real time detailing any issues in Spotfire and the wider infrastructure. So we can act immediately if we see any problems in the system. Because we've raised modern development practices, it's very easy for us to do Spotfire upgrades. We can upgrade to a new version of Spotfire within minutes with almost zero downtime. We only have downtime if we're upgrading kind of a major release where we have to do a Spotfire desperate upgrade. So what from a technical point of view then, so what are we looking to do in the future? So recently, we set up Spotify Copilot, which we want to use to boost our developer productivity and create dashboards, data functions and data connectivity. In addition, we want the ability to expose Copilot to our customers, to the crews, to engineers, allowing them to ask complex questions and get deeper insights into the dashboards and the data that they see. To add this, we have loaded documents provided by the Society of Petroleum Engineers, which we store in S3 and use the data loading capabilities of Spotify Copilot to load into our MongoDB Vector Databases. This allows us to ask very detailed industry specific questions and receive explanations in the context of decades of research provided by those papers. Liberty is a big AWS user, so so we have integrated Spotify Copilot with the large language models provided by AWS, etcetera, particularly Anthropic, Cloud, Sonae. Spotify Copilot is deployed to Kubernetes, and so we've used the same deployment and monitoring capabilities that we have used for Atlas and the wide infrastructure in general. I would now go to Justin. Thanks, Greg. Now that we've looked at how the data is routed to Spotfire and our infrastructure, I'll give you a tour of Atlas. So our our Spotfire external facing dashboard to, our internal employees and also our customers. I'll start in the the data canvas and then move on to some of the pages in the dashboard. So here in the data canvas, you can see we've structured Spotfire streaming sources by geographical location, so we kind of increase the performance of that DXP. We also utilize information links, and we run real security on that. And then through data table relations, relate all the tables to that specific one so a customer can always see their specific data. As well as info links, we also use SBDS. So we create those SBDS with automation services, that we run on a schedule update back end. So a customer logging in gets to see real time data or, gets to see updated data when they access that. And as you can see, we fully utilize, data canvas. So we we perform transformations and calculate comms when we can in Atlas and also in upstream DXPs that we use to create, the the SBDS. So the main splash page that our customers and log into is is right here. We have the frack and wireline tile where they can dive into live one second data and KPI analytics. Top left, we have the home button, take them back to the portal. And in the bottom right, we have the user guide. So this is a user guide we embedded into the dashboard, so we didn't send them an email that they wouldn't read about a PDF on how to use a dashboard. We used HTML text areas in here and, the the scroll of options, Spotfire, to really fit as much data as we can on this page. Really utilize for new users to to really figure out how to use the dashboard. Go back to the main splash page, and we'll dive into the the frac data, the crew select page. So this page here, that pad selection tile, cross table where we have customer pad crew, stages pumped, today's pump hours, if that crew is streaming and what type of job that is. So that table is is, is pretty high level metrics that they can use when they log directly in to see what's going on for that specific specific pad. On the bottom right, we have we have pad progress. So how many stages have been pumped and how many stages are to be pumped on that specific pad? Above that, we have a a map chart, which we utilize a WMS layer to show weather, the crew location, and then also some some base and shading in there as well. To the left, we have base and selection filters if we use JavaScript to run iron Python, iron Python scripts that access Spotfire's native filters. And then at the top, the master there, we have navigation options for our users to go to a specific page that they want to go to. Navigate to those page. We use markings. So on that table, you select a crew. You click on it, and then on the top, you you select that specific page you wanna go to. And then we run iron Python scripts, which through markings, grabs document proper grabs values and then paste them into document properties, which we use to filter and data limit all the visualizations. So to start out, we select a crew on the cross table, and then we'll click, an option in that set. And, we have a little JavaScript routing wheel to let users know that they're being navigated to that specific page. And this page is really the backbone of Atlas. It's the live track page where our customers can make real time decisions with live stream data. So the top there, we have the customer rep. We have the engineer on location, the treater on location, engineering, or the Liberty Energy personnel. The current well and stage being pumped, and we configure this to what you'd normally see in a frac data band. So the treatment plot on the top, the chemical plot on the bottom, the numerics to the right. And we've created a a JavaScript pop out settings menu that we we pulled from the the community site and then customize customize it for our use case where users can add channels through document properties from from a drop down list from a data table in the data canvas that we pulled in. They can choose those properties, save them to that document property, and then change colors, mins and max values, set set it how they want. And then at the bottom of that pop up menu, they can run an iron pipeline script, which aggregates those document properties into a string list, document property, and sets the data limiting for that specific configuration. As well as the treatment plot, we have the chemical plot that you can you can adjust as well. The numerics to the right, we have, well, alias options, time setting options, template saving options as well. I'll go through a couple of those here. With with the well alias, you have the option to name the well how how you want it with input field. So if you want to call it well b like they do in the field, you can name it well b and it will change on the cross table up there for you. As well as change of the well name, we have a pressure where you can set the max pressure. So on that cross table there, a trading pressure on the top right, you can input a value there, 6,000. The current value is 5,500 PSI. So it won't turn red until you type in the value of 5,000, and you'll see that cell value kind of indicate that you went over this pressure. Just kind of visual indications for our end users. And then we also can dive deeper into that specific stage. So we can data limit to the last five minutes. We can data limit to the last two hours so you can see that full stage and the time zone that you're in if you want to change your time zone. And the last option we have is template saving options. So once you save all of your document properties, once you make changes to the color, the channels, the maximum values that you want, You can save that, so we write that back to our database. And then you can also edit that template. You can share it. You can delete it. So we give them full customizability, full management of that specific page. So from here, we'll dive into the stage summary tab. So, this page is used for our customers to see completed stages already. So we tap into that live one second or that one second data historically saved here, query it, and give them the ability to market, see min max values. The top right, we use more document properties to query on demand that specifics, well in stage. And then to the left, we have a default job summary setup where they're familiar with seeing this information like this, where we have pressures, rates, volumes, all that information laid out in a nice table for them to come in and log and see. And if there's a value that they don't see in that table and they want to add as well. At the very underneath that one, we have customizable channels that they can then add using that settings menu pop out at the top. So it looks very similar to that treatment plot pop out. So they have the document properties, all the values that were submitted from field. They can choose those in those document properties, and then they'll show up in there using iron Python as well. And on the bottom, we we can, on demand query the specific design summary for that stage, so they can dive a little deeper into the analysis of that stage. And then we have the chemical, sand, and fluid pump for that stage as well. If we scroll up further here, we also have created a PDF report. So if they wanna save a report at this stage, send it off to some of their colleagues. They can do that as well. And the last page we'll go over is the the pad and well KPI. So, our end users can come here and see, you know, specific data for a pad, a well, or a stage when they log in to Alice as opposed to waiting for the end of a pad or well. So we, again, have utilized the scroll scroll page area, fit as much data as we can on a page with multiple visualization types to really, you know, make it pop for them and help them to make some decisions. And on this page, we utilize marking. So if they want to select specific data, they can select it. I think that'll be a little later here in the video. Okay. So they'll they'll be able to select that information, exclude that from the analysis, and really see a a good average for that specific that specific day or that specific path. So some of the major challenges that we face during this and what Spotfire really helped us overcome was data load times. So data load times can really prevent people from using your dashboard. It takes them too long to get in there, too long to use it. So utilizing SBDS and the automation services really really sped up loading into our dashboards and just allowing our end users to access that one file in the library as opposed to accessing multiple, multiple multiple people accessing one information link in one table. And then the ease of use. So not not everyone knows how to use Spotfire out of the box. Although it is relatively intuitive, we were able to mold it to, you know, look similar to a website to where all all of our end users can can use it intuitively and interact with it. And then the the last one was data consistency. So just, educating educating our employees to QC the data. So we made other Spotfire dashboards for them to use to to see the data that they've submitted from the field, QC that data, fix that data, and really clean it and make it make it look good for our, for our end users to use. And, of course, you know, we we've used a lot of resources to help make these dashboards and help make this this whole process. Spotfire support, Nico Maresco is a huge help. Doctor Spotfire and and Jose and the the channel there. Big Mountain Analytics, Blue River Analytics, and our our our development team. So plenty of help with this, this ongoing project. And we don't we don't stop with just Atlas. We were continuing to push the balance of Spotfire and Atlas as we continue to develop, other dashboards. So we're working to integrate fuel consumption and natural gas substitution into Atlas for our customers to see. As well as Atlas, we have a suite of complex dashboards that we use for our supply chain department. You know, these two dashboards are focused on twenty four seven sand delivery and, containerized containerized gas as well. And then that last dashboard as well, the field view dashboard, which I talked about, which is our engineers used to QC that data before it comes into, our database. So I wanna thank Liberty Energy and thank for thanks to our team for all our support, and thanks to Spotfire today for the opportunity to present.