Name: Launching Winning Features Every Time with Mixpanel Experiments
Uploaded: 2026-04-08T06:16:04.410Z
Duration: 56 min 12 s
Description: Launching Winning Features Every Time with Mixpanel Experiments

Transcript for "Launching Winning Features Every Time with Mixpanel Experiments": Hi, everybody, for joining. We will wait for one or two more minutes before officially beginning. Okay. We will kick off in around, like, one more minute. So we'll get settled in. You know, we have, like, some people in the chat already, so feel free to come in and introduce yourself as well. Okay. Alright. We shall begin. Hi, everybody. Thanks so much for joining us today. I'm excited to welcome you to our session on launching winning features every time with mixed panel experiments. So over the next forty five minutes, we will be exploring how modern product teams are increasing their odds of shipping successful features, not by guessing, but by validating every step of the experiments using data. So for today's session, it is designed especially for product and engineering teams, as well as marketing, design, and UX teams who partner closely to ship customer facing experiences. We also always have our analytics team who are the backbone of data driven decision making. It's great to have you here as well. And for today, our goal is to show you how experiments sits at the center of building products. We will be walking through a very practical narrative. Firstly, an introduction on how teams are moving from insights to hypothesis to validation and then measuring impact of your experiment all within the Mixpanel platform. Everything will be grounded in a clear fintech related use case, and I will be walking through a demo with you end to end so you can see exactly how this is going to look like. Alright? And at the end of today's session, we will be walking away with, you know, a clear understanding of why analytics and experimentation live alongside each other, why this combination matters in today's fast changing product and AI landscape, and how we can actually start using this to launch our new features with confidence as well as speed. So before we dive in, we will do a quick review of what we'll be looking at today. First of starting off on a primal on why experimentation and analytics belong together. This is really the foundation for understanding why teams that combine both are able to ship better features and faster. We will then spend the bulk of our time, like I've mentioned, in a live demo. This is going to be driven on a fintech related use case, and we will show you how Mixpanel can be used to support the entire cycle from identifying an insight, forming your experiments hypothesis, running your experiments, and then measuring your experiment impact. So this is going to be very practical, showing you the end to end use case. We will also be wrapping up with a q and a, so feel free to drop your questions in the q and a box at any time throughout the webinar. We will be keeping an eye on them and address as many as we can towards the end of today's session. Alright. So a quick introduction to myself. I'm Celine. I'm a solutions engineer at NextPanel based out of the Singapore office. I've spent the last several years working across the product and analytics ecosystem to help teams to build strong data foundations, run better experiments, get better analysis, and also ship features with more confidence. So in my role as a solutions engineer, I work closely with the product engineering, design, and marketing teams across APAC and get them through what we are now calling the modern AI analytics era. It's going to be an exciting space where companies are not just tracking what happened for the analysis, but using it more proactively for you to steer decisions, build faster, and then accelerate your growth. Alright? So we are also keen to hear a little bit more about you. There will be a poll coming up within this session itself to ask you what's like your day to day kind of concerns. So feel free to to answer that. That will also help us understand more around your priorities and what types of questions you may have later when we show you the demo. Okay? Now let's kind of, like, set the stage for, you know, today's session. Right? Why are we talking about analytics and experimentation? Basically, innovation is moving faster than ever in today's world. Competitors are pivoting quickly, broadening feature sets or, you know, focusing on, like, a more narrow kind of capability. We often see our user features and product features evolving constantly, and user expectations from your apps and products can also shift very quickly. So in that kind of environment, we need to really shorten the cycle between our insight, decision, as well as action. And in order for us to truly build at the speed of innovation, teams will need a continuous loop and not con disconnected tools and silos that you have to manually stitch together. So mixed panel analytics is going to help you understand what is happening in your product. Our experimentation model will also help you validate and plan what should be happening next. So these two different functions, experiments, as well as analytics, when they live in different system, three things are going to start to break. Firstly, the metrics that we see in the ecosystems I'm not going to be matching across different tools. Conversion rate in your experiment may look different from, like, actual successful transactions in your application itself. So you are going to be debating is the number that we are looking at actually tracked correctly or not. Secondly, analysis also becomes very fragmented because you will be working with multiple, like, data sources and also struggling to figure out what might be your main source of truth. Jumping between dashboards, exports, spreadsheets to answer your questions is going to make this process with, you very friction a lot of give you a lot of friction and make it very difficult for you to answer your simple questions. Alright. The third one is also the most important in today's day and age. Insights, we want to translate them to actions as quickly as we can. So by the time, you know, we want to align on the data, set up an experiment, and get it running, the opportunity to launch something new may already have changed to something else. Alright. So this is going to be the gap that makes panel experiments is designed to close. We want to bring analytics and experimentation together so that you can move from insight to action in a single workflow and build better and then build faster. Alright? So that's what we'll be walking through today. Alright. Going on to the next slide. Alright. So just wanted to share with you the shift in what we are seeing across, like, the most successful product teams. Right now, teams are going to be shipping a lot more experiments. That's also driven, you know, very majorly by AI lowering the cost of building and shipping products. Whenever we have an idea on what may, you know, be used as leverage to improve our conversion rate or our engagement rate. Right? It is going to be so easy for us to generate our product variations within minutes or within days instead of within, like, a week long cycle. So that means your feature shipping is going to be a lot faster. Your iteration cycles are going to be shorter, and that also gives us the need for running more AP tests simultaneously so that we get a higher velocity of innovation. Secondly, we also are going to see a lot more focus in personalization of user experience. So users are also going to expect a lot of AI power and individualized experiences. And instead of one specific product flow that you may want to, like, roll out for everybody, you may actually have, like, feature gates or dynamic configurations where you are going to have multiple types of personalized variance, and you want to understand how can we actually measure success across this kind of broad personalization, and how do we optimize our metrics when we are no longer looking at a single user journey or a single user metric in order for us to instrument all of these updates. Alright? So teams are no longer treating analytics and experimentation as two separate workflows. We basically want to bring them into one unified and data powered loop. Alright? So you should be able to see this feature experimentation cycle over here. This is how we envision you unifying these pieces. Your workflow essentially becomes a straight line. You move from insight to hypothesis to testing to validation as well as impact all within a single process. You do not have switched your systems or reconcile your numbers. We want to help you innovate quickly by bringing your analytics and experimentation together on one trusted layer. Alright? So with this kind of, like, you know, new feature experimentation cycle, we want to help you iterate faster, get cleaner data, be more confident in your decisions, and help you get every feature release to be measurable from day one. Alright? So what does that actually look like for us? Right? In the mixed panel demo, you will be seeing two new capabilities in the UI. The first one is going to be there on feature flagging capabilities that allow you to do the planning and the running that you see on the first on the left hand side. Alright? Planning and running over here. Once you launch this feature flag or, you know, the experiment, what will actually happen is that we will start to collect all of that data for you in real time so that you can move on to experiment analysis very seamlessly, call and diagnose whether the experiment has actually been successful or not. Alright? And to prime everybody for the demo, let's ground this in a real example of what has been possible with our existing customers. Alright? So for today, we are going to focus a little bit more on the fintech world, and I will be sharing with you to start the story of one of our fintech customers, STEP. So STEP is one of our early users of mixed panel experiments. They are a digital bank that has been designed especially for teens as well as young adults. So what they needed to do was to appeal to this generation of young tech savvy users with a good user experience that will set them apart from other platforms. Alright? So the goal is to set apart and then become these users' primary bank accounts. To drive this shift in the user behavior, we need to lean heavily on Mixpanel for not just analytics, but also experiments. So by running a series of experiments within Mixpanel, STEP was actually able to identify exact moments as well as behaviors that were key and had a high impact in creating sticky users that eventually went on to make step their primary banking account. Then after identifying, you know, this flow to get users to make step their primary account, we can use these insights to test targeted future changes and nudges to the user experience. The result of this experiment and analytics close look is to see, you know, an increase in users setting up the deposit and also making step their primary bank account within the first thirty days. And more importantly, we are not just seeing a lift in a single metric. We are also helping to support a culture of experimentation where ideas can be validated, every new feature can be released, and teams can decide to move on with clarity on whether a feature is working or not, whether something deserves more effort or should be decommissioned. So this is exactly the type of impact that we are trying to enable. And in the demo, you will see how things similar to STEP are running this process inside Mixpanel. Okay? So we will be going into the demo of the Mixpanel platform. There will also be a quiz coming soon after this demo to kind of, like, recap your knowledge on what we've covered. So, like, you know, do pay attention if there are any things that, you know, pop out and, you know, catch your eye. Do feel free to, like, put something inside the q and a box. Okay? So I'm going to stop my my slide deck, and then we'll move into the browser instead where we can see an example website. Okay. So now we are in an example website for the company fixed panel. This is going to be an example fintech, you know, company where users can essentially, you know, use it like a bank. They can create accounts. They can apply for credit cards. And most importantly, they can make different type of trades for investments. So we have, like, a working website where we have different features for the fintech capabilities, such as investment tracking, bill management, for example. And you can also, like, try to use the product out, kind of, like, determine if you want to, like, move forward with this fixed panel bank as well. Okay? So when we click and, you know, interact with all of these different capabilities on the website, This is going to immediately trigger different types of events to Mixpanel for you to analyze, hey. What is adoption looking like? What is conversion looking like, for example? Alright? We are now going to reframe this in the context of an experiment. Okay? So imagine you are someone who looks at all of the metrics and the overall health of this, you know, fintech iBank over here. Alright? I'm going to start off with my metric tree over here. Alright? So this chart is going to be like a very unique chart that basically tells me how is the health of all of my metrics. Alright? At the top, I'm going to see my total revenue that I've gotten, alright, over the last thirty days. If I have all on the individual cards over here, I can also see week on week growth, quarter on quarter growth as well. Alright? And I will be able to understand on, like, the activation phase, what my KYC is looking like, how many active traits am I getting, and whether I'm able to grow my active users or not. So we have this as, like, a high level high level way to understand how our metrics are performing. Let's click on something important like trades, for example. Alright. I can see that there's, like, 6,000 trades recently. Over the last week, there was, like, an improvement week on week, But quarter on quarter, there's actually, like, a drop over here. Alright? And we go down, we can see actually the drop is mostly in mutual fund conversions. Now if I click onto this particular card, right, there are a couple of things that I can do. I can go in and understand what's happening, but I also want to tie it to the experiments that I have been running. So let's say I want to have, like, a goal of, like, improving my trade conversions over here. Alright? I start to launch my experiments. Alright? I want to I can actually track them over here and put a logbook entry to indicate that, okay. I set up my planning on the March 1, and then on, like, the March 16, I've been running on my experiment. I should have around three weeks of data right now. Alright. So from here, what I will want to do is go in, create an experiment within Mixpanel. So how can we do that? If we go to the left hand side, we will actually be able to see there are two new sections over here on the Mixpanel UI. If you go into feature flags, that's where we want to begin our planning as well as as well as our launch. Okay? If you go into new feature flags over here, alright, there are going to be three options for me. Alright? The first one is going to be that of a feature gate. Second is going to be that of dynamic config. And third is going to be that of an experiment. Alright? So the main difference between these three different options is what kind of information gets, you know, passed from Mixpanel to your app logic as well. If it's a feature gate where all you want to know is, okay, is the user enrolled and able to see perhaps this new feature that I'm, you know, building. Alright? But it's a yes, no kind of switch that I want to let the app user know, That's what I will use the official gate for. If I'm running the most classic type of experiment where you have, like, one control group and maybe two different types of treatment groups, that's where we will use the experiment option. And the third one is going to be dynamic config, which is very helpful for, you know, our age of personalization where, you know, I'm going to send specific payloads back to the user and based on what types of config that they have, any types of context that happened to them in the user session that will actually impact the different types of user experience that they see. So the difference mainly here is in what gets returned to the user. Alright? We're going to start off with the classic example of an experiment. Okay? So I'm going to create an experiment over here. I'll move on to the next step. I have already put in, you know, an experiment name, like, my Fintech onboarding update. Right? I want to change the UX so that I can increase my KYC conversion rates, or I want to increase my overall trade conversion rates. Okay? Over here, we can see under the flex settings that we can choose to target users via a few different ways. Alright? If I click on this piece, you can see that I am, by default, targeting users by the device. Alright? But if you are already a Mixpanel user or if your app deals a lot with, like, locked in users and you want to change the experience of your existing and known users, I will recommend you to use user as the assignment key. So we'll target based on user ID as well. If you also happen to use the mixed panel group analytics capabilities where you are tracking behaviors on, like, a company or, like, a shared account level. That is also something that we support. Okay? So we'll go a little bit further, and we can see that for the flat type, we have multivariate. So this brings us back to that previous to the classic, like, experiment setup of, like, one control, and then maybe we have, like, treatment one and then treatment two, for example. Okay? So these are options for me to specify over here. And one thing that I'm also able to, like, specify as part of the flexibility, right, is to decide whether I want the user variant assignment to be sticky or not. So what this means is if the user let's say for me, if I open the iBank app, right, and then I get assigned to treatment one throughout the entire life, the duration of the experiment, you don't want me to see any other options. I have to stay in treatment one for the entire life of the experiment. Just set this to sticky. You can set that on many variants if you want as well. Okay? Alright. So now that we have, like, you know, the option to add multiple types of treatment groups over here and also remove them. We'll go a little bit into the rollout. Alright? That means how can we target the users. Okay. So I'm going to go a little bit further down in the menu. You can see under the QA section that we can actually specify different users to validate the flag. So what you can do is just specify your different users by email or by username to kind of expose these users to the experiment first. Alright? The most important part of, like, our targeting over here is to make sure that we can target accurately and we can also target quickly. So we'll cover these in our roll out groups. Alright. First thing that I wanted to talk about was how can we target accurately. Alright? So you can see over here, alright, that we have some example events that I'm looking at. Alright? In the fintech world, I want to look at maybe high value users, or I want to look at users who have not reached a certain minimum spend threshold, for example. I can go in and take a look at purchase complete. Alright? And if I have a rule of users who have spent more than a thousand dollars on average, that is exactly how I'm going to specify that inside my rollout group. We have options for you that can very powerfully determine how you want to specify this this definition. So I'm going to look at my average spend amount has to be spend amount on other value has to be more than $1,000. So that can mean, like, users who are trading more than $1,000 on average in the last thirty days. So one once we do that, we will also be able to see that this calculation is immediately run-in real time, and we can determine that, okay, there are going to be, like, this many users within my cohort. Alright? This is going to look very similar to the types of Mixpanel cohorts that you are used to. So any types of cohorts that you had also defined previously within your Mixpanel project, you can also redefine them and just reference them. Alright? That is going to work, and this is what we mean by keeping the experiment piece as well as the analytics piece very closely tied to each other. Okay? Now so this is going to be on accuracy. If I add more layers over here by looking at more events, I can also put that in. Alright? The next part about rollouts. Right? We want that to not just be accurate, but we also want it to be timely. So let's say the moment user has, you know, encounter an error event or the moment that user receives like a saw like a quantity is invalid kind of message. We immediately want to trigger something for them so that they get, you know, put onto the right path for conversion. So how can we actually target the users in real time? Alright? Mixpanel now has two options for you to do that. Alright? The first one is going to be looking at your runtime events. So that is exactly what we talked about. We bind this very closely with the other events that you track on your project. So the moment, let's say, the user input has changed. I want to launch something new to the user or the moment the user sees, like, an error message or the moment the user, like, does, like, a big click more than, like, 10 times, for example. I'm going to try and launch something else to them. Alright? The other option that we have as well over here is a runtime property. So this can be something that exist inside the context of the app itself. Let's say, if I want to only target the user when they reach, like, page number five, for example. Alright? What I want to do is track something like page number, alright, as a runtime property event. I will tell Mixpanel, keep listening for this value, page number, the moment it hits, like, user hits page 10, for example. Alright? That is going to help us figure out who the this is gonna help us qualify the user in the cohort in the rollout group and then target them immediately. Okay? Now once we have these different types of rollout groups defined, alright, and we can target the users in real time, let's go on to the rollout conditions. Alright? So by default, I'm targeting my users by device IDs, and I've created, like, you know, one control group as well as, like, two different treatment groups. Okay? If I want to specify that split, how many percent should get control, how many percent should get treatment one, how many percent should get treatment two. I can specify that over here. Maybe I'm going to put, like, a 34, 33, as well as 33. Okay? And then I can see this. Imagine if we have, like, 1,000 users in this cohort, do we want to target all 1,000 of these users, or do we want to test the experiment on, like, 10% of them first? And if we are seeing some statistical significance, we roll that out. Okay? So that's an option for you. You can just set your rollout plus condition to, like, 10%. We will still continue to keep this control, treatment one and treatment two split in the users in terms of rollout to ensure that when we start the experiment tracking, this also conforms to this ratio that we've set. That's also one of the validations that we've put inside our experiment analysis report. K? So now that we have, like, prepared our feature flag config, what I want to do is hit save. I have my feature flag key over here. Alright? This is something that I want to pass to the developer team. The Mixpanel feature flag SDK is something that is very easy to add on to your existing Mixpanel SDK implementations. You basically just need to update your SDK initialization calls to listen and specify, okay, the user is also I'm also going to check for this particular experiment. Is the user part of it and what they will be for? And then we also have additional SDK calls that allow you to check if the flags have been updated for the user and also specify at that specific moment in your mobile app or in your web app that you want the user to be enrolled in the experiment. So all of these, the experiment planning, initialization, as well as tracking can be, you know, very granular and fine tuned within your control as well. K? Now now that we have this and we pass this to, like, the developer team to put it in the implementation code, we are going to start to have, like, more questions, you know, more events coming in. Alright? And we'll be able to see, you know, what the user what the user enrollment is looking like for the experiment. So we'll go well, what we will do is we'll move into the experiments report over here where we can see all of the different experiments that have been running. Alright. For this particular Fintech example, I'm going to look at, you know, an example report over here. Alright? So let's take a look at how we want to analyze the experiment. I'm going to hide this first. Alright? First thing that I want to do is definitely check my hypothesis. Right? So my hypothesis was my observation was that trades were down. Alright? They had a quarter on quarter trend of going down. So I want to check if improving my UX will affect my trade conversion rate positively. So we'll be able to track that inside the experiment report. We will also want to keep an eye on the exposure. Alright? So previously, we looked at an an example of, like, setting the different controls and the different treatment groups. I have, like, you know, one control group and five different variants over here. Okay? So I can see how the user enrollment has been, like, for control as well as for the different variants. Alright? Day on day, I'm able to understand how many people joined, you know, variant d or variant a, so on and so forth. If we hover over this piece, right, you will also notice that there is some exposure analysis that's automatically run for us. The first piece over here is called a pre experiment bias check. Essentially, what this is going to be is a retro AA analysis. Let's say we launch our experiment in October. Alright? And I have my six different groups for my experiment. I want to make sure that for these six different user groups, before and after the experiment, I wanna make sure that the metrics that, you know, they were that we were looking at for these users, alright, were properly randomized. So there are no significant differences in the pre experiment period before October for my control a, b, c, and d. So what this will do is will take your past data, we track the mix panel before your experiment has been launched to help you run this analysis. So that's another example of our experiments and analytics being very closely tied together. Now the next piece that we talked about, alright, sample ratio mismatch. So if I go in and I look at my feature flag, alright, I had this rule. Right? It had to be 34, 33, and 33. If I want to enforce this in my in my data collection, I can also do that by configuring this. So I can specify, alright, for my control, for my a, b, and c, what the target and location is supposed to look like, Mixpanel must also automatically tell me if it is not complying with the rule that I've set. Okay? So that's the first thing. Now when we come in and we measure our experiments, alright, we can take a look at what we've put over here. I've grouped the metrics into three different categories. Our primary metrics, such as my trade conversion, my KYC conversion. Alright? These are going to be, like, the main things that we want to measure our experiment against. Ideally, we keep them closely tied to, you know, our monetization. Right? So anything to do with, like, our overall conversion, let's focus on that as well. We will also have some gut real metrics over here. There are important metrics that you want to make sure don't drop too much during the experiment as well, which is going to be like BAU types of metrics such as number of use users who have completed KYC as well as the average trade amount. There may also be additional types of metrics that you may want to be measuring. So I've put something like my wish list to trade kind of conversion rate. Alright? Now how do we actually go in and analyze the individual metrics one by one? So first thing that we want to do for the metric, we'll take a look at the p value first over here, which is going to be our statistical significance. So we check whether we have that. If that is good, then we'll go on to the next few steps. So over here, you can see we have these, like, annotations in green that tells us, okay, we've achieved statistical significance. If it is in gray, it's telling us that it's not. Alright? If it is red, it's showing that it is, but the change in our metric is actually going in the wrong direction where, you know, KYC conversion or our trip conversion is actually going down. Okay? So if that is good, this is looking good, then what we'll do is we'll go into Lyft. So Lyft over here is basically calculating what is that percentage difference between control versus variant? So while we're taking a look at variant a, b, c, and d of my experiment, it is telling me, right, variant b and variant d actually doing very well in the trade conversion. Right? Huge improvements over here. Alright? Over a 100% improvement, over 89 improvement. But at the same time time, they are really affecting the KYC very, very, very badly as well. Right? Everything is going down by quite a quite a bit as well. Alright? So this is what we want to do. If we can take a look at this number over here, right, and we are able to understand, okay, why is my KYC looking so bad? Alright? I will just click into the individual report over here. I can jump into explore, and that will actually bring me into the individual report for my KYC conversion. So that's going to be my install, create, verify, and KYC complete kind of funnel. I've also done that breakdown by the experiment name that I've been running. So I can see over here, alright, for the different steps where the drop offs are looking like. For variant b as well as variant d, if I just remove these two, I know that these two different variants had very bad performance. Alright? Where are the steps that we are actually seeing a lot of drop offs in? Alright? So I can see over here, maybe between my if I look at the intermediate conversion rates over here, right, I can see that actually for Vivint d is performing very badly the moment we go into all of the verify kind of moments. Alright? So I can see if I click into any of these particular drop off points, what users have been doing during these drop off points. Alright? There are going to be a few options for us to also explore this. Because of our integrator analytics. We can go into, like, a user flow to see what are the different steps that the user has been doing. But in this day and age, you want something that's a little bit more summarized and more aggregated. So that's where we often see customers use this with the session replay. Alright? So what this is going to do I just like to a quick refresh of this page. Alright? It's going to launch into, like, a range of different session replays of where the users have been dropping off. Alright? And we can also use AI over here to summarize what the user has actually been doing. So we can take a look at all of these different, like, sessions that the user has been triggering to figure out, okay, where are they happen where do they tend to have, like, take clicks over here, for example. Alright? Or where do they tend to actually get frustrated and drop off? So this is going to tell us, okay, perhaps something on the homepage, for example, made the KYC process less accessible, and that's going to be our action item for brand new users who are coming in. We also want to be able to improve the KYC step so that we can reiterate on this improvement and kind of overall, like, get our trade conversion rates to go up without sacrificing our KYC performance as well. K? So now that we are done with the demo, we'll go back and review, like, the different types of reports that we've looked at so that, you know, we can comprehensively understand when to use each of these different reports. Okay? So we'll go back to the slide deck over here. Okay. Alright. So when should we actually use the different reports? We have established, right, for the feature flagging part, we want to use that in the planning and orchestration phase. The experiments part, we use that when we collect the data and want to make a call on how successful the experiment was. Alright? If you are also well versed with Mixpanel, you will also may you may also start to wonder. Right? Okay. Since my experiment data is showing up everywhere, is showing up in my metric trees, is showing up in the experiments report, is also showing up in the funnels report, is there, like, just one report that I should be looking at only? Alright? So this is how we are kind of, like, broadly categorizing when to use which particular report. So experiments report is for you to see how that experiment, how the different variants are performing. Your metric tree is going to be for you to understand how your overall experiments are influencing the North Star as well as your more exact level types of metrics. Original reports that we've taken a look at, such as our insights, our funnels, our session replays, are more for you to do that free form exploration between the different variants. Okay? So today's focus was more on the feature flags, the experiment analysis, and how we can deep dive from there into the rest of mixed panel analysis to kind of determine what the next thing that we should do, how can we build more quickly as well. Okay? So as we wrap up today's session, right, I want to leave you with a few core takeaways. Innovation today is moving very, very quickly. We basically need to have that continuous loop of learning, and that will happen a lot more quickly when we can see how our feature flag and experiment planning can be tied very closely with the existing analytics data that we have together. Alright? Once we have this unified foundation, this will also help us get to real outcomes. That means cleaner and more reliable data for us to run our statistical significance analysis on, analyze our list more confidently, and then measure the impact between behind every feature that we are shipping. Alright? You can see in the demo that Mixpanel is giving you the end to end workflow. So your insight hypothesis testing validation and measuring your impact are all within the same UI. You do not actually need to export any kind of data out. You do not have to, you know, build any types of, like, APIs or new types of KPIs, and there won't be any data drift because everything is condensed and put inside a single platform as well. Alright? So whether it is that that is increasing users who are doing primary banking behavior by 14% or looking at teams who are building different types of experiment cultures within Mixpanel as a tool. The message is the same over here that when your analytics can drive your experiment, insights can turn to impact very quickly, and you can build a lot more faster. Alright? So you can sign up for the mixed mixed panel experimentation feature walkthrough. You know, there will be, like, a QR code that we'll be sending to you so that you can get a more personalized walkthrough as well. Alright? Before we also open up for the q and a, we also wanna keep things interactive for a minute. I've mentioned just now that before we launch the demo, there will be like a little trivia round based on what we've covered today. So it's a quick way to kind of test, like, you know, a recap of what we've done. Alright? And we have some small prizes for the top responses. Okay. So yeah. So there will be three questions over here. This one is a true true false. Does Mixpanel actually have an experiment platform that allows you to deploy your experiment and analyze your experiment results? Okay. So you should have around, like, twenty seconds to run this. Sorry. Yep. Okay. Alright. We can bring the second question now. Okay. Can we actually run experiments in real time? This is also a true false. We will have around twenty seconds for this question as well. Just a quick fire round. Okay. And the third question, is Mixpanel able to help you connect your experiment results directly to your business metrics and to your user behavior. True false again. Okay. Alright. So thank you so much for, you know, answering the q and a. We will also have, like, a final poll if you are interested in a personalized demo or if you want to know more about, you know, the features that we have presented today as well. Okay? Alright. We will be moving to the q and a now. Maybe what we can do is begin to, like, review the questions and then pull them out. Okay. Maybe let's pick a few. Okay. Okay. So I see a question from Noveta. Yep. We are talking about continuous improvement, and we want to see whether we can currently support multi multi armed banded experiments for us to improve the machine learning models while also simultaneously maximizing performance? Okay. So this is a good question. Multi armed banded experiments are going to be a very useful tool for complex teams that are running sophisticated experiments. This is something that our product team has been looking at over the last few weeks. We will be able to, like, share with you and through the account team with you on, you know, what the timeline is like and what we can expect for launching, you know, multi armed band aid experiments within this panel. Yeah. Okay. Alright. So we can go let me see if I can, like, flag another question out. Okay. Alright. So there's another question on what level of plan do you need to be on to see the feature flex and experiments. I believe right now this is available for enterprise plan, you know, on mix panel. So do write in to us. You should be able to get, like, one free feature flag as well as one free experiment for you to test on. Alright? Okay. So how should we con coordinate our control groups between cohorts and potential segments from, let's say, LaunchDarkly? Okay. So that's a good question. Essentially, what you have seen today from mixed panel, because we have launched our own experiments as well as feature flat analysis, What we are able if an experiment related analysis, all of these sort of, like, are doing the same part of the experimentation as LaunchDarkly. Alright? It's all within Mixpanel. But if you already have launched directly within your current setup, we do have, like, a native integration with Audi. What that's going to look like is, let's say, you figure out your user cohorts in Mixpanel. Right? You have your user cohort builder that you want to target and put inside your rollout group for. You can send this out to LaunchDartly to launch your experiments. Once that happens, you will be able to, you know, LaunchDartly will automatically forward the experiments reports into Mixpanel. So what that will look like is the feature flag piece that we've shown you in the demo wouldn't matter. It it would be more within LaunchDarkly. But the experiment part, you can still send the experiment event into Mixpanel and measure how that affects the overall metrics within your experiment reports. Okay. Okay. So in cohort selection, we've noticed that the number of targeted users is under reported, and we're using the user property filter. So is it good for us to use one time events by selecting all events and use the same property filter there? I think for this, this is like a fairly, like, detailed question. I will want to actually go in and understand for your feature flex setup, are you targeting based on device ID or user ID? Because if we are seeing that the number of targeted users is underreported when using the user property filter, it could also be due to the user property only being propagated on users who actually have profiles. Alright? So depends on how we have set up the experiment. Alright? But if that is a problem, I would believe each we should actually go back to the implementation piece to kind of make sure that to kind of address why is there underreporting on these Using one time events, selecting all events instead, that will work as well, but it's more of an implementation piece that I would prefer to go back to as they are not exactly meant to do the exact same thing. Yeah. So we should go back to our targeting criteria. Yep. Okay. So we have another question on which significance model does this analysis use. Does it use the Bayesian or frequencies? Okay. So mixed panel actually has a few different option. Frequencies is something that we that we have released as well. Alright? So the config in terms of how we want to measure this p value and also our calculation. Right? We are going to have we do have multiple options that are already available within your feature flag and experiment setup. So if I just drop this link inside the chat over here. Alright. You will see that we have support for frequencies, sequential, as well as Bayesian. Alright? This is something that we can actually go in and explore. So if I yeah. We can just refer to that inside the docs. I may try to stop share and go to the UI to show you what that looks like as well. Okay. Okay. So we can see over here under our experiment analysis. Right? Under the advanced advanced analysis, how we want to calculate. Alright? We do have frequencies as well as sequential out of the box for you. Changing any of these settings is going to reflect within all of the metrics automatically. Alright? So beyond this, because changing the experiment type is often, like, one of the ways that we try to reduce, you know, variance in our analysis, there will also be other variance reduction options that are available for you, such as having cupe cupec variance reduction that basically also looks at past expert past data before we launch the experiment. Alright? Having one for only correction, which helps us control variance when we have multiple different types of treatment groups that we're rolling out to the users, as well as visualization, which can help us manage better when we have users who are perhaps transacting at, like we like very large outlier numbers as well and to help us bring, you know, the variance and the range of our analysis results to become more normalized. K? So there are also some advanced analysis options over here that we will be releasing very soon to you. Yeah. Okay. So we will have, like, one final question. Let me take a look at the different questions as well. Okay. So I see, like, there's a one there's one question on, like, you know, from Gita. I think it's a new user to Mixpanel, whether there are any documentation that we can refer to on how to set up the metrics for the experiments. Yes. You can refer to our mixed panel experiments help doc for you to kind of create the different metrics. That is going to be something that leans heavily on our existing UI for, like, the insights report, report, the funnels report for you to determine whether my experiment metrics should be something like average order value or, like, KYC conversion, but that's something that we will be able to do very quickly and easily within the Mixpanel UI. So I'll drop a link for you on setting metrics. Yep. Okay. Alright. Thank you so much for, you know, attending today's webinar and also coming with all of your questions. For the teams who are existing users of Mix panel, we will reach out to you with our accounting to address the different questions that you've had today. Otherwise, I hope this has been a helpful and informative webinar to share with you how Mixpanel has newly launched our experiments and feature flagging capabilities. This has also been used by trusted fintechs across the world in different industries as well. And, hopefully, this will, you know, be a very helpful session for you. Do feel free to also, like, drop us a message on LinkedIn if you have any questions as well, and let us know if you would like us to reach out to you for more personalized discussions as well as consultations. Alright. Thank you.