AI Playground: Katalon & BrowserStack

[00:00:00] Patty Parobek: Hello, and welcome to another episode of Leader Generation’s AI Playground. I’m Patty Parobek, VP of AI Transformation here at Mod Op. Today we’re shining a light on one of the most practical and often overlooked ways, AI-powered automation. Is delivering measurable results, and that’s for marketers, for content teams and businesses overall quality assurance and testing.

[00:00:29] Patty Parobek: Why does this matter for marketing and business leaders? In a world where digital experiences are central to the customer journey? Ensuring your websites, your campaigns, and your digital products work seamlessly across browsers and devices is more important now than ever before. The reality for most teams is that manual testing eats up a ton of time.

[00:00:56] Patty Parobek: It creates a lot of bottlenecks, and it slows down how quickly you’re gonna be able to launch new web, new applications, or even innovate. So with AI adoption accelerating across the business landscape. We know every leader is searching for opportunities where AI can drive efficiency and impact. In this episode, we’re diving into a pilot project where we put two different AI applications head to head, testing their ability to scale, increase test coverage and free up talent to focus on higher value work.

[00:01:34] Patty Parobek: We’re featuring two AI applications today, Catalan and BrowserStack, both are next generation test automation platforms. One with advanced technical capabilities. The other built to make automation more accessible to non-developers, we put them to the test with our own QA team.

[00:01:57] Patty Parobek: So you’ll be hearing from Carlos and Giselle today. And they’ll be taking you through our AI Playground framework where they were able to design the pilot, measure efficiency, and start even to manage change within their team as they start to scale and roll out more. It’s worth noting both of these platforms were evaluated through our very rigorous risk and compliance lens, and they both have strong security and privacy features.

[00:02:29] Patty Parobek: So we’re talking SOC two certification, GDPR alignment, clear data ownership policies where the data that you input or even the output data does not train their AI models and all that the testing was performed under Mod Op’s responsible use guidelines. So if you’re a marketer, if you’re a business leader, you’re just beginning your AI journey, you’re searching for actionable pilots and proven frameworks, this episode is packed with those insights.

[00:03:00] Patty Parobek: So you’ll be able to hopefully uncover that hidden value and even another place to lean in in your organization. So let’s jump into the conversation with Carlos and Giselle.

[00:03:13] Giselle Castro: My name is Giselle Castro. I’m the head of the Department of Quality Assurance and Authoring, and I’ve been at Mod Op for six years now.

[00:03:24] Carlos Obaldia: Hello, my name is Carlos Obaldia. I’m a Senior Author and QA Tester like since 2022 . But I worked here , since 2017, so I’ve been here for a long time now.

[00:03:39] Patty Parobek: Awesome. Well, thank you both again for talking to us. About the tools and the, the AI applications that you tested.

[00:03:47] Patty Parobek: Uh, we know that they’re Catalan and BrowserStack, but can you give us a sense of what those applications really are and what they’re meant to do or what even drew you to them in the first place?

[00:04:01] Carlos Obaldia: Yeah, so. First of all, let’s start with Catalan. It’s a test automation platform that combines low-code and full-code capabilities.

[00:04:12] Carlos Obaldia: It’s like ideal for teams that want flexibility to test coverage across web, mobile, APIs. Desktop applications. It’s many features. Uh, it records, um, playbacks, uh, tests with, uh, scripting. Test data assertions and reusable components. Um, CICD integrations like Jenkins or GitHub and advanced reporting and analytics.

[00:04:48] Carlos Obaldia: On the other side on BrowserStack, it’s the low-code test automation platform, what we were testing, which is focused on web testing. Uh, and it’s designed to help manual testers, um, non-technical users to, quickly build and run, cross-browser tests. Its main features, kind of the same as Catalan. , But it’s more, more visual. It has visual test builders, , more friendly user because it’s like drag and drop, um, records and replace user flows.

[00:05:25] Carlos Obaldia: , Built-in real browser and devices. With the BrowserStack main tool, , and it has AI-powered, , test healing and smart location, which is one of the main parts we wanted to test. , It also integrates , with CI tools and BrowserStack test management, which is another BrowserStack tool.

[00:05:47] Patty Parobek: So when you started looking for these tools and landed on Catalan and BrowserStack, were there specific challenges in your process that you were trying to solve for? Were there features you were already looking to use?

[00:06:02] Carlos Obaldia: Yeah, I would say, the first thing that I think most manual QA teams may want to fix is too much manual testing or load test coverage.

[00:06:16] Carlos Obaldia: So, yeah, the manual regression testing is, is time consuming and it’s prone to human error. . Catalan, has an automation of, web and desktop apps. So it helps, with , uh, creating, tests, um, BrowserStack, , lets non-technical users automate tests and flows quickly. Which is, uh, great for covering like high-volume UA paths without knowing how to code.

[00:06:47] Carlos Obaldia: The other one I would say is, change resistance. I haven’t talked to my teammates about the, the experience yet, but I know manual testers may feel threatened by automation or worry about their roles becoming obsolete.

[00:07:05] Carlos Obaldia: The other one was, the learning curve, skill gaps. And the initial time investment, Automation , takes time upfront. Yeah, because, uh, you have to delay manual tests or release cycles, so you can create automation tests.

[00:07:26] Giselle Castro: And this also, uh, all starts with the, actually when we were doing live QAs and regressions, those takes like many hours to test. So we were thinking on how we can be more proactive with our time. Mm-hmm. So Carlos came along with, with Catalan, but it’s more for like, for like tech savvy person. Right? So then BrowserStack came with, with this low code automation, this one is good.

[00:08:01] Giselle Castro: So whatever you are doing on a, on a webpage testing, they learn, uh, with the AI tool of course. So they replicate that on several browsers at the same time.

[00:08:14] Patty Parobek: Oh, wow.

[00:08:15] Giselle Castro: The difference with the manual testing is that you have to do it. At a time. Right. And our, uh, browser list, our default browsers list is six.

[00:08:27] Giselle Castro: So we have to test everything six times.

[00:08:31] Patty Parobek: Mm-hmm.

[00:08:31] Giselle Castro: Now with BrowserStack, for example, you only have to do it once and it will replicate the other five on different devices. So that’s a good thing. It also does with Catalan.

[00:08:44] Giselle Castro: Catalan is more tech and more matrix based. And, BrowserStack, it’s more like for who, whomever wanna use it. You don’t have to be a tech savvy, you just have to read. ’cause they give you all the, the details and instructions.

[00:09:00] Patty Parobek: Oh man, that’s awesome. So going in, it sounds like from the main challenges you were trying to solve for your faster, less cumbersome manual regression into something more automated or relieving bottlenecks, getting time back, seems like efficiency was your main measurement.

[00:09:20] Giselle Castro: Mm-hmm.

[00:09:20] Patty Parobek: So tell me a little bit about how you guys were measuring efficiency gains or what kind of results did you see?

[00:09:32] Carlos Obaldia: Um, okay. The first thing, um, it was the time to first automated test, like how long it takes, uh, for a team member to create and run usable automated test. That will be the first one. Um, second one, cross browsing and device coverage.

[00:09:55] Carlos Obaldia: The number of browsers, OS devices, combinations the team can consistently use. Mm-hmm. For test. Um, the other one, the tester productivity number of test cases, a QA, team member can create and maintain. . Um. Log detection efficiency, a number of critical bug can be caught by wow, automated tests before reaching, um, staging or product depending on how we are working on the page, on the site we’re working on.

[00:10:35] Patty Parobek: And what kind of results did you see with those things? Like what was kind of the biggest. Maybe a-ha moment that you saw ?

[00:10:43] Carlos Obaldia: Well, it varied depending on the tool. Oh, sorry. Um, for example, it was for the cross browser testing, Catalan required a third-party integration whereas, uh, the BroswerStack had it built in. . Also the measures, actual output into usability. I think it was longer and a bit more complex with Catalan.

[00:11:14] Carlos Obaldia: It just has a bigger, learning curve, . And book, , detection efficiency was , the only one that I could say, was a little bit stronger for Catalan., In the accuracy and coverage value compared to BrowserStack. Those were like the three main things I I found out while testing.

[00:11:38] Giselle Castro: And if you wanna know numbers, for example, uh, it, it varies from tester to tester. ‘ As I mentioned for example, you have to do at least one and it’ll emulate the rest.

[00:11:53] Giselle Castro: So it will depend on the time and the complex of the tickets t hat we are testing. So for example, for Carlos, it took like 50% off of his time. So that’s, that’s a lot. That’s a lot. Uh, for one ticket, like one small ticket. But for like regressions that mm-hmm. Was, uh, what we were looking for because regressions, we need to look, everything, everything that was built, everything was that we, the dev touch and retested and touch again and came back. So we are testing that the thing as a whole, and in this cases that we’re testing, web apps or websites is not that easy. ’cause you know, you are, we are testing like a user.

[00:12:44] Patty Parobek: Okay. So it sounds like we’re still kind of getting our arms around how best to quantify all of the efficiency. Or kind of figuring out how to use the tool even more impactfully to be able to measure that more accurately. Mm-hmm. But in small tickets where it’s a little less mm-hmm. Complex, we’re seeing efficiency right away.

[00:13:09] Giselle Castro: We have seen the, the efficiencies there mostly.

[00:13:13] Patty Parobek: Right.

[00:13:14] Giselle Castro: Right. But. The, but the prices varies. I mean, BrowserStack is more pricey than Catalan. Mm-hmm. And you can tell because of the way they build the, the, all the app, it’s, it’s different. And also they have like several features, not only low-code, they have accessibility, they have test cases built, they have the live web testing .

[00:13:36] Giselle Castro: So there are many features inside the same BrowserStack tool. We’re testing actually right now, everything that they have.

[00:13:44] Patty Parobek: Lemme ask about change resistance ’cause this is obviously near and dear to my heart, and I work with adoption all day long. When you brought the technology into your teams, did you have a small group using it or are you using it and plan on rolling it out to a larger group?

[00:14:04] Carlos Obaldia: Just just me. Okay. And later, let the other team members that have time for it to experiment with it.

[00:14:13] Giselle Castro: We are in that phase right now. Like the first, uh, guinea pig was Carlos. He was the one testing the tool. So he said that browser stack was good for everybody and now we are on the face that everybody’s testing all the testers in our team, they all are using the tool. So we need now on our next meeting, hear their feedback.

[00:14:37] Patty Parobek: But I mean, it sounds like there are folks using it that mm-hmm. I mean, when you introduced the tool to them, did they seem skeptical or did they seem resistant or?

[00:14:49] Giselle Castro: Not at all. Not at all.

[00:14:49] Patty Parobek: Everybody pretty much embraced it.

[00:14:51] Giselle Castro: Carlos, uh, gave us like a, a brief of what he was testing with Catalan and everybody was, uh, surprised and, and glad of the, of the tool. ’cause we spent like for in a regressions six hours testing the same page over and over so we were all excited about it.

[00:15:10] Patty Parobek: So for other teams starting a similar type of evaluation, especially within your role. What advice would you give them about running this type of pilot?

[00:15:23] Carlos Obaldia: Yeah, the first one I would say to start with a real pain point , or repetitive process that AI could replace, accelerate, or improve. The other one keep , the scope, small and contained., Like use the smallest version of, of a real deliverable, campaign brief or something like that, and focus on, augmentation. Not a replacement. And document learnings on friction points.

[00:16:05] Patty Parobek: If you, both of you were thinking about what you’ve experienced with each tool.

[00:16:10] Patty Parobek: What you’ve heard about each tool is as, as your team’s been testing it, Catalan BrowserStack, and you could rate these tools separately. From one to five, one being for your role in QA and authoring, it’s not there yet. All the way to five as highly recommended for this type of role and this type of engagement, what would you give each tool?

[00:16:35] Carlos Obaldia: I would give a five to BrowserStack and four to Catalan.

[00:16:39] Patty Parobek: That’s great.

[00:16:41] Giselle Castro: Yeah, same. Same. I, I will give, uh, a five and, and a four. So I think I, I mean this still not replacing the humans.

[00:16:54] Giselle Castro: Mm-hmm. So we are not there yet. ’cause we did do need like to someone, some manual actual testing so it can work, uh, and run everything. It’s not that you just click button and it will do it itself. No, you need to like, engage with the tool and do some manual testing so it can go and like do the shortcuts that you can’t while testing manually.

[00:17:24] Patty Parobek: Yeah. And I think that’s perfect and well said. And it human needs to be there. Yeah. To be able to show the AI what is the expert experience that you need to do, duplicate and emulate and take into additional browsers or additional ways. And, uh. The human in the driver’s seat is going to be the best way to make sure and validate that the tech is running the way that we intended to.

[00:17:50] Patty Parobek: So really well said and a great close, and thank you so much, both of you for your time today. It’s been so interesting learning about both of the technologies and I’m really excited to hear more as you guys continue to test.

[00:18:03] Giselle Castro: Yeah, thanks to you Patty.

[00:18:06] Patty Parobek: That concludes this episode of the AI Playground. I hope you gain some practical perspective on how automation tools like Catalan and BrowserStack can boost your team’s efficiency and elevate quality no matter where you are in your AI journey. If you are considering launching your own AI pilot, start with a real business problem.

[00:18:30] Patty Parobek: Keep it focused. Measure what truly matters for your team, and certainly define those things upfront. Remember, the road to effective AI is all about experimenting, learning, and building on what works.

[00:18:48] Patty Parobek: If you have suggestions for other AI apps or automation tools, we should review. Or if you’d like to share your own experiences, please email us at [email protected]. That’s podcast at modop.com. And of course, visit mod op.com/podcast to listen to more AI Playground/Leader Generation episodes. Thanks so much for listening, and until next time, keep testing, keep learning, and keep leading in this AI-powered age.