Testing structures that are not tailored to the specific needs of an account can lead to inefficiencies by wasting valuable time and resources. To create an effective testing framework, it's essential to consider the unique characteristics of the account to create a customized testing structure.

Landon Shaw, the co-founder of SweatPants Agency, lays out his roadmap for developing effective Facebook ad testing frameworks and how AI can streamline the process.

Key points covered

  • Getting the most statistical significance for individual variables
  • Determining testing structure based on the spending preference
  • Leveraging AI for efficiency and added value

Meet SweatPants Agency

SweatPants Agency has established itself as the only agency behind scaling multiple INC #1 companies, including Hunt a Killer and SnapNurse. At SweatPants, Nathan and his team believe in building long-term relationships and have with these hyper-growing brands from day zero.

Testing structure for multiple ad accounts

Single ad accounts might benefit from a standardized approach, but there's no one-size-fits-all solution for testing multiple ad account structures. Multiple ad accounts require a more nuanced strategy that considers the unique variables at play.

  • How Facebook allocates budget to ad sets is a crucial variable, and it can vary depending on whether a CBO (Campaign budget optimization) or ABO (Automatic budget optimization) strategy is used.
  • It's important to consider what happens when one ad gets all the budget when testing multiple ads and what happens when all ads get equal spending.
  • Spend preference also depends on budget and risk tolerance.
  • Individual variables to consider include video or image, body copy, headline, description, CTA, link, attribution window, and budget type.
"Look at testing frameworks and optimization inside of the ad account, in terms of structure, to get the most statistical significance for each variable."

Setting up ad tests using performance data

  • Define what to test. Focus on individual variables to define what needs to be tested. To do this, follow these steps:
    ◦ Pull pivot tables for each variable.
    ◦ Review the aggregated data of how the variables have performed in different environments.
    ◦ Find a control based on the best-performing variables.
    ◦ The control is determined by the most amount of sales or the lowest cost.
  • Set up the tests. With a better understanding of the controlled variable, set up the tests by providing the creative team with the best-performing copy and creatives to test.
  • Determine the testing approach. There are two ways to approach testing:
    ◦ Improve on what's already working.
    ◦ Come up with a new idea.
  • Count the cost. Testing a completely new idea will require testing in less controlled environments, equating to more spending.
  • Analyze the results. Monitor the results of the ad tests and use data to optimize the ad campaigns further.

Structuring frameworks based on spending preference

Before launching any ad test on Facebook, it's crucial to understand your spending preference based on the platform you’re using. For Facebook ads, two different structures to consider are the Champion Challenge and Discrete Testing.

Champion Challenge

  1. Load up with semi-proven and iterative tests. Start with a control creative that has shown some success and create variations by changing different variables such as copy, images, or targeting.
  2. Duplicate for all desired variables. Once the control creative and variations are set, duplicate them for all variables that need testing.
  3. Consider spend preference based on data. Facebook's algorithm tends to favor creative, so keep an eye on the data. We want the most amount of statistical significance for the least cost.
  4. Allocate a higher budget. Let each creative run with a budget of at least 2x CAC. This will help gather enough data to identify the best-performing creative.
  5. Find a new champion. Once there is enough data, identify the creative with the best performance and allocate more budget. Kill the underperforming ones and start the process again with new variations. The point is to increase the spend ceiling by finding a new champion.
"Use this where there's less of a strict preference to spend behind individual ads so that multiple ads can test simultaneously. We're looking for the king of the hill."

Discrete testing

The discrete testing approach is similar to the Champion Challenge, which involves testing individual variables to identify the best-performing creative. However, it's not as efficient as the Champion Challenge and is better suited for when statistical significance needs to be achieved quickly for each variable.

  • Individual ad-to-ad sets. Create unique ad sets for each variation of the control creative being tested. Have a separate ad set for each variable.
  • Use the same type of control. Like in the Champion Challenge, start with a control creative that has shown some success and create variations by changing a single variable.
  • Naming conventions. Name each set after the variable tested to keep track of all the individual ad sets. This will make it easier to analyze the data later on.
  • Ad set level and budget. Unlike the Champion Challenge, the discrete testing approach requires looking more closely at the individual ad sets and being stricter on the budget. Since each variable is tested separately, you'll need to allocate more funding to each ad set to achieve statistical significance quickly.

The impact of AI on efficiency and quality

I start paying attention to something new when smart people around me are spending their free time. AI has had hype cycles before, but none before ChatGPT have been able to excite my circle of smart friends. Generative AI has got me intrigued.

  • ChatGPT is a valuable AI resource that can be used to elevate ad copy.
  • The accuracy and relevance of a ChatGPT result rely on how well a user frames the question/prompt.
  • ChatGPT's value lies in its ability to quickly summarize and distill information into something that can be easily used.
  • While helpful in generating ad copy, it still requires human judgment to assess whether the output is good.
  • Fact-checking the generated content is crucial as there are known quality issues with AI models producing fraudulent information.

There are many fears regarding the use of AI in various fields. There are two ways of thinking about this new tech:

  1. AI as a crutch. This thought process believes that AI will replace humans or that excessive dependence on AI will result in a decline in critical thinking skills. The counter-argument is that mediocre executors will increase their output of mediocre quality, but enthusiasts will maintain (or even raise) the bar of quality.
  2. AI as a tool. This view is that AI is another tool for education. It helps people stay up-to-date with the latest trends, making them more competitive and efficient. Learning to leverage AI tools will bolster a person’s worth to their employer and marketability.
"AI itself will not replace people directly, but it will make people more efficient."