FAQ and Troubleshooting

Common issues, frequently asked questions, and troubleshooting guidance for the CreativeDynamics library.

FAQ: Could this be used to understand the “why” behind channel performance?

Not directly. As implemented, CreativeDynamics is not a causal inference system.

It is best framed as diagnostic + decision support, not causal explanation.

What it can do well

  1. Detect statistically and structurally meaningful regime shifts (change points) in irregular KPI paths.

  2. Summarise behaviour within each detected segment (stable / improving / declining).

  3. Quantify a decision-relevant impact metric using benchmarking (for example actual_overspend_gbp versus a benchmark period).

  4. Support monitoring and operational workflows: “Something changed here; performance is now in a worse regime; consider intervention.”

What it cannot do (by itself)

  1. Identify the “why” (attribution): competitor action, auction dynamics, creative fatigue, targeting changes, reporting artefacts, and so on.

  2. Make claims like “X caused Y” without an additional identification strategy (for example a controlled experiment, causal design, or instrument).

Is “predictive” the right word?

“Predictive” is often used loosely to mean “actionable signal for decisions”.

If your audience includes data scientists, more precise terms are usually better:

  1. Change-point detection / regime detection

  2. Retrospective inference (on historical data)

  3. Monitoring / early warning (if run routinely)

  4. Counterfactual-style benchmarking for impact metrics (“what if we stayed at benchmark?”)

If you run this routinely and treat detected regime shifts as a leading indicator that performance is likely to remain degraded without intervention, then it can be described as predictive in an early-warning sense.

It is not forecasting in the usual time-series sense (there is no explicit model producing future values).

Important nuance about benchmarking impact

Impact metrics such as actual_overspend_gbp and engagement_gap_clicks are based on a practical, quasi-counterfactual assumption:

“Benchmark performance is a plausible reference you could achieve again; impact is the gap while you are in a worse regime.”

This is not causal identification, but it can still be operationally useful if the benchmark period is representative and achievable.

Common Issue 1: Understanding Trend Line (slope: nan) in analysis output

If visualisation legend shows “Trend Line (slope: nan)” in creative analysis output, indicates no clear linear trend could be computed for that segment.

Statistical Interpretation:

  1. Insufficient data points: Not enough valid data points to establish trend

  2. Too many missing values: Many data points have missing values or zeros

  3. Perfectly horizontal data: All data points have exactly same value (no change)

  4. Computational singularity: Division by zero or other mathematical issues in calculation

Business Interpretation:

  1. Creative’s performance is erratic or random with no discernible pattern

  2. Too many days with zero impressions, clicks, or spend

  3. Creative might have irregular serving patterns (turned on/off repeatedly)

  4. Data could contain tracking issues or anomalies

Important Note About “Improving” Creatives with NaN Trends: May occasionally see creative classified as “improving” despite having NaN trend line. This occurs because:

  1. Signature-based analysis: Classification comes from signature pattern analysis, independent of trend line calculation

  2. Pattern vs. trend: Signature patterns can capture complex, non-linear improvements that linear trend lines cannot

  3. Data characteristics: Performance might improve in specific time windows even if overall trend is undefined

  4. Metric stability: Consistent improvement in aspects like reduced variance can be detected without clear trend

Solution: For creatives with “nan” slopes:

  1. Consider them neither definitively improving nor declining based solely on linear trend

  2. Examine signature patterns and detailed metrics for deeper insights

  3. Treat “improving” classifications with NaN trends as cautiously positive signals

  4. Consider manual review to understand actual performance patterns

Common Issue 2: Interpreting different performance/trend categories

When analysing time-series data, segments between detected change points classified based on trends:

  1. Improving Performance: Metric shows favourable trend (e.g., decreasing CPC, increasing CTR) with stable signature patterns

  2. Stable Performance: Metric does not show significant trend, signature patterns also stable

  3. Declining Performance: Metric shows unfavourable trend (e.g., increasing CPC, decreasing CTR), potentially accompanied by significant signature pattern changes

Solution: Exact interpretation depends on metric being analysed and specific analysis goals. Always consider metric context (e.g., for costs, lower is better; for rates, higher is often better).

Common Issue 3: Understanding distance calculations in output

Distance calculations represent mathematical distance between signature patterns calculated for consecutive time windows in creative’s timeline. Larger distances indicate more significant changes in performance pattern, may signal creative fatigue or other performance shifts.

Solution: Distance based on mathematical signatures derived from rough path theory, providing more nuanced insights than simple trend analysis. High distances flag segments worthy of investigation for pattern changes.

Common Issue 4: Installation issues with dependencies

Library now uses roughpy which provides pre-compiled wheels, eliminating most installation issues. If problems occur, ensure you have latest pip version.

Solution: Refer to “System Prerequisites” section in Installation guide for detailed instructions specific to your operating system. Covers requirements for Linux, macOS, and Windows.

Common Issue 5: Determining supported Python versions

Library tested with Python 3.10 and newer.

Solution: For specific compatibility details and recommendations, see “Python Prerequisites” section in Installation guide.

Common Issue 6: Whether to use a Python virtual environment

Yes, highly recommended.

Solution: Install and use CreativeDynamics within Python virtual environment to avoid conflicts with other projects or system-wide Python packages. See “Create and Activate a Virtual Environment” steps in Installation guide for guidance.

Common Issue 7: Expected format for input data

Library typically expects time-series data in CSV file or DuckDB database table. Data should include columns for creative identifier, date/timestamp, and metrics you wish to analyse (e.g., impressions, clicks, spend). All column names standardised to lowercase immediately after CSV ingestion to simplify data handling throughout pipeline.

Solution: Refer to examples in User Guide: CreativeDynamics Library guide and data/schema.md file (if available) for more details on data structure. Python API examples in usage guide show typical parameter names like item_col and date_col.

Common Issue 8: Differences between CLI, Python API, and FastAPI usage

  1. CLI: Recommended approach for most analyses, offering standardised workflows with YAML configuration

  2. Python API: Suitable for integration into custom Python scripts, enabling programmatic analysis and deeper customisation of parameters

  3. FastAPI: Provides web service interface for analysis, suitable for integration with other applications or building interactive dashboards

Solution: Consult User Guide: CreativeDynamics Library guide for detailed instructions on each interface, including examples for CLI, Python API, and FastAPI.

Common Issue 9: Recommended minimum days of data for analysis

The default minimum is often around 14 days, but for more reliable analysis:

  1. 14-30 days: Basic analysis (minimum viable).

  2. 30-60 days: Good reliability.

  3. 60+ days: High reliability, capable of detecting subtle pattern changes.

Solution: Setting min_days too low may result in premature pattern detection. Setting it too high might exclude too many creatives from analysis. Adjust based on your data volume and analysis goals.

Common Issue 10: Signature computation method used

The “CreativeDynamics” library uses the roughpy library for robust and efficient signature calculation.

Solution: This is the standard method used by the library for its core signature-based analyses.

Common Issue 11: Interpreting the threshold parameter for change point detection

The threshold parameter (default: 1.0 or 1.5 depending on the interface/function) determines sensitivity to pattern changes detected via signature distance:

  1. Lower values (e.g., 0.5-0.9): More sensitive, may detect subtle changes earlier but with potentially more false positives.

  2. Default (1.0 or 1.5): Balanced approach for most use cases.

  3. Higher values (e.g., >1.5-2.0): Less sensitive, only detects major pattern changes, fewer false positives.

Solution: Adjust the threshold based on how sensitive you need the detection to be for your specific use case. Experimentation might be needed.

Common Issue 12: ModuleNotFoundError when trying to import creativedynamics

This typically indicates that the CreativeDynamics library is not installed in your current Python environment, or you are not running your script from within the correct virtual environment.

Solution:

  1. Ensure you have activated the virtual environment where you installed CreativeDynamics.

  2. Verify that the library was installed correctly. From the project root directory, you can run pip install . (or pip install -e . for an editable/developer installation).

  3. Double-check your import statement: from creativedynamics import analyzer (or other relevant modules).

  4. Refer to the Installation guide for detailed setup instructions.

Common Issue 13: Seeing “nan” distances in output logs

“nan” (Not a Number) distances in computation logs can occur when:

  1. There are divisions by zero (e.g., zero clicks when calculating CPC).

  2. The analysis window contains days with missing or invalid data.

  3. The input data has extreme outliers or inconsistencies.

  4. CTR values are not properly normalised (see Common Issue 17).

Solution: These “nan” values are generally handled by the system (e.g., by not flagging a change point if distances are NaN). However, if they represent a significant portion of the data, it may affect overall analysis reliability and indicates potential data quality issues to investigate.

Common Issue 14: No creatives meeting the minimum days requirement

If no creatives in your dataset meet the min_days requirement:

Solution:

  1. Reduce the min_days parameter to a lower value (if appropriate for your analysis goals).

  2. Extend the date range in your input data to gather more history.

  3. Verify that your creative IDs are consistently tracked (inconsistent IDs may fragment the time series for what should be a single item).

Common Issue 15: Handling very large datasets

For very large datasets:

Solution:

  1. Filter the input dataset beforehand to analyze only the most relevant items or time periods.

  2. Consider summarising or aggregating data if appropriate for your analysis goals, though this might lose some granularity that signature analysis can capture.

  3. Increase the min_days (or equivalent data length filter) parameter to focus on items with substantial data history.

  4. Ensure your system has sufficient memory, as loading and processing large time-series can be memory-intensive. Consider processing in batches if possible.

  5. Use the CLI with YAML configuration for more efficient processing of large datasets.

Common Issue 16: A declining trend showing zero reported impact in an example application

In applications calculating actual_overspend_gbp (financial) or engagement_gap_clicks (operational), a declining performance trend might still result in zero calculated impact.

Solution: This can be due to the specific methodology:

  1. No Suitable Benchmark Found: If the analysis couldn’t find a long enough ‘Stable’ or ‘Improving’ period earlier in the creative’s life to set a reliable benchmark, impact calculation defaults to zero.

  2. Insufficient Data After Benchmark: If there are insufficient data points after the benchmark period, impact may not be calculated.

  3. Decline Never Exceeds Benchmark: The CPC trend might be statistically declining within a segment, but the actual CPC values during that decline might never rise above the benchmark CPC established from a previous, better-performing period. Since actual_overspend_gbp might be calculated as max(0, CPC_t - CPC_benchmark) * Clicks_t, it remains zero if CPC_t is always less than or equal to CPC_benchmark.

Common Issue 17: Understanding “excluded” creatives in reports

Creatives listed as “excluded” in the report summary or logs are filtered out before the main fatigue analysis. This is due to failing data quality checks based on parameters like:

  • Minimum Days (min_days)

  • Maximum Gap (max_gap_days)

  • Minimum Density (min_density)

Solution: The standard reports currently only show the reason for exclusion. The raw data for these creatives exists in your input file. To analyze them, you might need to adjust filter parameters or investigate their data quality. Displaying aggregate metrics for excluded creatives in the report would typically require a feature enhancement.

Common Issue 18: What is a “path signature” in simple terms?

A path signature is a mathematical tool that effectively “summarizes” the essential shape and order of events in a time series (like the performance of an ad over time).

Solution: It’s designed to be more descriptive and robust than simple averages or trends, capturing how the data stream evolves. For a deeper dive, see the explanation on methodology in Core Concepts or related concept documents.

Common Issue 19: Why use path signatures for creative fatigue analysis?

Signatures can detect complex, non-linear patterns and changes in how a creative performs over time, which simpler methods (like linear trend analysis) might miss.

Solution: This allows for a more nuanced understanding of performance shifts, helping to identify genuine fatigue beyond simple declines. Learn more at Core Concepts or related concept documents.

Common Issue 20: How to contribute to CreativeDynamics

We welcome contributions!

Solution: Please see our detailed guide in development/contributing.md (assuming it’s converted or will be) for details on setting up your development environment, coding standards, running tests, building documentation, and the overall contribution workflow. If this file is not yet converted, refer to the existing development/contributing.rst.

Common Issue 21: Customizing the metrics being analyzed

Yes, the library allows analysing metrics beyond the default CPC and CTR, provided they exist in your input data and are specified correctly during analysis setup (for example via CLI -m flag or API parameters).

Solution: Ensure your custom metric is present in the data. When analyzing, ensure the interpretation of “improving” vs. “declining” trends makes sense for your custom metric (e.g., for ‘conversion rate’, higher is better; for ‘cost per acquisition’, lower is better). Update any related configuration or parameters accordingly.

Common Issue 22: CTR values appear to be in percentage format

If your dataset contains CTR values stored as percentages (e.g., 0.37 for 37%) instead of proper fractions (e.g., 0.0037 for 0.37%), this can cause impossibly high CTR benchmarks and unrealistic impact calculations.

Solution:

  1. The library now automatically detects and normalises CTR values that appear to be in percentage format by dividing them by 100.

  2. You can verify this by checking the logs during data loading, which will show CTR normalisation range information.

  3. If you still see unrealistic CTR values (e.g., CTR > 0.1 or 10%), you may need to manually preprocess your data before analysis.

Common Issue 23: Understanding Financial vs Operational Metrics

CreativeDynamics reports two distinct types of metrics that should not be combined:

Financial Metrics (Actual Costs):

  • actual_overspend_gbp: Actual excess spending due to higher CPC than benchmark

  • Total Spend: Actual money spent on the creative

Operational Metrics (Performance Indicators):

  • engagement_gap_clicks: Lost clicks relative to benchmark (primary)

  • CTR Decline %: Performance drop from peak CTR

  • Creative Health Score: Composite performance indicator (0.0-1.0)

Why are these separate? In accounting, actual losses (money already spent) cannot be combined with potential losses (theoretical opportunity costs) in a single number. These are conceptually distinct and must be reported separately.

Solution: Use financial metrics for budget analysis and operational metrics for creative refresh decisions. Do not add them together; they serve different purposes.

Common Issue 24: Understanding correlation risk (do not combine metrics)

When CPC and CTR metrics are negatively correlated (when CTR drops, CPC rises), both metrics may reflect the same underlying performance issue.

Solution:

  1. The library provides correlation context between CPC and CTR to inform interpretation.

  2. Risk levels are categorised as:

    • High Risk: correlation < -0.5 (strong negative correlation)

    • Medium Risk: correlation < -0.2 (moderate negative correlation)

    • Low Risk: weak or positive correlation

  3. Do not compute any combined total. Interpret engagement_gap_clicks (operational) and actual_overspend_gbp (financial) separately regardless of risk level.

  4. The visualisation dashboard includes a correlation risk indicator to aid interpretation.

Common Issue 25: Does bot traffic invalidate Creative Fatigue measurement?

If a substantial portion of digital advertising traffic consists of bots (some estimates suggest 50-80%), does this make Creative Fatigue analysis meaningless?

Short Answer: No. CreativeDynamics measures platform behaviour economics, not pure human psychology. The analysis remains valid for its primary use case: financial optimisation and spend efficiency.

Detailed Explanation:

  1. Platform Filtering Is Imperfect

    • Meta and Google filter known bot traffic before reporting impressions and clicks

    • However, industry experts confirm that some bot traffic still leaks through these filters

    • Your analysis uses the platform’s reported metrics, which include this residual bot contamination

    • The exact bot ratio in your data is unknowable but likely consistent across campaigns

  2. What You’re Actually Measuring

    • You’re not measuring “pure human creative response”

    • You’re measuring what you actually pay for - the platform’s reported performance

    • Whether that’s 70% human + 30% bot, 50/50, or any other ratio doesn’t change the financial reality

    • CreativeDynamics optimises platform performance as reported, which directly drives your costs

  3. The Analysis Measures System Fatigue

    • Temporal decline patterns are real (something in the system is changing)

    • CPC increases when “performance” declines (you pay more regardless of audience composition)

    • Financial impact is real whether audiences are bots, humans, or a mix

    • You’re optimising your position in the platform ecosystem as it exists

  4. Why Relative Performance Still Matters

    • If bots are systematically distributed across campaigns:

      • Creative A declining faster than Creative B remains a meaningful signal

      • Relative performance guides budget allocation decisions

      • Rate of change indicates optimal refresh timing

    • If bots are NOT systematically distributed:

      • That’s a bigger problem than creative fatigue

      • CreativeDynamics would help detect unusual patterns suggesting this

  5. Where Bot Traffic WOULD Invalidate Analysis

    • Understanding human psychology for creative development

    • Informing brand strategy based on “what resonates with people”

    • Making qualitative creative decisions about emotional response

    • Academic research on consumer behaviour

  6. Where Analysis Remains Valid (Your Use Case)

    • Budget allocation and spend efficiency (primary use case)

    • Identifying when to rotate creatives to minimize costs

    • Detecting anomalies and performance shifts

    • Financial optimisation decisions

    • Comparative performance assessment across creatives

Solution:

CreativeDynamics is measuring platform ecosystem efficiency, not human preference. In a world where you pay platforms regardless of bot ratios, optimising spend based on observable performance patterns is rational and necessary—whether you’re advertising to humans, bots, or a mixture of both.

The Practical Reality: You’re optimising financial performance in the advertising system as it exists. The “true audience composition” is unknowable and, for budget allocation decisions, largely irrelevant. What matters is identifying when reported performance declines trigger higher costs, and responding accordingly.

Bottom Line: CreativeDynamics optimises what you can measure and control (platform-reported performance and associated costs), not what you cannot (exact audience composition). For its primary application—spend efficiency—bot contamination does not invalidate the analysis; it is already factored into the metrics you’re optimising.

See Also