Skriptované audity v Pythone

Prečo skriptované audity v Pythone

Skriptované audity sú systematické, opakovateľné merania webu, ktoré vytvárate ako kód. Pre oblasť programmatic SEO a technických kontrol prinášajú deterministickú reprodukovateľnosť, merateľnosť (výstupy do CSV/SQLite), jednoduchú automatizáciu (CRON/CI) a možnosť kombinovať HTTP, HTML, XML, JavaScript analýzu v jednom nástroji. Python poskytuje bohatý ekosystém (aiohttp/httpx, lxml/BeautifulSoup, urllib.robotparser, pandas, pydantic) a výbornú podporu pre paralelné I/O.

Architektúra auditu: od discovery po reporting

1. Discovery: získanie zoznamu URL (sitemap, logy, export z databázy, crawl).
2. Fetching: rýchle a ohľaduplné sťahovanie (HTTP/2, timeouty, retry, rate limit, robots).
3. Parsing: HTML/JSON-LD/XML hlavičky, meta tagy, link rel, structured data.
4. Kontroly: pravidlá (napr. title length, canonical, hreflang, meta robots, stavové kódy, redirect reťazce).
5. Uloženie: CSV/Parquet/SQLite pre audit trail a porovnanie v čase.
6. Report: agregácie, KPI, grafy, dify vs. predchádzajúci beh.

Štandardná štruktúra repozitára

python-seo-audits/ ├─ audits/ │ ├─ __init__.py │ ├─ sitemap_discovery.py │ ├─ fetcher.py │ ├─ html_checks.py │ ├─ indexation_checks.py │ ├─ links_checks.py │ ├─ performance_probe.py │ ├─ schema.py # pydantic dátové modely │ └─ reporters/ │ ├─ csv_reporter.py │ ├─ sqlite_reporter.py │ └─ md_summary.py ├─ bin/ │ ├─ run_audit.py │ └─ run_diff.py ├─ tests/ │ ├─ test_html_checks.py │ └─ fixtures/ ├─ data/ # do .gitignore (cache, dočasné výstupy) ├─ pyproject.toml # poetry/pip-tools ├─ Makefile # make lint, test, audit ├─ .pre-commit-config.yaml # ruff, black, mypy, yaml-lint └─ README.md

Zásady bezpečného fetchovania a etiky

• Rešpektujte robots.txt a Crawl-delay (ak robot akceptuje), nastavte vlastný User-Agent s kontaktným emailom.
• Dodržujte rate limiting (napr. 2–5 req/s/host), timeouty a retry s exponenciálnym backoffom.
• Nesťahujte citlivé sekcie, nespúšťajte testy počas špičky prevádzky, výsledky anonymizujte, ak obsahujú osobné údaje.

Modul: načítanie URL zo sitemap

# audits/sitemap_discovery.py from __future__ import annotations import asyncio import aiohttp from urllib.parse import urljoin from lxml import etree XMLNS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"} async def fetch(session: aiohttp.ClientSession, url: str) -> bytes: async with session.get(url, timeout=aiohttp.ClientTimeout(total=20)) as r: r.raise_for_status() return await r.read() def parse_sitemap(content: bytes) -> list[str]: root = etree.fromstring(content) # sitemapindex alebo urlset if root.tag.endswith("sitemapindex"): return [loc.text for loc in root.findall(".//sm:sitemap/sm:loc", namespaces=XMLNS)] return [loc.text for loc in root.findall(".//sm:url/sm:loc", namespaces=XMLNS)] async def discover_from_index(index_url: str) -> list[str]: async with aiohttp.ClientSession(headers={"User-Agent": "SEO-Auditor/1.0 <audit@example.com>"}) as s: content = await fetch(s, index_url) nodes = parse_sitemap(content) urls: list[str] = [] # drill-down do child sitemáp if nodes and nodes[0].endswith(".xml"): for node in nodes: c = await fetch(s, node) urls.extend(parse_sitemap(c)) return urls return nodes

Modul: rešpektovanie robots.txt

# audits/robots_guard.py import asyncio import aiohttp import urllib.robotparser as rp from urllib.parse import urlparse class RobotsGuard: def __init__(self, ua: str): self.ua = ua self.cache: dict[str, rp.RobotFileParser] = {} async def allowed(self, session: aiohttp.ClientSession, url: str) -> bool: host = urlparse(url).netloc if host not in self.cache: robots_url = f"https://{host}/robots.txt" try: async with session.get(robots_url, timeout=10) as r: text = await r.text(errors="ignore") except Exception: text = "" parser = rp.RobotFileParser() parser.parse(text.splitlines()) self.cache[host] = parser return self.cache[host].can_fetch(self.ua, url)

Asynchrónny fetcher s limitmi a retry

# audits/fetcher.py import asyncio, random import aiohttp from contextlib import asynccontextmanager @asynccontextmanager async def session_ctx(ua: str): timeout = aiohttp.ClientTimeout(total=25, connect=10) async with aiohttp.ClientSession(headers={"User-Agent": ua}, timeout=timeout) as s: yield s async def get_with_retry(session: aiohttp.ClientSession, url: str, retries: int = 2) -> aiohttp.ClientResponse: delay = 0.5 for attempt in range(retries + 1): try: resp = await session.get(url, allow_redirects=True) if resp.status in (429, 500, 502, 503, 504): raise aiohttp.ClientResponseError(resp.request_info, resp.history, status=resp.status) return resp except Exception: if attempt == retries: raise await asyncio.sleep(delay + random.random() * 0.5) delay *= 2

HTML kontroly: title, meta, canonical, hreflang

# audits/html_checks.py from bs4 import BeautifulSoup from urllib.parse import urljoin, urlparse def norm(s: str | None) -> str: return (s or "").strip() def check_html(url: str, html: str) -> dict: soup = BeautifulSoup(html, "lxml") title = norm((soup.title.string if soup.title else None)) meta_desc = norm(next((m.get("content") for m in soup.select("meta[name='description']")), "")) robots = norm(next((m.get("content") for m in soup.select("meta[name='robots']")), "")) canon = norm(next((l.get("href") for l in soup.select("link[rel='canonical']")), "")) hreflang = [(l.get("hreflang","").lower(), l.get("href","")) for l in soup.select("link[rel='alternate'][hreflang]")] issues = [] if not title: issues.append("missing_title") if len(title) > 70: issues.append("long_title") if len(meta_desc) == 0: issues.append("missing_description") if canon: # absolútny canonical a rovnaký host if not urlparse(canon).netloc: issues.append("canonical_relative") else: issues.append("missing_canonical") # hreflang páry if hreflang and ("x-default", None) is None: pass return { "url": url, "title": title, "meta_description_len": len(meta_desc), "robots_meta": robots, "canonical": canon, "hreflang_count": len(hreflang), "issues": ";".join(issues) }

Indexačné signály: stavové kódy, reťazce presmerovaní, noindex

# audits/indexation_checks.py def summarize_response(url: str, history, status: int, headers: dict, html_snippet: str) -> dict: chain = " -> ".join([f"{h.status}" for h in history] + [str(status)]) x_robots = headers.get("x-robots-tag", "") has_noindex = "noindex" in x_robots.lower() return { "url": url, "status": status, "redirects": len(history), "chain": chain, "content_type": headers.get("content-type",""), "x_robots_tag": x_robots, "noindex": has_noindex }

Výkonnostný „probe“ bez renderovania

Pre rýchle porovnanie latencií a prenosov využite iba sieťové metriky. Plné renderovanie (napr. pomocou Playwright) pridajte len pre vzorku alebo pre kľúčové šablóny.

# audits/performance_probe.py import time async def timed_fetch(session, url: str) -> dict: t0 = time.perf_counter() async with session.get(url) as r: await r.read() t1 = time.perf_counter() return { "url": url, "status": r.status, "ttfb_ms": r.headers.get("server-timing",""), # ak je dostupné "elapsed_ms": round((t1 - t0) * 1000, 1), "bytes": int(r.headers.get("content-length", "0") or 0) }

Spájanie modulov: hlavný runner

# bin/run_audit.py import asyncio, csv from audits.sitemap_discovery import discover_from_index from audits.fetcher import session_ctx, get_with_retry from audits.robots_guard import RobotsGuard from audits.html_checks import check_html from audits.indexation_checks import summarize_response async def audit(sitemap_url: str, out_csv: str): urls = await discover_from_index(sitemap_url) guard = RobotsGuard("SEO-Auditor/1.0 <audit@example.com>") rows = [] async with session_ctx("SEO-Auditor/1.0 <audit@example.com>") as s: for url in urls: if not await guard.allowed(s, url): rows.append({"url": url, "issue": "blocked_by_robots"}) continue resp = await get_with_retry(s, url) html = (await resp.text(errors="ignore")) if "text/html" in resp.headers.get("content-type","") else "" index = summarize_response(str(resp.url), resp.history, resp.status, resp.headers, html[:2000]) html_metrics = check_html(str(resp.url), html) if html else {} rows.append({**index, **html_metrics}) await asyncio.sleep(0.2) # rate limit with open(out_csv, "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=sorted({k for r in rows for k in r.keys()})) writer.writeheader() writer.writerows(rows) if __name__ == "__main__": import argparse ap = argparse.ArgumentParser() ap.add_argument("--sitemap", required=True) ap.add_argument("--out", default="audit.csv") args = ap.parse_args() asyncio.run(audit(args.sitemap, args.out))

Rozšírenie: validácia hreflang párov a návratových odkazov

Pre hreflang je kľúčové, aby každá URL mala return link (recipročný odkaz) a existovala aj x-default varianta. Rozšírte model o mapovanie párov a kontrolu existencie.

# audits/links_checks.py from collections import defaultdict def validate_hreflang(records: list[dict]) -> list[dict]: # records musia obsahovať polia: url, hreflang_count a extrahované páry (rozšírte check_html) # Tu len ilustrácia reciprocity na úrovni hostu/cesty. mapping = defaultdict(set) for r in records: # predpoklad: r["hreflang_pairs"] = [(lang, href), ...] for lang, href in r.get("hreflang_pairs", []): mapping[r["url"]].add((lang, href)) issues = [] for url, pairs in mapping.items(): for lang, href in pairs: rev = mapping.get(href, set()) if (lang, url) not in rev: issues.append({"url": url, "issue": "hreflang_missing_return", "lang": lang, "target": href}) return issues

Ukladanie do SQLite a porovnávanie behov

# audits/reporters/sqlite_reporter.py import sqlite3 DDL = """ CREATE TABLE IF NOT EXISTS audit ( run_id TEXT, url TEXT, status INT, redirects INT, content_type TEXT, title TEXT, meta_description_len INT, canonical TEXT, robots_meta TEXT, noindex INT, issues TEXT, PRIMARY KEY(run_id, url) ); """ def save_rows(db_path: str, run_id: str, rows: list[dict]): con = sqlite3.connect(db_path) con.execute(DDL) cols = ["run_id","url","status","redirects","content_type","title","meta_description_len", "canonical","robots_meta","noindex","issues"] with con: for r in rows: con.execute(f"INSERT OR REPLACE INTO audit ({','.join(cols)}) VALUES ({','.join(['?']*len(cols))})", [run_id, *[r.get(c.split('run_id,')[-1], None) for c in cols[1:]]]) def diff(db_path: str, run_a: str, run_b: str) -> list[tuple]: con = sqlite3.connect(db_path) q = """ SELECT a.url, a.status AS old_status, b.status AS new_status, a.canonical AS old_canon, b.canonical AS new_canon FROM audit a JOIN audit b USING(url) WHERE a.run_id=? AND b.run_id=? AND (a.status!=b.status OR a.canonical!=b.canonical) """ return list(con.execute(q, (run_a, run_b)))

Minimalistický Playwright „render check“ (vzorka)

Na detekciu problémov závislých od JS (hydration, prerender) použite Playwright iba pre malú vzorku URL.

# audits/js_render_probe.py import asyncio from playwright.async_api import async_playwright async def render_probe(urls: list[str]) -> list[dict]: async with async_playwright() as p: browser = await p.chromium.launch(headless=True) page = await browser.new_page() out = [] for u in urls[:50]: try: resp = await page.goto(u, wait_until="domcontentloaded", timeout=15000) html = await page.content() out.append({"url": u, "render_status": resp.status if resp else None, "html_len": len(html), "has_title": "<title>" in html.lower()}) except Exception as e: out.append({"url": u, "render_error": str(e)}) await browser.close() return out

Programmatic SEO: kontrola šablón a generátorov

• Šablónové konzistencie: fixná dĺžka a štruktúra <title>, povinné polia v JSON-LD (Product, Article, FAQ), unikátnosť H1.
• Kanonicita: správny rel=canonical bez samokanibalizácie (žiadne odkazy na parametre, staging, http).
• Indexačná hygiena: vyhnite sa noindex na kategóriách, ktoré majú byť prístupné; odstráňte reťazce presmerovaní.
• Interné prelinkovanie: pri programatickom generovaní stránok kontrolujte, či každá entita má aspoň N interných odkazov z relevantných hubov.

Automatizácia v CI: príklad workflow

# .github/workflows/audit.yml name: SEO Audit on: schedule: [{cron: "0 3 * * 1,4"}] workflow_dispatch: {} jobs: run: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: { python-version: "3.12" } - run: pip install -r requirements.txt - name: Run audit run: python bin/run_audit.py --sitemap "https://www.example.com/sitemap.xml" --out data/audit.csv - uses: actions/upload-artifact@v4 with: { name: "audit-report", path: "data/audit.csv" }

Reporty a KPI pre kontinuálne sledovanie

• Indexability rate: podiel URL s 200 a bez noindex.
• Redirect-free rate: podiel URL dostupných bez presmerovania.
• Canonical correctness: URL, ktorých canonical = finálna 200 URL a nejde na URL s parametrom.
• Meta coverage: percento URL s neprázdnym title a description v odporúčaných rozsahoch.
• Hreflang reciprocity: percento párov s korektnou návratovosťou.

Rozšírené kontroly pre štruktúrované dáta

# audits/structured_data.py import json, re from bs4 import BeautifulSoup RE_TYPES = {"Product","Article","FAQPage","BreadcrumbList","Organization"} def extract_jsonld(html: str) -> list[dict]: soup = BeautifulSoup(html, "lxml") out = [] for tag in soup.select('script[type="application/ld+json"]'): try: data = json.loads(tag.string) if isinstance(data, dict): out.append(data) elif isinstance(data, list): out.extend(data) except Exception: continue return [d for d in out if isinstance(d.get("@type",""), (str,list))] 

Práca s dátami: pandas & export do Parquet

# audits/reporters/csv_reporter.py import pandas as pd def to_parquet(csv_in: str, parquet_out: str): df = pd.read_csv(csv_in) df.to_parquet(parquet_out, index=False) def summarize(csv_in: str) -> pd.DataFrame: df = pd.read_csv(csv_in) agg = df.groupby("status")["url"].count().reset_index().rename(columns={"url":"count"}) return agg

Testovanie: rýchle unit testy a fixtúry

# tests/test_html_checks.py from audits.html_checks import check_html def test_title_and_canonical(): html = "<html><head><title>Ahoj</title><link rel='canonical' href='https://example.com/a'></head><body></body>" res = check_html("https://example.com/a", html) assert res["title"] == "Ahoj" assert res["issues"] == ""

Škálovanie: batching, priorita, sampling

• Batching: spracujte URL v dávkach (napr. 500–2000), ukladajte priebežné checkpointy.
• Prioritizácia: začnite od hubov (kategórie, top landing pages z analytics/logov).
• Sampling: JS render overujte len na malom vzorku, zvyšok stačí bez renderu.

Bezpečnosť a tajomstvá

• API kľúče (napr. do meracích služieb) ukladajte do ENV premenných alebo secret managera.
• Logger nikdy nevypisuje celé URL s tokenmi; parametre sanitizujte.

Roadmapa repozitára: čo ďalej pridať

• Diff engine: porovnanie dvoch behov a generovanie Markdown reportu s top zmenami.
• Grafy: jednoduché vizualizácie (matplotlib/plotly) pre KPI v čase.
• Queue: možnosť distribuovaného spracovania (RQ/Celery) pri veľmi veľkých weboch.
• Pluginy: rozhranie pre vlastné kontroly (napr. business rules na špecifické šablóny).

Makefile, pre-commit a štýl

make lint # ruff + black make test # pytest make audit # spustí bin/run_audit.py s default parametrami

# .pre-commit-config.yaml (výňatok) repos: - repo: https://github.com/astral-sh/ruff-pre-commit rev: v0.7.0 hooks: [{id: ruff, args: ["--fix"]}] - repo: https://github.com/psf/black rev: 24.8.0 hooks: [{id: black}]

Praktický checklist pred ostrým behom

1. Definovaný User-Agent a kontaktný email, rešpektovanie robots.txt.
2. Rate limit <= 5 req/s/host, timeout ≤ 25 s, retry s backoffom.
3. Logovanie do súboru + konzoly, zachytenie výnimiek a metadát požiadaviek.
4. Výstupy do CSV/SQLite s jednoznačným run_id.
5. V CI naplánované behy mimo špičky, upload artefaktov, notifikácie o zmenách.

Skriptované audity v Pythone spájajú rýchlosť a presnosť s možnosťou škálovať od stovák po milióny URL. Vďaka modulárnej architektúre a disciplíne „audit-as-code“ získate opakovateľný, auditovateľný a rozšíriteľný systém, ktorý sa stane základom vašej programmatic SEO praxe.