Probaballer

Building a robust football analytics pipeline requires combining data from multiple sources: match results from one provider, player statistics from another, odds from bookmakers, and tracking data from yet another. The challenge? None of them agree on how to name things.

This is the entity matching problem (also called record linkage or data deduplication). Without solving it, you cannot join datasets, track players across seasons, or build features that span multiple data sources.

There is no single perfect solution. The best approach combines multiple techniques, using simple methods for easy cases and more sophisticated methods for ambiguous matches. Here is the toolkit:

Fuzzy matching quantifies the similarity between two strings, allowing for typos, abbreviations, and minor variations. The workhorse algorithm is Levenshtein distance.

from rapidfuzz import fuzz, process

teams_source_a = ["Manchester United", "Liverpool", "Arsenal"]
query = "Man Utd"

# Find best match
best_match = process.extractOne(query, teams_source_a)
# Returns: ("Manchester United", 73.3, 0)

# Set threshold for automatic matching
if best_match[1] > 80:
    matched_team = best_match[0]
else:
    # Flag for manual review
    pass

Phonetic algorithms encode names by how they sound rather than how they are spelled. This catches variations that fuzzy matching might miss.

Use case: Pre-filter candidates before applying more expensive similarity measures. If phonetic codes do not match, skip the comparison entirely.

Modern NLP models encode text into dense vector representations (embeddings) that capture semantic meaning. Similar names cluster together in embedding space, enabling matching even when surface forms differ significantly.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer('all-MiniLM-L6-v2')

names = ["Heung-Min Son", "Son Heung-min", "Harry Kane"]
embeddings = model.encode(names)

# Cosine similarity matrix
sim_matrix = cosine_similarity(embeddings)
# Son variants: ~0.92 similarity
# Son vs Kane: ~0.45 similarity

from openai import OpenAI
client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=["손흥민", "Son Heung-min"]
)
# Returns 1536-dimensional embeddings
# High similarity despite different scripts

import faiss
import numpy as np

# Build index from source A embeddings
index = faiss.IndexFlatIP(384)  # Inner product (cosine after L2 norm)
faiss.normalize_L2(embeddings_a)
index.add(embeddings_a)

# Query with source B names
faiss.normalize_L2(query_embedding)
distances, indices = index.search(query_embedding, k=5)
# Returns top 5 matches with similarity scores

String similarity alone cannot resolve all ambiguities. Metadata provides crucial disambiguation signals that dramatically improve matching accuracy.

def match_score(name_sim, dob_match, nationality_match, team_match):
    score = name_sim * 0.4
    if dob_match:
        score += 0.35  # Strong signal
    if nationality_match:
        score += 0.15
    if team_match:
        score += 0.10
    return score

# Threshold: score > 0.75 = auto-match, 0.5-0.75 = manual review

A production entity matching pipeline combines these techniques in stages, from cheap/fast to expensive/accurate: