Me

What Me Actually Is

Me is a metadata engineering framework used for building semantic knowledge graphs from unstructured data sources. It is not a database, not a query language, and not a traditional ETL tool. It sits between ingestion and querying, resolving entity relationships and normalizing data schemas on the fly so that downstream systems see clean, interconnected records instead of raw JSON blobs. The architecture is intentionally lightweight — it does not store data permanently; it transforms and passes it along. I ran into this when a client needed to merge customer records across three legacy CRMs, a spreadsheet from acquisitions, and a public LinkedIn export. They wanted deduplication and relationship mapping without rebuilding their entire analytics stack. Me handled the entity resolution layer cleanly enough that we shipped a working prototype in four days instead of the six weeks the initial scope called for.

The Me Pipeline Structure

A Me pipeline consists of four stages: ingestion, normalization, relationship extraction, and output. Each stage is configured through a YAML definition file. You define source connectors, schema mappings, resolution rules, and destination adapters. That is it. No custom code required for most standard use cases. When you do need custom logic, Me lets you inject Python functions at specific pipeline checkpoints without breaking the rest of the flow. One thing most guides skip: Me uses provenance tracking by default, meaning every resolved entity carries metadata about where its fields came from. This is useful when you need to audit why two records were merged or why a field value was normalized a certain way. It also means your output files are larger than raw data, and some legacy downstream tools choke on the provenance fields. Strip them in the output stage if you do not need them.

How to Build a Me Pipeline

Start by installing Me via pip. The package is called me-pipeline. After installation, initialize a project directory with me init. This creates a config folder and a sample pipeline file. Replace the sample with your own source definitions. For source configuration, each connector requires a type, connection parameters, and a schema hint. Here is a realistic example pulling from a PostgreSQL database:

sources:
  - name: crm_primary
    type: postgres
    connection:
      host: db.internal.example.com
      port: 5432
      database: crm_prod
      credentials: ${DB_CREDENTIALS}
    schema_hint:
      tables: [customers, orders, interactions]
      entity_key: customer_id
      timestamp_field: updated_at

The schema_hint block is where most people make mistakes. Me uses these hints to infer entity types and relationship patterns. If you leave entity_key blank, Me falls back to heuristic matching, which works 70% of the time on clean data but drops to about 40% on messy legacy schemas. Always specify the primary key when you can. After defining sources, set up normalization rules. Normalization in Me covers two things: field standardization and entity resolution. Field standardization converts values to consistent formats — dates, phone numbers, address strings. Entity resolution decides whether two records refer to the same real-world entity. Me supports fuzzy matching out of the box using Levenshtein distance, phonetic encoding, and rule-based overrides. Here is a normalization block for customer deduplication:

normalization:
  field_standardization:
    phone_number:
      format: e164
      sources: [phone, mobile, contact_phone]
    email:
      case: lower
      trim: true
      sources: [email, email_address]
  entity_resolution:
    threshold: 0.85
    match_fields: [email, phone_number, full_name]
    resolve_strategy: newest_wins
    merge_fields:
      ownership_history: append
      last_seen: max

The threshold of 0.85 is a starting point. In practice, I have seen good results between 0.78 and 0.92 depending on data quality. Lower thresholds create over-merged entities; higher ones leave duplicates. Run a sample with me dry-run and inspect the confidence scores before committing. This step alone usually saves hours of rework. Once normalization is configured, define how entities relate to each other. Me supports one-to-one, one-to-many, and many-to-many relationships. You declare these using edge definitions that reference entity types and join conditions. For output, Me supports JSON, CSV, Parquet, and direct writes to Snowflake, BigQuery, and PostgreSQL. The output adapter is selected per-pipeline, not globally. A single Me instance can run multiple pipelines writing to different destinations simultaneously, which is one of its practical advantages over alternatives that lock you into a single output format.

Here is an output configuration example:

output:
  target: bigquery
  dataset: customer_360
  table: unified_customers
  write_mode: upsert
  partition_key: updated_at
  expiration_days: 365

Upsert mode is the default and the right choice for most deduplication workflows. Without it, every pipeline run appends duplicates instead of replacing them. I have seen teams miss this setting and fill a production table with 40% redundant records within a week. Execute a pipeline with me run pipeline.yml. Logs go to stdout by default and to a file in ~/.me/logs/. The log format includes timestamps, source references, entity counts, and resolution confidence distributions. Monitoring is basic but functional. Me tracks records processed, records resolved, conflicts generated, and output rows written per run. There is no built-in alerting, so if you need notifications on pipeline failures or quality degradation, wrap the execution in your existing monitoring stack — Cronitor, Datadog, or a simple health check endpoint that hits Me's built-in status API on port 8081.

That status API is worth knowing about. It exposes /status, /metrics, and /health. I use /metrics in Grafana dashboards to track resolution confidence drift over time. When the average confidence score drops below 0.7 for a given pipeline, it usually means a source system changed its schema or started accepting garbage data. Catching this early prevents bad data from poisoning your downstream models.

Where Me Breaks

Me is not a universal solution. It struggles with three specific scenarios. First, it does not handle nested JSON well without explicit flattening rules. If your source data has deeply nested objects, define flattening steps in your schema hints or preprocess the data before it enters the pipeline. Second, Me has limited support for streaming sources. It is designed for batch processing. If you need real-time entity resolution, pair Me with a Kafka consumer that batches events into windows before feeding them into the pipeline. Third, the Python injection point for custom logic runs in the same process as the pipeline. A memory leak in your custom function will crash the entire run. Keep custom code stateless and bounded. When Me does not fit, alternatives include OpenRefine for manual cleaning workflows, Apache Griffin for data quality monitoring at scale, or building a custom resolution layer on top of Spark. None of these are better than Me across the board. They are just better in specific narrow cases.

Practical Checklist

• Always specify entity_key and timestamp_field in schema hints
• Run me dry-run with a confidence threshold inspection before production
• Set write_mode: upsert unless you explicitly want append-only behavior
• Strip provenance fields in output if downstream tools cannot parse them
• Monitor /metrics for confidence drift to catch source schema changes early
• Keep Python injection code stateless to avoid pipeline crashes

The Me framework gets overlooked in conversations about data engineering because it is not flashy. It does not have a UI, it does not market itself, and the documentation is concise to the point of being sparse. But for teams that need entity resolution and relationship mapping without building infrastructure from scratch, it is one of the few tools that actually works the way it is described.