Making the complex simple
Mapador parses your production code bases to show how data and code flow across your applications, at a fraction of the cost of doing it by hand. This page explains how we do it, why our results hold up where general-purpose parsers fail, and what you get at the end.
From production code to a connected map
Every engagement follows the same path. Each step exists because real code bases, especially legacy ones, hide connections that a single generic parser can’t see.
- 01
Ingest the production environment
We work from what actually runs: source code, scripts, schedulers and every run-time parameter. Not documentation, not interviews.
- 02
Unpack, convert and recognize
Client code is unpacked, converted to a workable form and sorted by file type, so each artifact goes to the right parser.
- 03
Thin parse and compare
A fast first pass with search patterns finds the structure, and results are compared to catch anything the file-type step missed.
- 04
Grammar-specific parsing
Custom, fit-for-purpose parsers for each language (JCL, Metasuite, shell and others) break source code into usable components and relationships.
- 05
Instantiate, emulate and resolve
Embedded SQL is parsed in context (JCL SQL, Control-M SQL, shell SQL). We emulate code to fill gaps in the lineage and resolve every parameter to a real name.
- 06
Map into one repository
Everything is connected in a single repository, whatever technology it came from.
- 07
Refine and export
Raw technical lineage is never enough on its own. Customizable processes turn it into something more useful, delivered as PDF reports, JSON exports and the Mapador interface, with exports tailored for loading into your AI models, catalogue or MDM system.
Accuracy comes from iteration. We validate results against the running system, measure gaps, and refine parsers until the lineage holds.
Four levels built on one repository
Results are layered. Each level is built from the one below it, so you can work at the level of detail your audience needs, from an application overview to a single column.
L4Business & regulatory lineage
Built with selected clients on top of Level 3: a regulator-ready lineage in business language, connected to lineages from other areas.
L3Field-column lineage
Granular data flow at the field and column level, showing transformations between source and target and full paths from target back to source, or forward if required.
L2File-table lineage
Relationships between files and tables across the whole application. Each data movement is tagged with a table-file lineage instance showing how scheduled jobs, tables and files interact.
L1Application relationships
Application, job and script connections, call trees and control flows for schedulers and programs. These results feed Levels 2 and 3.
BaseFoundational repository
Every artifact collected and categorized: source code, scripts, parameters and scheduler information. The base of all metadata tagging.
Aggressive summarization
Level 4 reduces the technical lineage to only the nodes that matter to the business and the regulator.
Connectors to other lineages
Our lineage links to lineages created in other areas of the organization, so the picture is end to end.
Business language
Technical program code is translated automatically into business terms and tied to corporate data dictionaries.
Connecting to lineage from other tools
Most organizations already capture parts of their lineage in ETL, catalogue and BI tools. Our connectors stitch those fragments to ours into one end-to-end view, using business standards set at each integration point, customized first and then automated.
Real code is never as simple as it looks
General-purpose parsers read code line by line. Production systems don’t work that way. These are the patterns that break ordinary lineage, and that our approach is built to handle.
Scripts calling scripts calling scripts
A scheduler runs a script that calls another script, and so on, seven levels deep. Each one picks up information, may change it, and passes it on to one or more targets. Scripts may run CRUD operations on the same sources or share libraries, which hides the connections. All of these pieces have to be brought back together.
Many in, one through, many out
Multiple fields end up in a single buffer. Multiple columns collapse into one column of a fact table. Twenty files load into one table, which is then split back out into twenty output files. Without emulation the individual paths disappear. This matters most in systems that rely heavily on fact tables.
The table rename that ordinary parsing misses
What the code seems to say
- Warehouse
- ↓
- Loaded table
- ↓
- Production table
- ↓
- Production backup
What actually happens
- New data is loaded into a “load” table.
- The production table is renamed to “backup”.
- The load table is renamed to “production”.
Standard SQL parsing doesn’t follow the renames, and the lineage breaks. In one analysis, following them correctly changed the picture completely:
Every run is named in full
We resolve parameters until each table-file run instance has a fully qualified name. One name records where the data movement belongs, what ran it, how often and which program did the work.
bankProjectCTRL-MScheduler typeFINW#D210.FINW_D0831.2.4417Scheduler / job nameDAILY_RISKFrequency & business unitFIN08U02.FIN08P02Calling script & executed programIllustrative example. Real names come from your own scheduler and code.
Metadata tagging is the foundation
We tag metadata across the whole of an application’s proprietary code. That tagging is what lets us produce lineage for regulatory, modernization and analytics work from the same base.
Parsing & extraction
- Proprietary, fit-for-purpose parsers for each language
- New parsers built whenever a client’s code needs one
- Parameters resolved, flows instantiated and emulated where needed
- Specialists in legacy code such as COBOL, PL/I, JCL and Fortran, as well as modern languages
Attributes captured
- Field and column types
- Application codes
- Indexes and synonyms
- Specifications and more, wherever available and relevant
Classification
- Metadata classified using multiple standard classes
- Classes customized for each client
- PII and critical data elements traced for compliance
- Golden sources identified for analytics
Granularity
- Application level
- File and table level
- Field and column level
- Summarized or fully detailed, depending on the audience
Technical & business metadata
- Full technical metadata generated from code
- Business metadata created from the technical metadata
- Existing business glossaries ingested and used
- Translation into business language for regulators
Outputs & integration
- Data lineage, process flows and call trees
- Inheritance hierarchies, data access rules and CRUD
- PDF reports, JSON exports and the Mapador interface
- Exports built to each client’s requirements, loadable by other applications such as Collibra
One base, many uses
Because everything sits in one repository, the same work serves developers, the business and regulators.
Onboard & train
New and existing staff get productive quickly, even on legacy code.
Understand complex systems
Developers see the nuances of systems no one person fully knows any more.
Speed up development
See how a code change affects everything downstream before it’s committed, and cut rework.
Map data flows
See how data moves and which applications transform it along the way.
Satisfy regulators
Accurate lineage for BCBS 239 and similar frameworks, supporting governance frameworks and controls.
Feed analytics & AI
Our repository is the base data behind intelligent, business-focused knowledge graph and AI platforms.
See it on your own code
The best way to judge our approach is on a sample of your systems. Tell us what you’re working with and we’ll show you what we find.