The Tension Between Speed and Accuracy
For data teams in regulated industries, the pressure is often paradoxical: move faster, but don’t break anything.
Regulatory deadlines are non-negotiable. Whether it is IFRS-17 quarterly reports, AML transaction monitoring, or standard regulatory filings, these deliverables come with hard cut-off dates and severe penalties for late submission.
However, the tolerance for error is zero. A single miscalculation in actuarial reserves or a missed suspicious transaction can result in regulatory fines, reputational damage, or compliance failures.
This creates a high-stress environment where discovery is often too late. Teams frequently find data quality issues days before a deadline, forcing frantic manual reconciliations. It creates a “legacy trap” in which the business demands faster insights for decision-making and regulators demand perfect accuracy, yet IT teams manage legacy systems that were not built for either.
Overnight batch job failures often mean teams arrive at work to urgent problems with no time for proper testing or fixes. The result is a team constantly juggling the need for speed, full traceability, and pressure from multiple stakeholders—often relying on heroism rather than robust managed systems to get across the line.
The Risks of Legacy Approaches
Many organisations are still running critical compliance reporting on infrastructure that hasn’t evolved with their needs. This “Old Way” is characterised by opacity and manual effort.
Monolithic SAS macros often run as scheduled tasks with minimal visibility. You don’t know something has gone wrong until the entire process completes—or fails. To manage this, teams frequently rely on “human middleware”, emails or spreadsheets, to track which jobs should run after others, which slows down and causes errors in the end-to-end process.
Furthermore, a lack of version control means production code often lives on shared drives, with changes tracked by filename conventions such as AML_Report_v3_Final.sas. This makes it nearly impossible to trace exactly what changed, when, and by whom. Combined with resource contention, where unmanaged jobs compete for database connections, this leads to delays and unpredictable runtimes.
Modern Orchestration with Airflow and CI/CD
The solution isn’t necessarily to rip and replace the core logic—which is often sound—but to wrap it in modern orchestration and development practices.
Airflow for SAS Job Management
By introducing tools like Apache Airflow to orchestrate SAS jobs, we move from “scheduled tasks” to “intelligent pipelines.”
- Visual Transparency: Directed Acyclic Graphs (DAGs) provide a visual representation of all dependencies, making the workflow transparent and auditable. This helps proactively detect bottlenecks or failing nodes.
- Real-Time Monitoring: Instead of waiting for a phone call, teams receive email alerts the moment a job fails, with logs and error messages available immediately.
- Parallel Execution: Independent tasks run concurrently rather than sequentially, dramatically reducing total pipeline runtime.
- Automated Productivity: Automated pipelines reduce repetitive manual work, freeing up the team to focus on client needs rather than babysitting batch jobs.
Code-as-Infrastructure for Compliance
Adopting “Infrastructure-as-Code” creates built-in auditability. Every run, configuration change, and code update becomes traceable and reproducible.
- Git-Based Version Control: All changes to logic, data transformations, or pipeline settings are tracked. We know who made the change, when, and why.
- Peer Review: Through pull requests, no code reaches production without a review, creating an automatic audit trail before the code even runs.
- Automated Testing: Modern CI/CD pipelines allow for unit tests (calculation functions), integration tests (data quality), and regression tests (comparing outputs to known-good results).
- Environment Parity: Development, Test, and Production environments are configured identically through code. This ensures consistent results, better stability, and higher data quality.
- Rollback Capability: If an issue arises, teams can revert to the last stable version in minutes, not hours.
What Speed Gains Enable
Modernising the pipeline doesn’t just save time; it changes how the team operates. Experience shows that significantly reducing pipeline runtime yields downstream benefits that extend far beyond IT.
Data issues surface sooner, giving teams breathing space for earlier validation and the ability to investigate thoroughly rather than resorting to quick fixes. This leads to faster sign-offs, as Finance and Risk teams can approve reports earlier, easing end-of-period pressure.
Crucially, it creates capacity for deeper analysis. Time saved on routine troubleshooting can be reinvested in high-value work, such as enhanced risk segmentation or predictive modelling. It also unlocks business agility, enabling ad hoc regulatory requests, format changes, and new product launches to be supported much faster, removing IT as a bottleneck.
Ultimately, this builds stronger stakeholder relationships. Finance and compliance teams gain confidence when reports consistently arrive early and are accurate.
Advice for Teams on Legacy Tools
You don’t need to fear the regulator—or your legacy system. Modern practices like version control, automated testing, and documented pipelines are inherently more compliant than manual processes. Modern tooling isn’t just “faster”; it’s safer, more predictable, and far easier to audit.
- Start Small and Prove Value: Avoid the temptation to modernise everything at once. Select one painful, high-visibility workflow—such as month-end regulatory reporting—and prove the success of orchestration there. Tools like Airflow can run existing SAS jobs, allowing you to modernise the workflow while gradually refactoring the code.
- Strengthen Visibility Before Optimisation: Before optimising for speed, ensure you can see what is happening. Logging, monitoring, and alerting provide the foundation for improvement. Build dashboards that show pipeline health, runtime trends, and failure patterns—executives respond to data.
- Embrace the Cultural Shift: Moving from manual processes to automated, version-controlled pipelines is a cultural change as much as a technical one. It requires training, documentation, and patience, but the payoff is a team that spends less time fighting fires and more time delivering value.
Stop relying on heroism to meet your deadlines.
If your month-end reporting relies on a single person knowing which spreadsheet to update, it’s time to talk. Selerity helps compliance teams build robust, automated pipelines that keep the regulators happy and your weekends free.
Contact our Modernisation Team