As Arcesium's Field CTO, Matt leads Arcesium's Forward Deployed Software Engineering and Client Success teams. His work to empower clients and simplify technical challenges stems from a 25-year career in financial technology working with clients and software. Outside work, he enjoys books, bikes, and boards.
Any tech leader will tell you about the immense pressure they're under to drive innovation, optimize operations, and provide their business with a competitive edge.
Tech leaders are already aware that they should harness the power of data, unlock insights, and democratize their information. These catchphrases are all well and good — and quite honestly the goal of many skilled professionals — but getting there is much more intricate than the lexicon suggests.
The path to data mastery is often fraught with challenges — from the scarcity of skilled data professionals to the complexities of building and maintaining custom pipelines and models. The industry’s growth trajectory, data generation, and demand for skilled talent are all growing at hyper-paced rates:
The financial services industry will reach $37.5 billion by 2027, boasting a 7.5% CAGR.1
By 2025, the global datasphere is expected to exceed 180 zettabytes.2
Three in four employers report difficulty in filling roles.3
Many organizations are competing for skilled professionals who bring experience in driving innovation and operational efficiency. This competence is particularly acute in data management and analytics for the financial services industry, where the demand for technical expertise often exceeds supply. Recent research from Northern Trust found that 83% of survey respondents were considering outsourcing to augment their data management capabilities.4
Transforming your infrastructure with pre-built data models and connectors can reduce your reliance on hard-to-find technical talent. However, reshaping your approach is about more than hiring for the right skill sets. A big challenge CTOs often face is where to add staff when they do hire. Tech leaders want technical talent to concentrate on resolving the dilemmas that deliver high returns and on figuring out the best way to address problems portfolio managers or trading teams need solved and are unique to their business. But where do you start? How do you know which platform is best for your business? And how do you ensure your teams can maintain oversight of critical functions?
Setting up a data platform
A common misconception is that setting up a database equates to having a data platform. However, a database is not a data platform.
The database is the “easy” part. What comes next is the hard part. Setting up a database — whether cloud-native like Snowflake, CloudSQL, Azure SQL, and Aurora or closer to the metal like PostgreSQL, SQLServer, and Oracle — can often be straightforward. The real work begins after the infrastructure is in place. Efficiently running and managing a database is an art that marries knowledge of the DB internals with the state of your data.
If database management is a core differentiator for your business, investing in skilled database administrators is crucial. These experts understand intricacies like the implications of page cache sizes, disk storage structures, and optimizing for IO vs storage. However, if database management isn't a primary driver of your business's success, opting for a managed data service can save time and resources.
Once your infrastructure is set up, the next step is to model the "nouns" and "verbs" of your business. This involves making numerous decisions that balance flexibility, extensibility, and efficiency. Even a seemingly simple concept like a "security" can be complex and require you to carefully consider what fields are essential for your organization and which are more of a nice to have. Teams can spend considerable time debating these points, weighing trade-offs related to tuning, storage models, validation, disaster recovery, and more.
The people who implement your data models must understand data and have deep knowledge of your business domain. They ensure that the models reflect your current operations and can adapt as your business evolves. However, continuously refactoring models and migrating data can be resource-intensive. If you’ve done a data migration, you know nothing’s ever simple. A more efficient approach is to leverage pre-built, tested, and extensible models that cover the entire lifecycle of your business operations — and have your growth built in.
Loading data and connectors
With your data models in place, getting data in will be your next adventure. Initially, your teams can probably manage small amounts of data with custom scripts. However, as the volume and variety of your data increases, you may find your scripts are inadequate, leading to inconsistency and data loads that become difficult for your teams to quickly troubleshoot. You’ll find data quality and validation become more challenging and issues will hide in the data to be found much later. You’ll also have to divide your talented resources between building data infrastructure and solving business problems.
Your technical staff will likely encounter numerous puzzles related to different data sources, each with unique perspectives on reality, time, and data quality. These challenges, while valuable learning experiences, often lead to frustration and high turnover as staff seek more engaging work.
To streamline data ingestion, it's essential to build robust data connectors. Consider these critical choices as you build and load your data models:
Determine ETL vs. ELT: Determine whether to use extract, transform, load or flip the script and begin with the extraction, then load your data, and transform the information as the final step in the process. The choice depends on your data processing needs and infrastructure.
Choose snapshots or deltas: Decide whether to load full snapshots or incremental deltas. Snapshots provide a complete view but can be resource-intensive, while deltas are more efficient but complex to manage.
Be auditable: Implement mechanisms to track data changes, ensuring you have a bitemporal model that captures both effective and knowledge time. You must be able to show what you know and when you knew it.
Promote monitoring and supportability: Ensure each connector is debuggable, can be replayed, and includes alerting and monitoring tools. This helps quickly identify and resolve issues. You may skip this if no one at your business ever makes mistakes.
Plan for updates and maintenance: Keep connectors updated as data sources evolve. This involves regularly reviewing release notes and understanding what changes impact your data.
Avoid duplication: Address potential data duplication issues when re-running connectors. Ensure your processes can refresh and re-ingest updated data without duplicating entries.
Building a data platform from scratch requires significant technical talent — an increasingly scarce and expensive resource. As a result, companies often feel like they face a choice: hire highly experienced professionals to re-implement what they've done before or hire less experienced individuals who will learn and make mistakes along the way.
However, the challenge many firms run into is that this work doesn’t always align with the goals of talented professionals who are eager to grow professionally and test innovative ideas with your data. Many professionals are probably intrigued and excited at the prospect of helping your firm establish data models. But the routine tasks of maintaining connectors and models are not what they find fulfilling. Instead, they want to engage in activities that drive your business forward. Your best tech talent will want to work close to the problems that drive your profits.
Building a data model from scratch is a daunting task, but there's a more efficient way: leveraging platforms with pre-built models and connectors. The pre-built approach offers several benefits:
Reduced technical debt: By using pre-built solutions, you avoid the pitfalls of reinventing the wheel. These solutions are designed to handle common data management challenges, reducing the need for extensive custom development.
Faster time to value: Pre-built models and connectors accelerate your data journey, enabling you to derive value from your data much sooner. This agility is crucial in a fast-paced business environment.
Attracting top talent: High-caliber technical professionals are more likely to join and stay with your organization if they can focus on innovative and strategic tasks rather than routine data management.
Scalability and flexibility: Pre-built solutions are designed to scale with your business. They offer the flexibility to adapt as your data needs evolve, ensuring long-term sustainability.
Cost efficiency: By reducing the need for extensive custom development and ongoing maintenance, pre-built solutions can lower your overall costs.
Battling back
As firms vie for top talent, having a data platform with pre-built models and connectors can be a game-changer. It reduces dependence on scarce technical talent, accelerates your data journey, and frees your team to focus on strategic, high-value tasks. Platforms purpose-built to support your data needs offer these capabilities, enabling your organization to leverage the full potential of its data without the heavy lifting of building everything from scratch.
