Starting at the end, architecture first and building cost strategy into design – this is what we learned in high-volume data management in major regulated industries.

Handling several million file uploads a month teaches you a thing or two about scaling, security and smart design.

At Kohde, we recently helped a large, multi-site regulated enterprise modernise its data infrastructure. The goal: move massive volumes of sensitive records to the cloud securely, reliably and without disruption.

See how we helped them cut their data costs with Azure tiers.

Not a straightforward lift-and-shift, the system needed to handle millions of files from multiple applications, detect and prevent duplication, enforce unique retention rules per application, and meet strict regulatory and network constraints. All while reducing costs and improving performance.

It was a technically demanding storage challenge, with plenty of lessons in what makes or breaks systems at this scale. Here are five lessons any IT Systems Owner managing enterprise data should take to heart.

1. Architecture matters more than you think

At high scale, bad architecture compounds fast. That’s why we built the system on an event-driven architecture, with a single REST API as the entry point and background workers processing heavy tasks asynchronously.

This allowed us to:

• Maintain responsiveness under huge load spikes
• Decouple core workflows (like file uploads) from backend logic (like duplicate detection and retention enforcement)
• Scale horizontally using container orchestration

“We needed to support several million file uploads every month, all through a single entry point and without storing a single duplicate. That’s where the real engineering challenge lies,” explains Christoff Labuschagne, Technical Solutions Architect at Kohde.

If your team is still pushing all logic into tightly coupled APIs or struggling with performance bottlenecks, rethink your event flow and scalability model. It’s not about brute force — it’s about elegant separation of concerns.

2. Your cost strategy needs to be built into your design

We reduced storage and backup costs from a seven-figure monthly spend to the low five figures — not through vendor discounts, but through smart use of Azure Blob Storage tiers.

By tracking access patterns, we moved files from hot to cold storage after just two weeks. Cold storage is dramatically cheaper for long-term retention, especially for infrequently accessed files.

“The key difference is usage. Hot storage is cheaper to access but expensive to store. Cold storage is the opposite. Once we understood usage patterns, tiering became a game-changer.” – Christoff Labuschagne.

Storage cost isn’t just a procurement line item; it’s a design choice. If your current system treats all files the same, you’re likely overpaying.

3. Compliance isn’t a step — it’s a foundation

In highly regulated environments, compliance isn’t optional. Some records must be retained for decades. Others must never be accessible to unauthorised apps. Data residency constraints apply. And it all has to work seamlessly, at scale.

That meant designing access controls and retention enforcement per application, even if multiple apps referenced the same file. It also meant keeping the cloud-hosted environment connected to the enterprise’s internal network under strict security controls.

“Granular access control was a must. We had to support file-sharing via secure links between applications while still respecting each system’s specific retention and access policies.” – Christoff Labuschagne.

If compliance is something you retrofit later, you’ve already lost. Design for it from day one.

4. Monitoring is your safety net — automate it

When a system reaches this scale, manual checks break down fast. That’s why we built in real-time monitoring using Azure Application Insights and Exceptionless for error tracking and performance bottlenecks.

This gives the enterprise visibility across:

• Upload performance and throughput
• Background task efficiency
• Error rates by application or API
• Capacity planning signals

“Big systems demand big thinking. It’s not just about fixing inefficiencies, it’s about building smart, scalable infrastructure that can stand the test of growth.” – Christoff Labuschagne.

Monitoring isn’t just about dashboards. It’s about observability: knowing the right thing before it breaks and having systems that self-report intelligently.

5. Start with the outcome and work backwards

One of the most common mistakes in enterprise IT is trying to modernise everything at once. That wasn’t the brief for this engagement; they needed a solution that worked with their legacy systems, respected uptime, and still delivered real performance and cost improvements.

So we didn’t start with platforms. We started with outcomes: eliminate duplication, retain sensitive records securely, scale without latency and cut costs. And everything followed from that.

“Start with the end in mind. Know your outcomes, then design backwards. That’s how you stay compliant, scalable, and resilient from day one.” – Christoff Labuschagne.

If you’re being pulled into full-stack overhauls or costly “digital transformation” without clear business results in mind, pause. Outcomes come first. Tools come second.

Want These Lessons in Your Stack?

At Kohde, we help South African enterprises modernise data infrastructure securely, with scalability and cost efficiency, without sacrificing everything that works.

If you’re dealing with bloated storage, high compliance pressure or just need to move faster without risking uptime, let’s explore how these lessons could apply to your systems.

Chat to Christoff and the team for help designing a small-scale pilot that proves value from day one.