Introduction
Hello everyone! This week was huge for me. I've been building an open BIM data platform, and I finally made a breakthrough that I'm really excited about.
For over 5 years, I've been struggling with unit formatting and public view limitations in Autodesk Platform Services when working with Revit data. The API doesn't give you clean options to handle unit formats consistently, which becomes a nightmare when you're trying to build a proper data platform. People have been asking about this on Stack Overflow for years (here and here) with no real solution. I kept requesting fixes, but nothing changed. So I finally decided to stop fighting with it and rebuild the entire pipeline from scratch using IFC format instead.
The results blew my mind. This gave me so much motivation to keep going, even though there's still a ton of work left to do.
Demo
What I Built
Here's what the new stack looks like:
- DuckLake from DuckLabs for the lakehouse foundation
- FastAPI for the backend
- SDK and CLI tools to make it easy for developers
- IFC format as the data source (no more fighting with proprietary formats)
Why Not Databricks or Snowflake?
A lot of people ask me why I didn't just use Databricks or Snowflake for this. Fair question. Here's my thinking:
Cost. When you're processing thousands of BIM models and running millions of queries, the costs on these platforms can get out of control fast. I wanted something I could optimize and control without worrying about huge bills every month.
Control and flexibility. With DuckLake and an open architecture, I can experiment with different storage strategies, query optimizations, and data formats without being locked into a vendor's way of doing things. If I want to try Polars or swap out components, I can do it.
Open source and transparency. I want this to eventually be an open platform that anyone can run, modify, or contribute to. That's harder to do when you're dependent on proprietary cloud services.
Don't get me wrong, Databricks and Snowflake are great platforms. But for what I'm building here, a more lightweight, open, and cost-effective stack makes more sense. Plus, building it from scratch means I learn everything deeply, which helps when things break or need optimization.
What It Can Do Now
Right now, the platform can:
- Build a BIM lakehouse directly from IFC files
- Ingest and index more than 1,000 IFC models in under 5 seconds
- Query millions of records in milliseconds
The Benchmark That Surprised Me
One test that really stood out:
I took an IFC model (about 220 MB) with roughly 3.8 million rows across five datasets. I ingested it 5 times to get about 19.4 million total rows. Then I ran queries against it.
Result? Around 100 milliseconds.
That's crazy fast. Instead of waiting minutes to process large models, you can now explore massive datasets almost instantly.
Why This Matters
If this becomes an open platform, I think it could really change how we work with BIM data:
- Explore massive BIM datasets instantly
- Run analytics across thousands of models in real-time
- Build AI agents and MCP integrations on top of it
- Get sub-second query performance at scale
- Access data through an open standard (IFC) instead of being locked into proprietary formats
Basically, we move from slow, locked-down data silos to a fast, open platform where BIM data actually becomes useful for analysis and automation.
What's Next
There's still a lot to build:
- Expand IFC schema coverage and property mapping
- Add graph-based relationship queries
- Figure out incremental update strategies
- Build SDK libraries for different languages
- Make the CLI tools better
- Add real-time streaming ingestion
- Test with 10,000+ model datasets to see how far it can scale
Current Challenges I'm Tackling
Not everything is perfect yet. Right now I'm dealing with a storage issue that's bugging me.
The system is creating tons of snapshot parquet files and saving them to blob storage. This works, but it's not ideal. The problem is cost. All these snapshot files add up quickly in storage costs, and I'm starting to see it become a real issue as the data grows.
I'm thinking about exploring Polars as a replacement for the current data processing pipeline. Polars is a blazingly fast DataFrame library that might help me optimize how I handle these parquet files. Instead of creating so many snapshots, I could potentially process and consolidate data more efficiently in memory before writing to storage.
The idea is to reduce the number of intermediate files and cut down on storage costs while keeping (or even improving) the query performance. It's going to take some experimentation, but Polars looks promising for this use case.
If anyone has experience optimizing parquet workflows or using Polars for lakehouse architectures, I'd love to hear your thoughts.
Update on Autodesk AEC Data Model
While working on this, I also noticed Autodesk has been making some interesting updates to their AEC Data Model API. These two resources are worth checking out if you're curious about recent changes:
It looks like they might be trying to address some of these data access issues. I haven't fully tested it yet, but it's encouraging to see movement in this direction. Maybe in the future we won't need to work around these limitations.
Final Thoughts
This is one of those moments that makes all the late nights worth it. Switching from fighting with platform limitations to building on open standards feels liberating. And seeing these performance numbers tells me I'm on the right track.
The future of BIM data should be open, fast, and queryable. Let's build it together.