By Tom ShultzA few months ago, I started a company called Huron Consulting Group.
It was going to be my second job as a consultant.
I’d been doing that for about a year and a half, working on some small projects for startups.
My clients weren’t all big brands like Google, Facebook, Twitter, Amazon, etc. But I knew there were a lot of small companies that I could work with.
My experience as a software developer gave me an appreciation for small, independent businesses that need help, and I was eager to help.
The first step was deciding what kind of project I wanted to work on.
Huron was a nonprofit, so it wasn’t a big business.
But it was growing, and we needed more revenue.
The second step was determining how many hours per week I wanted.
We were a small, nonprofit startup, so we needed a team of consultants who would be responsible for keeping track of what we were working on.
We wanted a flexible team with a flexible work schedule.
We also wanted to make sure we were building an awesome product, so I decided on a consulting fee.
My first year, I had a very good start.
I made a good salary and got a good pay bump.
Then, things started to go south.
By year two, I was on my third and final round of pay cuts.
I felt really betrayed.
I was thinking, Oh, we should have known better than to work with them.
We should have been working on a different product, or building something better.
We’d been building software for a while, and it seemed like we were just getting started.
I wanted more money, but I also wanted more time.
I had an idea that I wanted a real-time, real-data-driven business model.
I started working on projects with other Huron consultants and found that they had very different goals.
Some wanted to create a platform for real-world data and data-driven decisions.
Others were focused on helping small businesses grow faster.
And some wanted to build a fully automated and scalable product.
I also began to see that Huron had some big problems with its data-science infrastructure.
Hron had two data centers in Seattle, which were designed to be efficient for data analytics and to minimize latency and data collection.
One of those data centers was a disaster site that was used for storage and retrieval of disaster data.
In a couple of years, we learned that the data from that disaster site had leaked to the cloud, where it was eventually released to third parties.
That information was used to track real-estate trends in Seattle.
That’s a really big problem for an open data platform.
It made the platform less usable, and more valuable to third-party data vendors.
Another problem was the fact that Hron was so expensive to build.
The cost of building an application to track Seattle real-life data was $150,000.
That means that for a startup to run a large data analytics project, it needed to be building hundreds of servers and hundreds of thousands of racks.
There were other costs associated with building that infrastructure, too, including hardware, software, and the like.
We decided to scale Huron’s infrastructure up and put it on our own cloud servers.
We would use a combination of open-source software and Amazon Web Services.
And, since we would have to pay AWS a subscription fee, we were hoping to avoid the cloud fees that would be associated with Amazon Web services.
We started with a very small team.
I asked for volunteers to join the team.
They all had jobs and were eager to get started.
A couple of weeks later, we had a team with around 50 people.
In the next few weeks, we went from 40 to 75 people, and by the end of the year, we’d grown to around 100.
We built the entire infrastructure on AWS.
That meant that we were able to have a much higher level of control over how Huron worked.
We could make sure that everything that happened on the platform would be logged.
We didn’t have to worry about what the vendor or the customer might want to do with our data.
We built Huron with a simple data analytics framework.
We had a single data source that tracked real-name, address, and phone numbers.
The data came from our Google Analytics tracking API, which lets us track real time data from Google Analytics.
We added two new layers to the system.
We created a new data set called “city” that tracked a city in Seattle and a whole host of other metrics.
We also added a new layer called “census” that kept track of who lives in Seattle at any given time.
We needed to keep track of the census, and then we needed to maintain a database of all the people who live in Seattle who had voted for the city council.
We created a “city history” that tracks how many times people voted