I run an app (ciboulette.net) in the cloud. This app generates about 1000 € / month of revenues. I try to run it in a way that doesn't cost too much, while keeping good performance.
I started by hosting it on Heroku with a paid mLab database. It quickly got expansive, the database in particular. At the time I had barely 6 clients and the hosting was already costing around 40 € / month adn getting laggy.
I switched to Digital Ocean, with a small instance at 5€ per month I could run the app much more smoothly. Hosting the db on the same machine as the app itself got rid of most of the lag.
I couldn't have made that switch without the help of meteor up, an amazing free tool that helps you deploy meteor apps very easily.
For backups, I just have a cron on my dev machine that ssh to production and pulls a mongo dump every day. They are about 650MB each now, and I've upgraded the production instance to "premium intel" : 2 vCPUs, 4GB ram, 80 GB SSD storage. It costed 28$ / month.
I host static files on an s3-like "spaces" bucket that cost 5$ more per month. So in total i got good performance for 30 € / months
I heard that hetzner could be a bit cheaper than digital ocean, so I opened an account there. I was happily surprised, it is indeed better value. I use two cx32 machines, one for prod and one for staging. They each cost 7.56€/month, with 4 vcpu and 8gb ram, basically double the power of the DigitalOcean machine for less than a third the price.
I thought it would be interesting to compare cloud and physical machines, here are the result, and below are the details
| Machine | CPU | RAM | Sysbench | MD5 | Idle | Peak | Initial price | Monthly cost (idle) |
|---|---|---|---|---|---|---|---|---|
| Dell 7010 SFF | i7-13700 | 15Gi | 66315 op/s | 2s | 30 W | 215 W | 1600 € new | 5.40 € |
| Lenovo M700 | i5-6400T | 15Gi | (absurd) | 22s | 10 W | 30 W | 55 € used | 1.80 € |
| Hetzner cx32 | 4 x vcpu | 7.6Gi | 3362 op/s | 17s | 7.56 € | |||
| Mini forum | N4020 | 3,6Gi | 3172 op/s | 38s | 6 W | 175 € new | 1.08 € | |
| DigitalOcean | 2 x vcpu | 3,6Gi | 1987 op/s | 23.89 € | ||||
| Heroku | (8x8375C) | (61Gi) | (7159 op/s) | (11s) | (7 $) |
Note that the heroku machine does not match the specs you pay for at all, it should have 0.5GB of ram not 64GB, as explained below.
I would like to compare cloud machines with computers I physically own, so cpu tests on both of them. I measure the electrical consumption of the machine, both while performing the test and idle. Realistically, the machine is mostly idle but needs to be fast enough to work in the day.
The monthly price was computed this way for physical devices :
monthly_cost = power consumed idle (watts) * 24 (hours per day) * 30 (days per month) / 1000 (Kw per w) * 0.25 (€/kwh price in france)
For cloud devices, the price is simply the one demanded by the hosting service
I got the ram with this command
~ $ free -h
total used free shared buff/cache available
Mem: 61Gi 30Gi 2.0Gi 67Mi 30Gi 31Gi
Swap: 121Gi 1.2Gi 120Gi
I then took the total Mem size
For cpus, I ran
cat /proc/cpuinfo | grep "model name"
I ran two tests on each machine, the first one is sysbench
sudo apt install sysbench
sysbench cpu run --threads=16
The result would look like the code below, "events per second" is the result, higher is better
sysbench cpu run --threads=16
sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 16
Initializing random number generator from current time
Prime numbers limit: 10000
Initializing worker threads...
Threads started!
CPU speed:
events per second: 11162.86
General statistics:
total time: 10.0009s
total number of events: 111655
Latency (ms):
min: 0.33
avg: 1.43
max: 160.65
95th percentile: 4.49
sum: 159767.60
Threads fairness:
events (avg/stddev): 6978.4375/1210.28
execution time (avg/stddev): 9.9855/0.01
To test most devices, I just logged in by ssh and ran the commands.
My M700 gave an absurd result for sysbench because I ran freebsd on it, and the compiler for that target optimized away most of the benchmark code. That's what pushed me to add a second benchmark test that's easier to run on any pc (the MD5 hashing).
# MD5 benchmark
time (for i in {1..16}; do (dd if=/dev/zero bs=1M count=1512 2>/dev/null | md5sum > /dev/null) & done; wait)
You should check the "real" line in the output, lower is better
real 0m2.121s
user 0m28.930s
sys 0m11.597s
For heroku, I used the "heroku run" command, but the results are suspiciously good. I suspectrf this kind of command gets a beefier dyno assigned that any regular process.
mkdir benchmark
cd benchmark
git init
heroku create benchmark
heroku git:remote -a benchmark
echo "web: bash" > Procfile
echo "python-3.11" > runtime.txt
echo "sysbench" > Aptfile
heroku buildpacks:add --index 1 https://github.com/heroku/heroku-buildpack-aptck-apt
git add .
git commit -m "Initial commit"
git push
heroku run bash
sysbench cpu run --threads=16
So I then changed my approach and used the procfile directly.
Procfile
web: bash bench.sh
bench.sh
free -h
cat /proc/cpuinfo | grep "model name"
time (for i in {1..16}; do (dd if=/dev/zero bs=1M count=1512 2>/dev/null | md5sum > /dev/null) & done; wait)
sysbench cpu run --threads=16
This still showed very hight ram and CPU
heroku logs -t
[web.1]: total used free shared buff/cache available
[web.1]: Mem: 61Gi 26Gi 4.5Gi 111Mi 31Gi 35Gi
[web.1]: Swap: 121Gi 175Mi 121Gi
[web.1]: model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
[web.1]: model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
[web.1]: model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
[web.1]: model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
[web.1]: model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
[web.1]: model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
[web.1]: model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
[web.1]: model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
[web.1]: real 0m8.683s
[web.1]: user 0m38.177s
[web.1]: sys 0m12.991s
[web.1]: sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)
[web.1]:
[web.1]: Running the test with following options:
[web.1]: Number of threads: 16
[web.1]: Initializing random number generator from current time
[web.1]:
[web.1]:
[web.1]: Prime numbers limit: 10000
[web.1]:
[web.1]: Initializing worker threads...
[web.1]:
[web.1]: Threads started!
[web.1]:
[web.1]: CPU speed:
[web.1]: events per second: 12035.49
[web.1]:
[web.1]: General statistics:
[web.1]: total time: 10.0012s
[web.1]: total number of events: 120387
[web.1]:
[web.1]: Latency (ms):
[web.1]: min: 0.33
[web.1]: avg: 1.33
[web.1]: max: 168.64
[web.1]: 95th percentile: 2.39
[web.1]: sum: 159648.02
[web.1]:
[web.1]: Threads fairness:
[web.1]: events (avg/stddev): 7524.1875/1082.56
[web.1]: execution time (avg/stddev): 9.9780/0.02
The specs of the web "basic dyno" is 500MB ram one "VCPU", but it seems like we are getting way more than that here. There's something wrong with my method for Heroku. Let me know (renan@lecaro.me) if you have an idea why that is so.
My favorite framework has a cloud hosting offering, this is how they finance the development of the open source framework, and it shows. Once you pass the free usage threshold, the pricing looks scary : a "professional" container with 2 GB of ram and 2 "ECU", whatever that is, costs 158 $ / months. Instead of 5$.
1€ / month of electricity can run a production app on premise, with an initial investment for the server.
10€ / month lets you rent a server from Hetzner with similar performance and greater stability.
100€ / month lets you pay for other people's free tier and "not worry about scaling" with things like meteor cloud / heroku / aws