Imagehoster Upgrade: What Changed and Why It Matters

As many already know Ecency running our own instance of imagehoster for long time (6+ years) and we have made many improvements and iterations of software over the course of last few years. Our modifications became so messy that it was kept on private repo until recently. We decided to push our changes into public repo and do some cleanup https://github.com/ecency/imagehoster so community can check our changes and improvements.

This is a comprehensive modernization overview of the Ecency image hosting and proxying service - from a 2017-era codebase to a production-hardened, performant system. Over the years our changes and instances were focused around serving webp image format as priority. As it would help website and app become faster to load due to superior image compression and size, that has changed in last couple years as technology improved we saw switch to avif image compression. So our current instance prioritize avif image compression then fallback to webp and then serve original or legacy format. Entire Ecency mobile app and website usages were using dedicated path for /webp/, and doing content negotiations on app side, we have migrated away from webp dedicated path to better content negotiation directly from imagehoster. Which improved performance across the board and became scalable and easier to maintain.

Codebase is much more modern and utilizing latest library versions. Below tables show differences from old imagehoster codebase to new one.

At a Glance

Area	Before	After	Impact
Node.js	8+ (Alpine)	20 (Bookworm)	Modern APIs, better performance
TypeScript	3.9	5.7	Stronger types, faster compilation
Linter	TSLint (deprecated)	ESLint	Maintained, better rules
Sharp	0.32	0.33.5	Bundled libvips, AVIF/HEIC support
AWS SDK	v2 (monolithic, 60MB+)	v3 (modular, ~3MB)	Smaller builds, tree-shakeable
Redis client	v2 (callbacks)	v4 (async/await)	Clean async code
`@hiveio/dhive`	0.14	1.3.4	Stable API, better types
Docker image	Alpine + manual vips build	Debian slim, Sharp bundles vips	Simpler, more reliable

Key Improvements

Storage: Custom S3 Store (replaces s3-blob-store)

Feature	Old (s3-blob-store + aws-sdk v2)	New (custom S3BlobStore + SDK v3)
Upload method	Stream-only	Direct `putBuffer()` - no stream overhead
Write stream	Double-buffered in memory	Streams directly to S3
Prefix deletion	Not supported	Native `ListObjectsV2` + batch `DeleteObjects`
Bundle size	~60MB (full aws-sdk)	~3MB (`@aws-sdk/client-s3` only)
Error handling	Generic errors	Maps S3 404 → NotFound correctly

Image Format Support

Format	Before	After
JPEG, PNG, GIF, WebP	Yes	Yes
SVG	Yes	Yes
AVIF	No	Yes (encode + decode)
HEIC/HEIF	No	Accepted (decode limited by Sharp)
APNG	No	Yes (animation preserved)
BMP	No	Yes
Content negotiation	Partial	Auto via Accept header (AVIF > WebP > original)

Rate Limiting

Aspect	Before	After
Library	`ratelimiter` npm package	Custom sliding-window (Redis sorted sets)
API style	Callbacks	async/await
Precision	Second-level	Microsecond-level
Window type	Fixed window	Sliding window (fairer)
Weekly limit	300 uploads	700 uploads
Max upload size	15MB	30MB

RPC Failover

Aspect	Before	After
Endpoints	1 (`api.hive.blog`)	6 endpoints with auto-failover
Timeout	None configured	2000ms per request
Profile caching	None	30s TTL (node-cache)
Profile API	`get_accounts` + JSON parse	`bridge.get_profile` (no parsing needed), valid reputation check

Cache Invalidation

Aspect	Before	After
Invalidation	Not supported	`?invalidate=1` with token auth, DMCA requests quick cleanup
Cache bypass	Not supported	`?ignorecache=1`
CDN purge	Not supported	Cloudflare API integration
Stale detection	None	Auto-detects corrupt/non-image cache entries
File deletion	N/A	Direct key deletion (no directory scan)
Avatar invalidation	N/A	Enumerates 12 known variants (4 sizes × 3 formats)
Cover invalidation	N/A	Enumerates 3 known variants

Fallback System

Aspect	Before	After
Mirror sources	1 (steemitimages.com)	5 mirrors + default fallback
Retry logic	None	Sequential with 5s timeout each
Corrupt cache	Served as-is	Detected, purged, re-fetched
Fallback caching	None	Cached under both fallback key and image key
URL migrations	None	Automatic (3speak, InLeo, esteem.ws)

Security

Feature	Before	After
SSRF protection	None	Blocks private IPs, IPv6 loopback, link-local, ULA
IPv4-mapped IPv6	Not checked	Detected and blocked (::ffff:127.0.0.1)
Blacklists	Static JSON arrays	Dynamic remote fetch with TTL + local fallback
URL validation	Basic	Protocol check (http/https only)
Invalidation auth	N/A	Token-protected header

Docker

Aspect	Before	After
Base	Alpine (manual vips build from edge repos)	Debian Bookworm slim
Sharp setup	System libvips required	Sharp bundles own libvips - zero system deps
Runtime packages	vips, heif, aom libraries	wget only (for healthcheck)
Healthcheck	None	Built-in, respects `$PORT` env
Build reliability	Fragile (Alpine edge repo dependencies)	Stable (npm handles native modules)

Performance Optimizations

Optimization	Detail
Direct buffer upload	`putBuffer()` skips stream overhead for S3 writes
No directory scanning	Invalidation deletes known keys directly instead of scanning millions of files
Background deletion	CDN purge fires immediately, file cleanup runs async
Streaming directory reads	`opendir()` for O(1) memory when prefix scan is needed
ETag support	304 responses save bandwidth for repeat requests
Fallback caching	Failed images cache the fallback under the specific key, preventing repeated re-processing
Animation detection	Checks `metadata.pages` for APNG, not just GIF mime type

Code Quality

Metric	Before	After
Source files	~15	~22 (better separation)
Test files	~5	~10 (expanded coverage)
Type safety	Minimal (noImplicitAny: false)	Type guards, explicit interfaces
Error handling	Basic try/catch	Structured errors with codes and metadata
Logging	Basic bunyan	Structured with request tracking, context tags
New modules	—	s3-store, fallback, fetch-image, constants, blacklist-service, image-resizer, cache

Some of the changes are focused towards improving our maintenance. For example, DMCA requests that we receive are unified inside website codebase and served to imagehoster as json file so redeployment of imagehoster becomes unnecessary, simple change to website codebase handles DMCA requests. Local and cloudflare cache cleanup also easy, previous we used to run separate script https://github.com/ecency/cf-cache to clean up cache manually, now that's handled automatically.

Overall codebase is much easier to maintain and uses best approach and improvements we added and polished over the years.