Programming Challenge: Scalable Job Queue System (Jan 22, 2025)

Design and implement a job queue system that can process multiple tasks asynchronously, ensuring scalability for enterprise workloads.

Basic Requirements:

✅ Implement a task queue where jobs can be added and processed in the background.
✅ Use multi-threading or multi-processing to handle tasks efficiently.
✅ Each job should have a status (e.g., Pending, Processing, Completed).
✅ Allow users to submit new tasks dynamically.

Bonus Features:

🔹 Implement priority levels for jobs (e.g., High, Medium, Low).
🔹 Store job information in a database (SQLite, PostgreSQL, etc.).
🔹 Expose a REST API for adding jobs and checking statuses.
🔹 Add retry mechanisms for failed jobs.
🔹 Integrate a message queue system (e.g., RabbitMQ, Redis, or Kafka).

Example Usage:

# Adding tasks to the queue
job_id = job_queue.add_task("generate_report", {"user_id": 123, "format": "PDF"})
print(f"Job {job_id} added!")

# Checking job status
status = job_queue.get_status(job_id)
print(f"Job {job_id} status: {status}")

Why This Matters for Enterprises?

Enterprise systems often need asynchronous processing for tasks like:

• Data processing pipelines

• Report generation

• Background automation

Building a scalable job queue system teaches key concepts in parallel computing, distributed systems, and cloud scalability.