This paper presents SWIFT (Swarm-Wise Inference for Fast Trajectory Planning), a distributed one-stage planner designed for efficient multi-quadrotor trajectory optimization in cluttered environments. SWIFT unifies depth-based perception, interaction-aware modeling, and trajectory prediction into a single lightweight network, enabling decentralized real-time planning without reliance on global maps. Each quadrotor processes its local depth observation and asynchronously received neighbor intentions to predict multiple candidate trajectories, leveraging a confidence-guided cross-attention mechanism to selectively model critical inter-agent interactions. The proposed framework achieves sub-millisecond planning latency while ensuring collision avoidance, dynamic feasibility, and cooperative behavior under limited communication. Furthermore, a visual-inertial odometry drift correction strategy based on inter-agent visual detections enhances positioning accuracy during real-world flights. Extensive simulations and forest experiments validate that SWIFT outperforms existing baselines in trajectory smoothness, success rate, scalability, and robustness across diverse scenarios. Note to Practitioners—This work presents a fully decentralized system that enables a team of quadrotors to navigate cluttered environments without GPS, global maps, or centralized control. Each quadrotor plans its trajectory in real time using onboard depth sensing and lightweight asynchronous communication with nearby teammates. The proposed SWIFT planner quickly evaluates multiple motion options through a single neural network pass, enabling safe, smooth, and dynamically feasible trajectories with sub-millisecond latency. A collaborative visual-inertial odometry correction module further improves positioning accuracy by using visual detections between agents. This system is well suited for tasks such as search and rescue, warehouse logistics, and environmental monitoring, where fast and reliable navigation in complex GPS-denied settings is critical. Current limitations include reduced adaptability to unknown moving obstacles and challenges under extremely high-speed flight. However, the SWIFT architecture is designed to be extensible, allowing future integration of dynamic obstacle handling, predictive interaction modeling, and swarm-level objectives such as formation control or cooperative tracking.